Thanks Jason, for getting the PLUG website back up and running

[ssfc72]

Thanks Jason! What was broken?

[Jason]

Thanks, Bill. I appreciate that.

The first thing I noticed was that the server load was heavy. And by heavy, I mean the load average was 40x what it should be AT MAX! The CPU was running at 100% and the RAM was near maxing out. The database, which the forum relies on, was the culprit. Why that was, I wasn't entirely sure. I'd restart the database service, things would improve but only briefly. Then it'd be insane again.

Cod3poet helped me walk through some steps and gave me some suggestions. Where I ended up, I should have started. In the SMF logs, one error came up repeatedly, a total of 7700+ instances over 7 hours. Yes, you read that right, over 1000 database read errors per hour! I believe it was causing the issue.  I've attached it so you can view it. It's tiny unless you click on it.

As you can guess, it checks a viewer's IP address to give a total of the unique users online at any given time. You see that total at the bottom of the main SMF page. So every time someone visited, it was choking on this issue.

Fun, huh? I still don't understand the first cause but it works. And that's more than we can say about most things today.