Thanks Jason! What was broken?
Thanks, Bill. I appreciate that.
The first thing I noticed was that the server load was heavy. And by heavy, I mean the load average was 40x what it should be AT MAX! The CPU was running at 100% and the RAM was near maxing out. The database, which the forum relies on, was the culprit. Why that was, I wasn't entirely sure. I'd restart the database service, things would improve but only briefly. Then it'd be insane again.
Cod3poet helped me walk through some steps and gave me some suggestions. Where I ended up, I should have started. In the SMF logs, one error came up repeatedly, a total of 7700+ instances over 7 hours. Yes, you read that right, over 1000 database read errors per hour! I believe it was causing the issue. I've attached it so you can view it. It's tiny unless you click on it.
As you can guess, it checks a viewer's IP address to give a total of the unique users online at any given time. You see that total at the bottom of the main SMF page. So every time someone visited, it was choking on this issue.
Fun, huh? I still don't understand the first cause but it works. And that's more than we can say about most things today.
NECRO BUMP!
Amazon crawler that went awol and caused a ton of issues, could be for indexing OR it could very well have been an offshoot of the AI training craze where companies are crawling public data to train their AI models.
The IP in question is for an amazon crawler out of Ashburn Virginia. Cool beans.
The most interesting part of your post was this:
Quote from: cod3poet on August 29, 2024, 11:20:01 AMNECRO BUMP!
Please explain.
The rest of your post is intriguing. I haven't deleted the error logs yet so I'll have to check and see how many IPs were doing this. I know it was a limited number.
I'm eternally thankful it didn't create such a flurry of traffic that I'd be staring down a $2000 bandwidth-exceeded charge.
Necro bump, old school forum slang for commenting on an old and generally considered dead post.
Also seemed relevant because it seems I have magically risen from digital silence.
I'd be happy to review the results with you I have been working on some data science(ey) stuff as of late and may start a new post just for that.
Quote from: cod3poet on September 10, 2024, 11:56:01 AMI'd be happy to review the results with you I have been working on some data science(ey) stuff as of late and may start a new post just for that.
Sure, I'd love to see that. Others might, too. I've been taking a couple of courses related to data analysis. Making dashboards is fun!
Oh that cod3poet!, likes to be a toad but doesn't knowit. . . . . ribbit !