Swamp Rat's Rat Pack

The database server used by this site (and all my other websites) experienced a failure earlier today. The SRRP website is now using its backup database server. Unfortunately, the most recent database backup is from 4:30 AM April 7th, so any posts made in the past 21 hours are currently lost. Once the primary database server is back up and running I may be able to retrieve them, it all depends on what was going on when it crashed.

The backup server is not as powerful as the primary server, so the site might be a bit slower than usual until I can get the primary back up.

I'll be taking backups every 6 hours (instead of every 24 hours) until everything is resolved. I would do more frequent backups but don't want to take the chance of overloading the backup server.

I'll post a reply once everything is back up and running. In the meantime, continue to use the site normally. If you experience any errors, problems, or excessive slowness (pages taking more than 10 seconds to load), please post a reply to this thread with as much information as possible (precise date and time, what you were doing when the problem occurred, the exact error message if you received one, etc.).

We were back up and running on the primary database server for a couple of hours, but more problems began to crop up. I put the site's database back onto the backup server while I try to figure out what the heck is going on with the primary. Hopefully the site is not loading too slow right now for anyone -- for me, the backup server is noticeably slower loading webpages, but it's not too bad (2-3 seconds to load, versus < 0.5 seconds normally).

Part of the reason why it's slower is because the backup server is less powerful than the primary server, but also because the backup server is located in a different datacenter 2,500 miles away in Las Vegas. The webserver in New Jersey needs to connect to the backup database server in Las Vegas to retrieve data, instead of connecting to the primary database server just a few feet away in the same room.

Please let me know if you see any error messages or pages are taking a very long time to load (> 10 seconds).

Thanks for the update, I was wondering why the site was down when I tried to connect. Hope your other sites are still up as well and you will find the problems. Good luck with it!

I'm going to be switching everything over to the new server soon, most likely this weekend. There should be zero downtime; it will be a seamless transfer.

In case anyone is interested in how this is done, it really is quite simple:

• This site is running on two separate servers, the web server, and the database server.
• The web server holds all the files (website code, images, avatars, attachments, etc.). It processes the PHP code for the site, and sends the HTML to your web browser. The database server holds user data, profiles, passwords (encrypted), and all posts.
• Because they are two separate machines, the webserver has to connect to the database server in order to retrieve this data.

So, to get a seamless transition to a new database server, you do the following:

1. Set up and configure the new database server.
2. Use a process called replication to "sync" the old server with the new server. The old server is called the master, the new is the slave. This means that anytime data is added or changed (posting a new topic, or editing a post, for example), the master updates its database. It then automatically connects to the slave, and tells the slave to perform the same operations to its own copy of the database.
3. Once replication is set up and confirmed to be working properly, you adjust the site's configuration on the webserver, telling it to connect to the slave (new) server.
4. Now the site is using the new database server, and nobody using the site would even notice a thing.
5. You can then stop the replication, or you can reverse the roles and set the slave to be the master. The old master is now the slave, and can be used as a backup.

It's a very simple but very effective solution. When you have a massive website with thousands of hits every second, and/or you need to be 110% sure that no data is lost (a banking system, stock trading system, or the system that logs/records the instruments for a nuclear power plant, for example), things can get much more complex, but the overall idea remains the same.

We're up and running on a brand new database server located in the same datacenter as the web server. Now the site doesn't need to make a trip from New Jersey to Las Vegas and back again to retrieve its data, so things should be just as fast as they used to be.

Please let me know if you experience any issues or receive any error messages.

Thank you for your time

All good, Great job Jersey!