Instead of blasting social media with post after post of a rundown on something only affecting 1 server, I’m making this thread for updates.
This morning it seems MySQL has his a corruption in the innoDB setup, now I am working on this to resolve the issue as customer data is fine. That’s what is weird, if I bring it all back up in a read-only mode, customers sites load (you can’t login as its read-only) but it works.
So currently working on a solution that has worked in the past on other servers however it does require me to keep it in this read-only mode to get data squared away. So I’ll update this thread as I have more information so if you are currently having issues on Lilly, stay tuned here for more updates.
10/22 @ 9:14AM : Currently running individual backups of all databases in real-time to ensure we have all of the latest data 100% secured and safe, before attempting any further modifications. A lot of data so hang-tight. Updates shortly.
10/22 @ 9:21AM : Backup is completed, working on a forced re-index, if this works (sometimes it does) it shot-cuts our reduces the time required so we can get back online - but if not moving on to option #2 which is a bit more sure fire - but trying to keep downtime minimal if at all possible.
10/22 @ 9:30AM : Forced re-index seems to be holding. Watching still - not wanting to give the all clear just yet - reviewing other sites manually on Lilly looking for any issues or database connection problems - if you are still having problems hang-tight and watch for email/ticket replies. Feel free to post a comment here if you like!
10/22 @ 9:56AM : So far so good, found a few sites with issues, working on them now - Updating tickets and emails shortly with updates.
Resolved: All is well! If you are still having issues - please open a ticket, but all sites I’ve spot checked are looking good!