Yes, it lives again! I would like to point how extraordinary Alidar Moxie is, her skills for a complete catastrophe recovery totally leaves me in awe. Kudos sweetie, you really made it in such a short time despite all of the challenges. So now we can concentrate on Codali and expanding our activities as we see fit.
Read this for a description of the problems we experienced and how we solved them.
Anatomy of the Problem
As our loyal customers know, this website and all of MechanizedLife’s Main product lines went offline on November 14th 2008. This was due to several factors that seemed to happen in an order that was improbable and should never have happened.
Network Area Storage Failure
Part of my backup strategy was a home Network Area Storage (NAS) device that contained two mirrored hard drives and contained daily snapshots of both the MechanizedLife website ecosystem and databases. Sometime in late October, I noticed that this device was not only having trouble serving files back to me, but was having issues storing files, as well. While I thought I was covered, as I could place the drives into a computer and pull the files out, this did not prove to be possible due to a spreading corruption issue with the controllers themselves. This, of course, spread to the worst possible sections of the drives, the MechanizedLife backups, rendering them useless (along with several areas of dear personal data).
Internet Service Provider Wonkiness
My first internet service provider, a friend, supported MechanizedLife as best as he could and only asked that I move on when I overwhelmed his servers. When I moved to my new provider, I took their ‘unlimited’ plan hoping to never have to worry about bandwidth or storage issues as they touted unlimited amounts of each. They gladly accepted my money, 1 year in advance, and set up my account. The first time they told me that I had overwhelmed their server, I apologized and tried to optimize my databases further (keep in mind, please, that I am an Oracle Certified professional, have worked as a DBA full time, and currently have clients who contract specifically for my DBA skills). The second time they told me this I realized that they simply wanted a scapegoat, but asked if they had dedicated or virtual dedicated servers I could upgrade to, which they did not, and I started looking for upgrades. I complained and they did not say anything else to me for a long while.
Recently, it started happening again. My business grew, thanks to loyal customers like you, and they started complaining again. I was actively looking for a new provider when they permanently suspended my account and said ‘we will not give you any backups’, which in light of the above NAS problems was a problem. I offered to pay for them, they refused. I threatened to sue, they refused. They, of course, pointed to some obscure point in the terms of service that said they didn’t have to. Long story short, I wasn’t getting them no matter what I did (I even offered to let them keep the last of my year’s service agreement money in exchange).
The Outage
Since 95% or so of MechanizedLife products have an online component, I took them immediately offline and placed an order for a VPS, essentially a virtual dedicated server of my own that I can run as I see fit. This will allow me to do things like take daily snapshots of databases and run custom software and operations that my old provider would not allow. I am also on my own server, so I cannot effect anyone else’s service. Its more of a ‘you’re on your own’ environment, which suits me fine.
Codebastard Redgrave, Codie, immediately got a notice up on her blog and helped me to pick out the new service and has helped me tremendously in selecting service providers and options. I chose a company where my needs are the minimum they offer so that if I have to grow, which I hope I do, I can do so, and in this environment its as simple as rebooting the server.
Product Status
At the moment, all products (with the exception of the Del.ici.ous tagger which will be back online by the end of the week) are back online. StatsCollectors came online the middle of last week automatically as the domains became available. CalendarCogs are back online, but the database was destroyed meaning customers will need to go back onto the website and redo all of their settings. Unless they happened to be near the device when it reset they do not know their password and the easiest way to reset it will be to either contact Paypabak Writer or to purchase a free Password Resetter.
Going Forward
With the help of Codie, snapshots of the database and website will be downloaded not only to my home computer, but also to her location as well, ensuring this cannot happen again. The backup process for the database is in place already and the process for the websites will be in place later today. This move is actually good for MechanizedLife, as it gives me the confidence to introduce some products that I had been reluctant to release in my old hosting environment. It also coincides with work Codie and I have been collaborating on in regards to our joint venture called Codali that we already had press releases and logos made for when all this mess occurred.
Thank You
Thank you for sticking with me through all of this. I know that it is inconvenient and not at all fun. In order to say thank you, we are going to offer a promotion. Between December 1st and the end of the year, any MechanizedLife product you own that was effected by this outage, we’ll deliver one of the same item any to avatar on the main grid you choose. Simply send us a notecard with your name, the name of the product, and the name of the avatar to whom you wish to have the promotional item delivered. Notecards should be sent to Paypabak Writer and she will see that they are delivered as quickly as demand allows. Thank you again for your loyalty and patience. Stay tuned for more new and exciting tools to be released under the new Codali brand.
