Signups are disabled until I can complete the work needed to fix uploading photos.  I'll leave the signups disabled until I can get the paid accounts who have requested it moved over.  This process will take place in two steps, with the first step being an import of the account itself, followed by the second step of importing the photos.

Getting there.  Lots of moving parts here.

Loading mentions Retweet

Whew.  The drive recovery company just called and told me they recovered almost the entire drive, save for about 500KB or so of the data.  The last recovery I did on the failed drive in September recovered all but 100MB of data, and only ended up affecting ~60 photos total out of about the 7 million we have on the system (this includes thumbnails).  I would expect only a few photos to be affected in the recovery, so it's unlikely anyone will lose anything.

Next step is to install the recovered drive and then immediately dump this data to a new array I've built while the drive has been out.  All photos will be moved to this new array, and then from there will be used to import your photos to the new server on Amazon.

One Amazon server has ~800GB of storage on it natively.  I will utilize this space for photos and thumbnails (we make 10 for each photo) and then that drive will be duplicated onto what Amazon calls a "snapshot".  My plan is to do daily rolling snapshots, and rolling weekly snapshots.  That means there will be one instance of a photo on the server, a daily snapshot, and a weekly snapshot.  Additionally, I'll be rebuilding the NAS at the house with newer drives, and will be rsync'ing the photos from the servers back to the house as a fallback. Each photo will be backed up 4 times.

At some point the cost of storing free account photos will be more costly than I want to carry.  I'll be implementing cleaning scripts to delete expired free accounts from the system to ensure these photos don't clutter up the system.  It's usually a bunch of porn anyway.  Damn freeloaders.

Hang on - not much longer now!

Kord

Loading mentions Retweet

The recovery company is suppose to be swapping heads on the drive.  I sent email yesterday for an update and didn't get any more than "we're working on it back".  I looked, and I sent the confirmation to do the work in over a week ago, but the turnaround time is suppose to be 5-6 working days, so I guess they aren't that far off the mark, especially with the work they are doing - in a clean room even.  Shouldn't be much longer.

New Features
I've been wrestling with the uploader the past few days.  Suprisingly it's been the hardest thing to get running on the new server, and it's really starting to piss me off.  We used a code framework called Twisted when we wrote Zoto.  Twisted has been upgraded a few times over the years and Zoto's older code uses some stuff that has since been removed.  

Anyway, the uploader client may or may not make an return at relaunch.  As the current Flash uploader is also busted, I'll have to impliment a new one before you guys can upload stuff.  Luckly I implemented a portion of one of our older uploaders and there are a bunch of drop in uploaders that work like our old Flash one, so it should be fairly straightforward.  And you guys wondered why I never moved stuff to Amazon before!  ;)

One possible option for a "new" uploader would be the use of a product I've found called ExpanDrive.  Basically you'd download/buy it ($40 though) and then use it to mount a directory on Zoto.  You could drag and drop stuff in a local directory, and I could move/copy stuff out of this directory into your account.  There's a free version of something similar for the Mac called MacFusion, which I use a bunch at work.

Well, I have a bunch of ideas for stuff I can do once things get moved.  It becomes SO much easier to work on the software once I get it off these older servers.  Just a lot of work to get there.

Hearty Thanks to You Guys
You guys have been really supportive through all this - especially seeing how I haven't really communicated well with you over the last few years.  I've had about 80 people request their photos to be imported, yet not one refund request has been submitted.   I really appreciate you guys making my life easier.  Thanks a ton.

Loading mentions Retweet

We've been in a lawsuit over the last few years.  As we were a) not able to pay their licensing fees, nor b) willling to shut the site down,  I ignored it.  I was hoping it would go away, and just found out it did just that.

This doesn't mean we can't be re-sued, but it's a good thing. 

Now if I can just get the website back up and running.  I've bumped into a few problems with the uploader.  Still waiting on the return of the drive as well.

Loading mentions Retweet

Got the new server installed and running with all the newly updated software the server requires to run.  Should have the site itself up tomorrow sometime.  Will begin the process of installing paid account usernames on the new system, getting ready for the return of the hard drive, and images sometime next week.  Signups will still work, but conversions to paid accounts (and taking payments) will be suspended until I can port the system to use Recurly.  I fucking hate Paypal.

Fixing Some Other Stuff Too
Still thrashing around on getting the uploader client to run.  I'm in a better place to be able to fix the older Flash uploader - there are a ton of free ones out I'm looking at implementing.  Oh, and I'll be ripping out printing as soon as the site comes back online.  I think it was broken, but regardless printing goes.  Most people didn't use it way back when, and there are a few patents that are seeking renumeration for sites like Zoto using "their" technologies.  Whatever.

Loading mentions Retweet

I've assembled a Wufoo form for handling requests from everyone: 

http://zoto.com/import.html

This way I can better track and record who wants what.  Last time this happened a few people slipped through the cracks due to me not having the correct Paypal email addresses for people.  If I can't find the correct Paypal email address for you, I can't do a refund.  Paypal only let's me refund people within the last 3 months or so with a transaction option.  After that, I have to just issue a regular payment to your account.  Without that account, and the original transaction, it's impossible to do the refund!

For those of you looking to keep your account, or get a dump of your photos, just fill out the form and I'll process it as soon as I have the drive array back up.  I expect to have the site up by the time the drive gets back from the recovery service, and you'll be able to log in and use your account, albeit without any photos.  At that point you can start using the account again, upload photos, tag stuff, etc.  If you request your older photo import, and have uploaded the same photo, the system will restore any meta data you had on that older photo.

I'm sure there will be hiccups in all this process, but I'm doing the best I can to make it right.

Kord

Loading mentions Retweet

Estimate for recovery is $2,300.  :(

Loading mentions Retweet

Drive is in the hands of the recovery guys currently.  I'll update as I find out more from them in the next few days.  It may be Monday before I know more.  The outlook from them when I talked to them on the phone was that it was a bad read head, which is fixable in their book.  We'll see.

In the meantime I'm bringing up a new instance of Zoto on Amazon, with a temporary webserver already launched today to handle the inbound traffic.  Getting the site up (sans photso) will take me a few days - mostly work in getting the server image built and configured.  Photos will be stored on a RAID'd S3 instance, so the issues we've seen with our own NAS should become moot.

Thoughts on migration are many.  I'll probably have to write an import script to pull accounts, photos and tags from the older system.  Again, I would transfer over all the info if it were possible, but it's just a nightmare to figure out what is needed, and what is not.  There are a ton of older accounts on Zoto that aren't used, which are gumming up the works.  Better to let everyone sort it out on their own. 

BTW, I've gotten a few emails from people (who are suprisingly chill with the whole ordeal) that they just want their photos and/or money back.  That's my priority ATM to recover the photos and do refunds - will do more as soon as I have time.  I appreciate the support you guys have given.  Email for contacting me is zotohead@zoto.com.

Kord

Loading mentions Retweet

The bad drive has been shipped off to Dallas for attempted recovery.  The guy I talked to at First Advantage said that it was fairly common for one or more of the drive heads in these WD drives to fail, even after a long period of operation, and he had seen instances of this type of failure (multiple drive at once) before.  This doesn't bode well for using those drives much longer, as the remainder of them are all of the same ilk.

I'm committed to taking whatever steps necessary to recover the drive, FWIW.  One possible solution would be to configure a new system to import data (as needed) from the old one, allowing me to run the site on AWS where this sort of crap is much less likely to happen.

BTW, wy apologies to the Posterous guys - looks like when I set the CNAME record for the blog I inadvertently sent people to their main page.  Hasty work is shoddy work.  Should be working now thanks to their help.  I'll be trying to set a server up on AWS while the drive is off for repairs, and try to move this back to our normal format - not on Posterous. 

Kord

Loading mentions Retweet

Still working on recovering drive array.  Sending one drive off to recovery service to see if they can do a platter swap and recover.  Confirmed theory that drives failed on a rebuild of the array - looks like one drive failed then the recovery started and killed the other drives.

We're commited to trying all avenues at recovering the data, so please sit tight while we do this.  I'll report back on any findings.

Worst case it looks like a partial recovery can occur on the remaining disks, although there will assumably be data loss from that process, given the other drive is not recovered.

Loading mentions Retweet