I've moved the main part of the site to AppEngine in preparation for work starting on the new Zoto.  I would expect it to take me around 3-4 months to complete all the work needed to emulate an updated portion of Zoto's old functionality, which is to say "it's capable of hosting photos for peeps".

My mission is to build a photo sharing site that doesn't suck.  Everything I write for this will be Open Souce, so if you don't want an account on Zoto when I launch, then you'll be able to install it on your own Google account and run it yourself.

Over the last few years I've seen oodles of mainstay webapps supplanted by faster and simpler versions.  The trick is figuring out what one or two features you're going to totally nail, and then, well, nail them.

I start the process of trying to figure out what few features *I* want to implement in a really, really slick, kickass photo sharing site.

Time to get cracking.

Loading mentions Retweet

The code responsible for uploading and notifying you of your archive links is now running.  Anyone who filled out the request form and requested a photo archive dump in the last few months will be receiving a link via email sometime in the next few days as your archive is uploaded and processed.

If you haven't filled out the form, or filled it out and didn't request a photo archive but want one, post here and I'll add you to the list of accounts being processed.

I'm actually not sure how long this entire process will take - perhaps a couple of days at most.  It's uploading as fast as my connection at the house can stand!

Kord

Loading mentions Retweet

Been working on the export code for about 8 hours today and am happy to say the process of exporting photos is now underway. 

The first step is to pull a given accounts photos out of the database, locate the images on the backups, then place those images in a directory strucutre that echos the dates and filenames from the original images.  The zip files are by year/month, so I'll be providing some tips on how to do bulk downloads and unzips to Windows or OSX.  Some images don't have the right date on them, so I'm doing best effort to put them somewhere in the directory structure.

As soon as a few of these accounts are zipped, I'll start the second step of uploading the zips to the server on Amazon.  When the upload is complete you'll be emailed a link to download the image archives.

I took the extra step of reporting how many photos you had, how big they are all in total, and a report of which photos (name, date, size) were lost in the drive crash.  I ran my account just now and I had 16 lost photos out of about 5,500 total.  Another account I just ran had only one photo lost, probably due to the fact he was a user from a later stage in the company.

I'll keep you guys posted as this progresses.  Going to dinner.

Loading mentions Retweet

I'm working on getting the export code done this weekend to start emailing people a link to their photos for download.  If you didn't indicate you wanted an archive of your photos on the Wufoo form, I won't be emailing you this link.

Upon futher consideration, and given the circumstances of where we (you guys and me) are, I think it's best to just shut things down.

All in, only 114 people have filled out the form to either get a refund, export their photos, or get their accounts back.  While I've been partially successful at bringing the site itself back online, I have been unable to get the database exported and moved over.  The machine the database is on is now the last box left from the Zoto cluster that is still operational.  I'm worried that if I ran the service off it, it would fail and bring us down again.  I could try to do a partial export, but chances are that I'll fuck things up if I do.  Chances are I'll run into development roadblocks that I don't have time for.

A few angry people have emailed me about how long this has taken and what "we" are doing about it.  First, there is no "we" here, it's just me, Kord, sitting around on the weekend trying to fix this stuff.  Yeah, over the last 2 months I've not been working on it much due to my work schedule, but I am still intent on getting photos back to people who need them.  I'm not making money off this people, quite the contrary, but I am still keen on doing what I can to make things right - no matter how fast you think I should do it.

I promise you don't want me bringing the site back up at this point.  I wouldn't have time to work on it anyway, and it's just a big source of irritation for everyone.  I may rework the site into a Twitter image hosting service though, as the name works well for that, and such a service would be like 20x easier to maintain.  You have no idea how complicated this code is - it's a fucking nightmare to work with all it's dependencies.

Anyway, I've prattled on enough.  Sorry for all the dead air.  Trying to get this done finally.

Kord

Loading mentions Retweet

Last night the last dump of photos from the old drives was completed. After two months of effort, I've been able to recover 99.9% of the photos that were on the system. I'm in the process now of making a complete backup of the recovered photos, and I'm ready to begin work on getting the proper photos moved over to production to bring the site back up.  Backups of these photos will also be available for raw download in a few weeks time - it'll take a while to process everyone.

Percentage-wise, if you had a thousand photos on the system, chance are you lost one of them.  I'd say that's better than losing all of them.

Anyway, I'll keep you guys posted.  The worst is behind me now.  Won't be long to get things back to "normal".

Cheers,

Kord

Loading mentions Retweet

About 2 weeks ago, midway through copying photos, I lost yet another drive out of the array on the NAS.  So far, during this whole ordeal, I've lost a total of 4 drives out of the NAS.  Thankfully, I have complete block-level backups of all disks in the array, so I was able to rebuild the disk with a little swapping around of cables in the NAS.

To give you an idea of the mess involved, I have 11 cables running out of 3 cards on a borrowed PC snaking into the NAS chassis.  I couldn't install the controller cards in the NAS because the motherboard didn't have enough PCI slots to accomidate them.  The problem with this approach is that I'm deathly afraid of disconnecting the cables run because the RAID recover software from Runtime depends on the drive order to work correctly.  This makes it hard to swap back in the backup drives to recover and rewrite the backup images.

As I type this, the copy has be restarted, and is underway again.  I'm starting from the beginning, but telling the system to to ignore existing files.  When I get about 1/4 of the way through, I'll swap back to copying all files, due to the failure of the other drive causing a few photos to be corrupted.  I'd like to ensure all data is recovered normally here.

This ordeal will soon be over, and I aplogize again for leaving the site down for so long.  I felt that, in the event of complete recovory failure, that I would be well to leave it down permanantly.  If, and I'm giving it good chances now, I'm able to do a full recovery, we'll be back up again sometime soon.

Cheers,

Kord

Loading mentions Retweet

Well, after a long month of data recovery, I'm finally seeing intact photos from the drives!  This has been an enormous undertaking, and I've honestly not done anything as difficult as this before.

In theory, I should be able to start uploading photos to the site this week.  I still have to do a "recovery" copy to a set of backup disks before I start uploading, but right now I'm browsing the file system and looking at intact photos, so I think that should go fairly quickly.

Here's a partial breakdown of what had to take place to get where I am today:

• on NAS failure, tried to discover which drive failed first (about 2 days)
• tried doing a controller board swap to reactivate drives (about a day)
• sent suspected last drive off to get recovered (almost 3 weeks)
• on getting the drive back, discovered it was the first failed drive (which means the array won't reassemble)
• do research on software/solution that will provide recovery from a broken (yet intact) array
• decide that backing up current drives is required and an imminent priority
• began backing up old (11) 500GB drives to new compressed drive images, one at a time (about a week) 
• figured out I could have copied faster with another server I had at hand (damn)
• discovered making compressed drive images was a bad idea if i was going to do a virtual reassemble (double damn)
• assembled a new RAID 0 array from a bunch of random drives I had at hand (about 2 days)
• copied drive images yet again to new RAID partitions (two at a time, very fast, about 2 days)
• set new backups aside and put old drives back in NAS
• found out the controller card I had didn't support direct drive access
• went to Fry's and bought (2) controller cards to give me (11) total SATA ports
• discovered the NAS didn't have enough slots for said cards
• stole my son's gaming rig to install cards and access drives
• installed windows 7, then found out recovery software won't run on it
• installed XP (thought I'd never do that again)
• installed about (3) different pieces of software to try out
• ran one for 3 days in "filesystem discovery mode" before deciding it wasn't working
• installed software from runtime.org and then it told me I needed to scan and send in data
• called runtime and open-e for tech support again to double check my approach
• started runtime scan (took about 5 days to scan all drives)
• upgraded my cable to 8mbps upload in preparation for sending photos to AWS
• uploaded runtime scan (about a day)
• waited on results (about 4 days)
• installed yet another piece of software that reads results and finds an XFS filesystem (about a day)
• PHOTOS FOUND OMG

So I still need to copy stuff over, and then figure out a way to upload the photos that have been requested.  This will take AT LEAST another few weeks to fully complete.  There will assumably be issues in doing this!

In the meantime I will activate all the paid accounts on the new servers and turn the site back on.  I was waiting to do this until I knew photo recovery was possible.  Now I do, I see no reason to not bring the site back up.

I'll keep you posted.

Kord

Loading mentions Retweet

Short update on the work done this weekend.  Things have progressed slowly because of the difficulty in installing the software on an Amazon servers.  I finally found a good server image that basically emulates the older servers I was running on the dedicated Zoto servers.  Most of the software is running correctly now, including the uploader!  With the exception of not having user account info loaded, I think most of everything is working correctly.  I should have the site active for paid users sometime later this week.  Seems pretty snappy too!

I've forwarded all traffic to the main page to the blog for now.  I've also disabled signups until things get better, and those will remain disabled until I can get all the details (like emails) sorted.

The recovery of the photos are another story.  The drive that was recovered by the service ended up being the degraded drive, which sucks.  With it being degraded, I can't immediately boot up the array for backup.  Instead, I'm having to copy each drive, one at a time, to a set of backup disks so they can be reassembled for recovery.  It takes about 8-9 hours per 500GB drive, with a total of 11 drives needing to be backed up.  In theory, this process will be done sometime tomorrow afternoon.  Backups good.

The problem with this approach is that I need to copy all these images back again to the NAS to rebuild them correctly.  This will take even more time, so it's unlikely that I'll have any photos recovered before the end of the week.  If I had to guess, I'd say end of the month for sure, but not sooner.  Recovery of a broken RAID 5 array is possible with the data I have, just slow.  I'm still about 95% certain all the data can be recovered, FWIW, so nobody should be worried yet.  I'll keep you posted on the recovery.

This is 3 weekends in a row I've grinded out time for this.  The wife is tollerating it, but not for much longer.  :)

Kord

Loading mentions Retweet

Bit of confusion as I'm trying to get the site working again - have to do it live folks.  The recovery form I had up on the main page is now here: http://kordless.wufoo.com/forms/zoto-accounts/

Kord

Loading mentions Retweet