Hopville Blog

The main outage is expected to last most (if not all) of August 16th, but here you’ll find regular updates on what’s happening behind the scenes.  All times listed are PDT (U.S. West Coast).

• [Latest] Aug 16 @ 11:30pm: Stripped-down site is now LIVE.  Basic features will get added back in over the course of tomorrow (and through the weekend) as the new systems get tested for stability and performance.  G’night folks!
• Aug 16 @ 8:00pm: I would like to take this moment to (again) declare that Internet Explorer is vile software and the scourge of the internet.
• Aug 16 @ 7:00pm: Image migration completed successfully, which means all information from the old website has now been carried over and rejiggered for the new site. Next step is to fully test the site in its new home now that it’s running with a complete data set.
• Aug 16 @ 6:30pm: Image migration is just about complete. While the processing script was adding roughly 100,000 newly generated image files to the new world, I caught a little more sleep so I can be more lucid when I flip the switch later tonight.
• Aug 16 @ 4:00pm: About half way through processing the brewer profile images now.
• Aug 16 @ 3:00pm: One aspect of today’s major migration is a changeover from a cheap shared hosting account to expensive, more robust cloud servers. This provides much more insight into the health and performance of the website. The image processing script puts really high load both on server memory and on network I/O, so I’m already getting very interesting performance metrics that I would have been blind to under the old Hopville.   Cool stuff.
• Aug 16 @ 1:00pm: Okay, image server config is fixed up, processing profile images again – this time for keeps. Have to take thousands of images, generate copies of each in a variety of sizes that will be used on the site, and then upload all the variations to the new image server – so this is another step that takes a while.
• Aug 16 @ 12:30pm: Found a little problem with the image server configuration so I’ll have to make a few tweaks to the processing script to get all the profile images migrated successfully.
• Aug 16 @ 11:00am: Now migrating all the brewer profile images from the old site to make them available to the new image service. Copying them to the new server now, then the next step is to run a processing script over them.
• Aug 16 @ 10:00am: Haven’t found any issues with the DB migration.  Woo-hoo!  Now configuring the “gone fishing” pages so that I can update the DNS for hopville.com. This means the website officially moves to a new address.  The same “down for maintenance” page will display, but it will get served from the new server instead of from the old server.
• Aug 16 @ 9:30am: Database migration complete. I’ll have to run a few quality assurance passes over the new database.
• Aug 16 @ 9:00am: Database migration is progressing smoothly. It’s looking like the “practice” launch in July helped iron out many of the kinks in this, the most sensitive of all the transfer steps.
• Aug 16 @ 1:30am: Okay, successfully generated a clean source on which the DB migration will be based.  DB migration will take hours and require consistent monitoring and QA, plus I’ll need some sleep, so updates will be irregular through early morning.
• Aug 16 @ 1:15am: Now backing up the Hopville 1.0 software, as much for amusement as for safe-keeping.
• Aug 16 @ 12:30am: Redundant database backups complete. Starting brewer profile image backups. This will take a while.
• Aug 16 @ 12:15am: Initiating master backups of the entire site before migration begins.
• Aug 16 @ 12:01am: Old site is now switched off. Let the games begin.
• Aug 15 @ 11:50pm: Pre launch systems check okay.  Going forward with the outage.
• Aug 15 @ 7:20pm: Outage scheduled to begin around midnight PDT.

Since the initial release of the Hopville 2.0 beta will have a stripped-down feature set – and I mean, a really stripped down feature set – I thought it might be good to share some of the details of what 2.0 represents under the covers.

Hopville grew to 100,000 recipes as one of several side projects, without having serious focus given to it.  With Hopville 2.0 I’m beginning to give it real focus, getting it ready to grow to its next milestone of 1,000,000 recipes. First step was to overhaul its infrastructure.  Here’s what happened this week:

• New web host: Since its inception, Hopville has run on an affordable hosting account through Site5.  While they’ve served Hopville well, the nature of that hosting service (cheap hosting on shared servers without system-level access) is too limited for a site operating at professional capacity. The tallest hurdle jumped this week was the migration from their service to cloud servers hosted by Rackspace, with whom I’ve had great experience in a few startups where I’ve worked.
• New web framework version: Hopville was most recently running Rails 2.3, but the version released this week represents an upgrade to Rails 3.2.
• New codebase: Due to some delicious tech tweaks introduced into Rails as of its 3.1 version, plus a completely new approach to how Beer Calculus works, I took the opportunity to overhaul the codebase, rewriting nearly everything from scratch.  (Might’ve bit off more than I can chew, frankly, as this project ran far longer than my original estimates.)
• New database: The site still runs on the open source MySQL database, but for 2.0 it has been restructured and expanded.  Where it took about 20 tables to run the old Hopville, it’s taking about 30 (so far) to run the new service.  One side effect is that folks may have to reset their passwords as the new server and framework use stronger encryption for passwords.  I’ve also isolated the database onto its own server which will help the site to scale and eventually facilitate things like a public API.  You’ll start seeing the reasoning behind the restructuring (in the form of new features) soon…
• New image service: One of the embarrassing historical aspects of the old site was how images were handled.  I thought to myself when Hopville launched, “since nobody is using this site, I don’t have to optimize its images in any way – it doesn’t matter how huge and slow the pages are with zero traffic” Then I proceeded to never upgrade the service after the site grew to have traffic.  I let the browser resize images into thumbnails, etc – if you’re a web developer you know that this is an absolute no-no.  Hopville 2 will have a lot more image functionality (eventually).  Accordingly, with this launch images are finally processed and handled correctly, as well as being stored on Amazon’s cloud, which should speed up some pages quite a bit.

July will be a bit bumpy since Hopville has no QA department and does testing in the wild.  I’ve also lucked into an unfortunately-timed but also all-expenses-paid vacation to Alaska, so as of Friday I’ll be mostly offline for a whole week.  Sorry folks!  Y’all will notice a distinct lack of features during the initial beta testing, as I removed lots of features to focus on the systems migration.  But you’ll just have to trust me that I can see far into the future that was enabled by the Great Migration of Hopville 2.0.  And what I can see out there on the horizon is damn exciting.  The summer of 2012 is the awkward puberty that Hopville has to transition through on its way to becoming an adult.  Stay tuned…

Final recipe count for Hopville 1.0: 118,222

The 2.0 project was a major undertaking, involving a complete rewrite of the application and a restructuring of the database powering the site.  (Don’t worry, all your data will be preserved.)  It’s also the time when the site graduates from a cheap hosting provider to a professional-grade, not-so-cheap one.

So, the migration is quite a juggling act.  With so many technical hurdles to worry about, the website feature set has been cut back dramatically to launch a pared-down, more stable beta site.  The 2.0 beta will roll live in a minimalist state and then features will begin to reappear (along with the inevitable bug fixes) over the next few days/weeks as things stabilize and get tested in the wild.  Hopville will continue to be 100% free until all the major features planned for the 2.0 release are live, after which it will transition to the “freemium” model as announced previously.  At this point the full feature set is at least a few weeks out, maybe a few months.

Meanwhile, Hopville has to move house.  Starting tonight I’ll post updates about outages and major bugs below for those who’d like to follow along.

• [Latest] July 4 @ 5:30 PM: The stripped-down site is finally live!
• July 4 @ 5 PM: Setting up DB replication, which helps the site stay fast and ensures that there are multiple copies of the data in case of catastrophic failure. If I waited until after the site launched, I’d have to disable it again to start the replication process, so I’m taking care of it now.
• July 4 @ 3 PM: Looking good to switch on the minimalist initial version of the new site later today.  Currently working to migrate all of the brewer profile images to the new service.
• July 3 @ 9:30 PM: Okay, I think I have ironed out everything with the data migration, my suite of test recipes is looking solid, so I’m now prepping to release the initial minimalist 2.0 site tomorrow. I have the whole day off tomorrow…but I’m being extra careful about the migration part of this process.  Since the data model is changing in a few ways, there is no turning back once the new site goes live – that’s why I’ve kept it dark during the transition.  For safekeeping, I will continue to run the “final state” of old website under a private domain so that I can use it to QA any issues and run incremental data fixes as necessary.
• July 2 @ 9 PM: At this point I’m going to run another complete import to help ensure a clean, finalized database before releasing anything to the brewing public.  I’ll need to test that tomorrow once the fresh build is complete, but I won’t have the whole day free as I did today. Sorry folks, but  it’s looking like the site might remain in the dark through much if not all of Tuesday July 3.
• July 2 @ 6 PM: Working out some kinks in the migration script for recipes, making sure all customizations saved under the old codebase are preserved in the new one.
• July 2 @ 4 PM: Data migration looks okay but I’ll continue the QA process to make sure recipes transfer successfully. Also migrating brewer profile images, etc.  A few hours of work still remain.
• July 2 @ 7:30 AM: All brewer information has been fully migrated. Next up: recipes!  That’s the juicy one – it might take several hours to migrate all the recipe data and QA it before switching the site back on.
• July 2 @ 6 AM: The migration script caught a couple data validation problems overnight. That’s good, because it’s enforcing stronger data integrity for version 2.0.  But I’ll have to spend this morning running the final migration incrementally so I can babysit each step to make sure everything gets carried over correctly and completely.
• July 2 @ 1:15 AM: Migrating DB from 1.0 to 2.0. This is a long-running process so I’ll let it complete overnight and then start work again in the morning. No further updates this evening, folks!
• July 2 @ 12:30 AM: Creating final, redundant backups of Hopville’s 1.0 database.
• July 2 @ Midnight: Hopville.com has been disabled.  Off we go…
• July 1 @ 10 PM: Site migration scheduled to begin at midnight PDT tonight.