Archive for August, 2012

Hopville 2.0 Migration – Take 2 (August 16 – 19, 2012)

The main outage is expected to last most (if not all) of August 16th, but here you’ll find regular updates on what’s happening behind the scenes.  All times listed are PDT (U.S. West Coast).

  • [Latest] Aug 16 @ 11:30pm: Stripped-down site is now LIVE.  Basic features will get added back in over the course of tomorrow (and through the weekend) as the new systems get tested for stability and performance.  G’night folks!
  • Aug 16 @ 8:00pm: I would like to take this moment to (again) declare that Internet Explorer is vile software and the scourge of the internet.
  • Aug 16 @ 7:00pm: Image migration completed successfully, which means all information from the old website has now been carried over and rejiggered for the new site. Next step is to fully test the site in its new home now that it’s running with a complete data set.
  • Aug 16 @ 6:30pm: Image migration is just about complete. While the processing script was adding roughly 100,000 newly generated image files to the new world, I caught a little more sleep so I can be more lucid when I flip the switch later tonight.
  • Aug 16 @ 4:00pm: About half way through processing the brewer profile images now.
  • Aug 16 @ 3:00pm: One aspect of today’s major migration is a changeover from a cheap shared hosting account to expensive, more robust cloud servers. This provides much more insight into the health and performance of the website. The image processing script puts really high load both on server memory and on network I/O, so I’m already getting very interesting performance metrics that I would have been blind to under the old Hopville.   Cool stuff.
  • Aug 16 @ 1:00pm: Okay, image server config is fixed up, processing profile images again – this time for keeps. Have to take thousands of images, generate copies of each in a variety of sizes that will be used on the site, and then upload all the variations to the new image server – so this is another step that takes a while.
  • Aug 16 @ 12:30pm: Found a little problem with the image server configuration so I’ll have to make a few tweaks to the processing script to get all the profile images migrated successfully.
  • Aug 16 @ 11:00am: Now migrating all the brewer profile images from the old site to make them available to the new image service. Copying them to the new server now, then the next step is to run a processing script over them.
  • Aug 16 @ 10:00am: Haven’t found any issues with the DB migration.  Woo-hoo!  Now configuring the “gone fishing” pages so that I can update the DNS for This means the website officially moves to a new address.  The same “down for maintenance” page will display, but it will get served from the new server instead of from the old server.
  • Aug 16 @ 9:30am: Database migration complete. I’ll have to run a few quality assurance passes over the new database.
  • Aug 16 @ 9:00am: Database migration is progressing smoothly. It’s looking like the “practice” launch in July helped iron out many of the kinks in this, the most sensitive of all the transfer steps.
  • Aug 16 @ 1:30am: Okay, successfully generated a clean source on which the DB migration will be based.  DB migration will take hours and require consistent monitoring and QA, plus I’ll need some sleep, so updates will be irregular through early morning.
  • Aug 16 @ 1:15am: Now backing up the Hopville 1.0 software, as much for amusement as for safe-keeping.
  • Aug 16 @ 12:30am: Redundant database backups complete. Starting brewer profile image backups. This will take a while.
  • Aug 16 @ 12:15am: Initiating master backups of the entire site before migration begins.
  • Aug 16 @ 12:01am: Old site is now switched off. Let the games begin.
  • Aug 15 @ 11:50pm: Pre launch systems check okay.  Going forward with the outage.
  • Aug 15 @ 7:20pm: Outage scheduled to begin around midnight PDT.

Comments (19)