Archive for Uncategorized

Hopville 2.0 Migration – Take 2 (August 16 – 19, 2012)

The main outage is expected to last most (if not all) of August 16th, but here you’ll find regular updates on what’s happening behind the scenes.  All times listed are PDT (U.S. West Coast).

  • [Latest] Aug 16 @ 11:30pm: Stripped-down site is now LIVE.  Basic features will get added back in over the course of tomorrow (and through the weekend) as the new systems get tested for stability and performance.  G’night folks!
  • Aug 16 @ 8:00pm: I would like to take this moment to (again) declare that Internet Explorer is vile software and the scourge of the internet.
  • Aug 16 @ 7:00pm: Image migration completed successfully, which means all information from the old website has now been carried over and rejiggered for the new site. Next step is to fully test the site in its new home now that it’s running with a complete data set.
  • Aug 16 @ 6:30pm: Image migration is just about complete. While the processing script was adding roughly 100,000 newly generated image files to the new world, I caught a little more sleep so I can be more lucid when I flip the switch later tonight.
  • Aug 16 @ 4:00pm: About half way through processing the brewer profile images now.
  • Aug 16 @ 3:00pm: One aspect of today’s major migration is a changeover from a cheap shared hosting account to expensive, more robust cloud servers. This provides much more insight into the health and performance of the website. The image processing script puts really high load both on server memory and on network I/O, so I’m already getting very interesting performance metrics that I would have been blind to under the old Hopville.   Cool stuff.
  • Aug 16 @ 1:00pm: Okay, image server config is fixed up, processing profile images again – this time for keeps. Have to take thousands of images, generate copies of each in a variety of sizes that will be used on the site, and then upload all the variations to the new image server – so this is another step that takes a while.
  • Aug 16 @ 12:30pm: Found a little problem with the image server configuration so I’ll have to make a few tweaks to the processing script to get all the profile images migrated successfully.
  • Aug 16 @ 11:00am: Now migrating all the brewer profile images from the old site to make them available to the new image service. Copying them to the new server now, then the next step is to run a processing script over them.
  • Aug 16 @ 10:00am: Haven’t found any issues with the DB migration.  Woo-hoo!  Now configuring the “gone fishing” pages so that I can update the DNS for This means the website officially moves to a new address.  The same “down for maintenance” page will display, but it will get served from the new server instead of from the old server.
  • Aug 16 @ 9:30am: Database migration complete. I’ll have to run a few quality assurance passes over the new database.
  • Aug 16 @ 9:00am: Database migration is progressing smoothly. It’s looking like the “practice” launch in July helped iron out many of the kinks in this, the most sensitive of all the transfer steps.
  • Aug 16 @ 1:30am: Okay, successfully generated a clean source on which the DB migration will be based.  DB migration will take hours and require consistent monitoring and QA, plus I’ll need some sleep, so updates will be irregular through early morning.
  • Aug 16 @ 1:15am: Now backing up the Hopville 1.0 software, as much for amusement as for safe-keeping.
  • Aug 16 @ 12:30am: Redundant database backups complete. Starting brewer profile image backups. This will take a while.
  • Aug 16 @ 12:15am: Initiating master backups of the entire site before migration begins.
  • Aug 16 @ 12:01am: Old site is now switched off. Let the games begin.
  • Aug 15 @ 11:50pm: Pre launch systems check okay.  Going forward with the outage.
  • Aug 15 @ 7:20pm: Outage scheduled to begin around midnight PDT.

Comments (19)

Growing Pains of Going Pro

Since the initial release of the Hopville 2.0 beta will have a stripped-down feature set – and I mean, a really stripped down feature set – I thought it might be good to share some of the details of what 2.0 represents under the covers.

Hopville grew to 100,000 recipes as one of several side projects, without having serious focus given to it.  With Hopville 2.0 I’m beginning to give it real focus, getting it ready to grow to its next milestone of 1,000,000 recipes. First step was to overhaul its infrastructure.  Here’s what happened this week:

  • New web host: Since its inception, Hopville has run on an affordable hosting account through Site5.  While they’ve served Hopville well, the nature of that hosting service (cheap hosting on shared servers without system-level access) is too limited for a site operating at professional capacity. The tallest hurdle jumped this week was the migration from their service to cloud servers hosted by Rackspace, with whom I’ve had great experience in a few startups where I’ve worked.
  • New web framework version: Hopville was most recently running Rails 2.3, but the version released this week represents an upgrade to Rails 3.2.
  • New codebase: Due to some delicious tech tweaks introduced into Rails as of its 3.1 version, plus a completely new approach to how Beer Calculus works, I took the opportunity to overhaul the codebase, rewriting nearly everything from scratch.  (Might’ve bit off more than I can chew, frankly, as this project ran far longer than my original estimates.)
  • New database: The site still runs on the open source MySQL database, but for 2.0 it has been restructured and expanded.  Where it took about 20 tables to run the old Hopville, it’s taking about 30 (so far) to run the new service.  One side effect is that folks may have to reset their passwords as the new server and framework use stronger encryption for passwords.  I’ve also isolated the database onto its own server which will help the site to scale and eventually facilitate things like a public API.  You’ll start seeing the reasoning behind the restructuring (in the form of new features) soon…
  • New image service: One of the embarrassing historical aspects of the old site was how images were handled.  I thought to myself when Hopville launched, “since nobody is using this site, I don’t have to optimize its images in any way – it doesn’t matter how huge and slow the pages are with zero traffic” Then I proceeded to never upgrade the service after the site grew to have traffic.  I let the browser resize images into thumbnails, etc – if you’re a web developer you know that this is an absolute no-no.  Hopville 2 will have a lot more image functionality (eventually).  Accordingly, with this launch images are finally processed and handled correctly, as well as being stored on Amazon’s cloud, which should speed up some pages quite a bit.

July will be a bit bumpy since Hopville has no QA department and does testing in the wild.  I’ve also lucked into an unfortunately-timed but also all-expenses-paid vacation to Alaska, so as of Friday I’ll be mostly offline for a whole week.  Sorry folks!  Y’all will notice a distinct lack of features during the initial beta testing, as I removed lots of features to focus on the systems migration.  But you’ll just have to trust me that I can see far into the future that was enabled by the Great Migration of Hopville 2.0.  And what I can see out there on the horizon is damn exciting.  The summer of 2012 is the awkward puberty that Hopville has to transition through on its way to becoming an adult.  Stay tuned…

Final recipe count for Hopville 1.0: 118,222

Leave a Comment

Hopville 2.0 Migration (Week of July 2, 2012)

The 2.0 project was a major undertaking, involving a complete rewrite of the application and a restructuring of the database powering the site.  (Don’t worry, all your data will be preserved.)  It’s also the time when the site graduates from a cheap hosting provider to a professional-grade, not-so-cheap one.

So, the migration is quite a juggling act.  With so many technical hurdles to worry about, the website feature set has been cut back dramatically to launch a pared-down, more stable beta site.  The 2.0 beta will roll live in a minimalist state and then features will begin to reappear (along with the inevitable bug fixes) over the next few days/weeks as things stabilize and get tested in the wild.  Hopville will continue to be 100% free until all the major features planned for the 2.0 release are live, after which it will transition to the “freemium” model as announced previously.  At this point the full feature set is at least a few weeks out, maybe a few months.

Meanwhile, Hopville has to move house.  Starting tonight I’ll post updates about outages and major bugs below for those who’d like to follow along.

  • [Latest] July 4 @ 5:30 PM: The stripped-down site is finally live!
  • July 4 @ 5 PM: Setting up DB replication, which helps the site stay fast and ensures that there are multiple copies of the data in case of catastrophic failure. If I waited until after the site launched, I’d have to disable it again to start the replication process, so I’m taking care of it now.
  • July 4 @ 3 PM: Looking good to switch on the minimalist initial version of the new site later today.  Currently working to migrate all of the brewer profile images to the new service.
  • July 3 @ 9:30 PM: Okay, I think I have ironed out everything with the data migration, my suite of test recipes is looking solid, so I’m now prepping to release the initial minimalist 2.0 site tomorrow. I have the whole day off tomorrow…but I’m being extra careful about the migration part of this process.  Since the data model is changing in a few ways, there is no turning back once the new site goes live – that’s why I’ve kept it dark during the transition.  For safekeeping, I will continue to run the “final state” of old website under a private domain so that I can use it to QA any issues and run incremental data fixes as necessary.
  • July 2 @ 9 PM: At this point I’m going to run another complete import to help ensure a clean, finalized database before releasing anything to the brewing public.  I’ll need to test that tomorrow once the fresh build is complete, but I won’t have the whole day free as I did today. Sorry folks, but  it’s looking like the site might remain in the dark through much if not all of Tuesday July 3.
  • July 2 @ 6 PM: Working out some kinks in the migration script for recipes, making sure all customizations saved under the old codebase are preserved in the new one.
  • July 2 @ 4 PM: Data migration looks okay but I’ll continue the QA process to make sure recipes transfer successfully. Also migrating brewer profile images, etc.  A few hours of work still remain.
  • July 2 @ 7:30 AM: All brewer information has been fully migrated. Next up: recipes!  That’s the juicy one – it might take several hours to migrate all the recipe data and QA it before switching the site back on.
  • July 2 @ 6 AM: The migration script caught a couple data validation problems overnight. That’s good, because it’s enforcing stronger data integrity for version 2.0.  But I’ll have to spend this morning running the final migration incrementally so I can babysit each step to make sure everything gets carried over correctly and completely.
  • July 2 @ 1:15 AM: Migrating DB from 1.0 to 2.0. This is a long-running process so I’ll let it complete overnight and then start work again in the morning. No further updates this evening, folks!
  • July 2 @ 12:30 AM: Creating final, redundant backups of Hopville’s 1.0 database.
  • July 2 @ Midnight: has been disabled.  Off we go…
  • July 1 @ 10 PM: Site migration scheduled to begin at midnight PDT tonight.

Comments (6)

Countdown to Hopville 2.0

I’m asking users of Hopville to fill in a short (10 question) survey on the future of the site. If you haven’t filled in the survey yet, feel free to do so over here.

I created the survey mostly to get an idea how people might respond to Hopville changing over to the freemium model, where the site remains free up to a certain point, but then requires a paid membership (a la Flickr, Pandora, Dropbox, etc).  I’m working on the next major version of the site right now and plan to use the freemium business model to help usher Hopville into its next phase. What I’m currently calling the next phase is, “Holy crap, this website is really fun to work on and thousands of people rely on it so how ’bout I double down on it and turn it into something awesome.”  That’s too long of a phrase to type all the time so instead I’m using geek speak: Hopville 2.0.  Hopville 1.0 is the site you see now, the one that will get us to the milestone of one hundred thousand homebrew recipes.  Hopville 2.0 is the kind of site that will get us to a million.  Think about that: one million homebrew recipes.  I’m already building the renovated website, working toward a self-imposed deadline, and so far I’ve stayed on track.  If all goes as planned, the new site will launch by late Spring.  Here’s the logo, designed by one of my favorite rock poster artists, Frida Clements:

Hopville logo

I’ve gotten a significant amount of feedback from the survey, and the results confirmed a few of my basic assumptions:

  1. Lots of brewers would be willing to pay for Hopville’s services, as long as those services continue to be actively maintained and improved.
  2. Lots of other folks only use Hopville because it’s free.
  3. Lots of folks want a mobile/tablet app for Hopville ASAP.

Hopville 2.0 will cater to the #1 crowd, will mostly accommodate the #2 crowd (i.e. Beer Calculus and Hopville’s recipes will remain public and free), and will lay the foundation for #3 by establishing an API that a future Hopville app (or third-party apps that any enterprising software developers could build) could use to synchronize with Hopville’s data.

At launch, brewers who are already using Hopville at a “Pro” level of activity will be granted a free evaluation period of at least three months, even longer for long-time users of the site. I think most people who like the current site will love the new one, but as a crotchety old seasoned web developer, I know that a website can’t please all the people all the time.  For those who try out the new site and would rather not subscribe, I’ll continue to provide functionality to export all their recipe data as BeerXML – that way they can easily migrate their recipes over to their brewing software of choice.  For those who’ve already supported the site with voluntary donations – the finest people in the world – I’ll provide extended free evaluation periods that correspond to their level of donation.

Feel free to contact me with any questions or comments you may have, either here or directly via email.

Current recipe count: 87,890

Comments (7)


Okay folks, Hopville and Beer Calculus are likely to be a little drunk and out of sorts this weekend.  Or maybe it’s me that will be drunk and out of sorts, and the site will be merely rough around the edges.  I decided to roll out the current state of the major revision I described a bit in my last post.  But this is definitely a case where I’m pushing it live in order to get more eyeballs on it and to light a fire under my butt to finish the parts that aren’t really ready for prime time.  Bear with me through a slew of changes, probably some times when the site is a little broken, and please report any bugs – I plan to focus all my (limited) spare time on Hopville this week, so bugs should get fixed pretty quickly…within a day or so.  The ingredient “info” links and ingredient search will be back on the site ASAP – just have to get them working properly with the new Beer Calculus code.

Current recipe count: 14,960

Comments (12)

Defaulting to Tinseth

It’s been many moons since I’ve updated anything on Hopville.  I blame the old job and the quitting of the old job and the traveling all summer and the search for a new job.  I’ve got excuses, see.  I’ve been lax, so I thought I’d announce this update.  Even though it’s small, it can have a large effect on recipe design. Previously, the default IBU calculation for Beer Calculus was based on an average of a few popular formulas. It did four calculations (Garetz, Rager, Tinseth, and the legacy Hopville calc) and averaged them together.  I chose to blend a few conflicting numbers together instead of committing to a single one by default. That neutral position tended to cause some confusion among both types of brewers: those who cared which formula was in use, but didn’t know you could change it, and those who didn’t care at all. Plus, the only indication that a formula selection was being made was a subtle message “avg” near the IBU result – pretty vague about what was happening behind the scenes. Recipes now default to the Tinseth formula. Hopefully this will satisfy those who prefer this formula, and also clarify the default calculation to folks who don’t really care.

Comments (1)

Little feature weekend

Checked off a few random things on the ever-growing TODO list for Hopville this weekend.

  • Added BJCP style validation to Beer Calculus.  As you work on recipes in the calculator, you’ll see warnings whenever the calculations seem to put your recipe out of bounds for the chosen beer style.  These warnings can be ignored entirely, but at least the information is now available where it would be most useful.
  • Added a random recipe link to the recipe navigation menu.  Self-explanatory.  And fun!  Try it!
  • Moved Beer Calculus so that it lives under the domain at  This shouldn’t have any effect on users, but it’ll help with Hopville’s “Google juice”, keep me from having to jump through hoops to maintain state between two domains, and make it more apparent that Hopville is there to support Beer Calculus.  Since the calculator preceded Hopville by so many years, lots of folks don’t even know they can save their BC recipes yet!

Current recipe count: 2026

Comments (11)

Older Posts »