Credit Rob Sayer

Not everyone knows this, but my first degree is in theatre, specifically theatre lighting design.  Before I ever learned about differential equations, stress analysis, or lift-to-drag ratios, I studied color theory, script analysis, and worked lights for dozens of shows ranging from “A Christmas Carol” to “Evita“.

One of the secrets to making sure a play comes off without a hitch every single night for weeks, months, or even years on end is something called a dimmer check.  Before each performance (even if there was one earlier in the day), the lighting crew chases everyone out of the theatre, and one by one turns on the 50-500 lights over the stage to make sure everything is still working.  The crew checks to make sure the lights come on, that the color filter in front of the light has not faded or burned through, that the light is still pointed at the correct location on the stage.  And, believe it or not, for medium to large shows, there is almost always something that needs to be fixed before each and every performance.  Yes, even when the last performance was just a couple of hours ago.

This attention to detail, and insistence that every time the equipment is turned back on it should be tested, is an essential element to getting live performances right every single time.  It’s even more important when you have just changed something, whether it is to make a repair or an improvement.

I pride myself on how my experience in theatre influences the way I approach live events at Mach 30 and elsewhere.  I always insist on rehearsals, especially when technology is involved (we had two separate technical rehearsals for this year’s Yuri’s Night Party), and I do my own version of the dimmer check for any gear I plan to use during an event.

Mach 30 Hangouts happen each Thursday

But this week, I got cocky, and I made a change (to improve our Google+ page) without running through any tests afterwards, and this change broke our ability to host On-Air Hangouts (on a week when we had an important one scheduled).  #Oops.  Apparently, linking one’s YouTube channel to a Google+ Page causes some squirrelly behavior with On-Air Hangouts.  Behavior we did not notice until during and after this week’s OSHW Documentation Jam Round Table Hangout, which not only led us to starting twenty minutes late, it also appears to have prevented the video from becoming sync’ed over to our YouTube Channel (which is too bad, I think our panelists and guests had some really great things to say, and I am sorry we won’t be able to share them with the Open Source Hardware Community).

So, that’s the bad news.  Of course, the good news is no one died or was injured from my failure to properly test things.  But, Mach 30’s work is building to a day when people’s lives will be on the line, so it is important to recognize small failures so we can learn from them.  In this case, the lessons are

1. Remember to test everything associated with a system after making changes to the system (there is likely a balance of risk vs reward to be struck, but clearly the key features of a system should be checked when significant changes are made)
2. Mach 30 needs to identify the core features we are using Google+ for (such as On-Air Hangouts) and create a test plan (or dimmer check) to be run when changes are made to our Google+ infrastructure, either because Google upgrades a feature or because we turn on an existing one we had not been using.

And, in the mean time, I will look into trying to recover our lost hangout video, and schedule the already discussed second round table hangout (after I have fixed our YouTube settings).

ad astra per civitatem