Saturday, I took my daughters to the kids day/open house down at our local PBS station. Getting past the claustrophobia of jamming hundreds of kids, strollers, and overwrought parents into the narrow hallways and anechoic studios, it was great kidly fun–lots of booths, sing-alongs, and face painting. On our way out, we noticed a smart-looking medium duty truck painted like a firetruck, pulling a trailer packed to the gills with equipment. Inside were two “simulator” booths where two robust gentlemen were showing how they train police officers and firemen to drive their vehicles in all sorts of conditions and situations.
Watching a ten-year-old attempt to pilot a full-sized firetruck in a city (protip: make wide turns), it struck me:
There is effectively no simulation training for Software Developers.
That’s right: For the pilot of your 777, there’s a simulator. She’s required to log so many hours in life-threatening situations so that she can respond with confidence when the big crisis occurs. You’re trusting her with your life, and the only thing telling you she’s worthy is her logbook of hours and her simulation record. Apparently, there’s something similar for firemen and police officers.
However, there’s nothing like that for software developers, and I’d argue it’s sorely missed.
First, professional software development gives you tunnel vision. Software development is problem solving, and human beings have two methods to problem solve: Autonomous pattern matching and deep cognition. Autonomous pattern matching is when you reach back into your cache of experience and retrieve THE WAY TO DO IT. Otherwise, you really have to think.
Let me show you an example. Ever see anyone solve one of those horseshoes with ring puzzles?
If you’ve never solved it before, your cache is empty. You scan the thing, handle it, feel around it looking for a way out. Your pupils are dilated, your heart rate increases, and your cortex is consuming extra oxygen–you’re really thinking. This is deep cognition.
Now, if you *have* seen one before, none of that physiology happens. You grab the shoes, twist them just so, and out pops the ring. Easy! Your body barely reacts at all, and studies confirm your brain uses no extra oxygen. Moreover, if you’ve done it enough times, you’ll have a hard time even teaching someone how to do it.
All that to say: Once you’ve been developing software for awhile, many, many things move into that autonomous pattern matching system. You become expert, but you also become blind. You crutch on your autonomous system and your deep cognition atrophies. (This is why my wife derides my omnipresent internet perusal as fluff. Rightly, she concludes that I’m just absorbing more information to pattern match, without thinking critically.)
Why does this matter? Because it costs money, and it can also cost lives. Psychologists conclude that our deep cognition system, while immensely powerful, is fundamentally lazy. It costs energy, and we’re wired to avoid using it if we can. That’s just our biology. When presented with problems almost like one in our cache, we’ll rationalize that it “must be that” and go with our gut.
Our gut is sometimes wrong. Some examples:
- That customer issue that looks like the 15 other customer issues with the same log statements. Did you test the fundamentals and reproduce the issue, or just send them the patch and pray it fixed it?
- That sneaky concurrency bug. Hey, all the code you’ve shipped worked that way, so why try and fix it? (This is also the Gambler’s fallacy, another product of our autonomous pattern matching system)
- How many times have you just stared at a stack trace, hex trace, or other telemetry and had a flash of insight as to what it “must be.” That’s a combination of your autonomous system jumping up and your cognition crying uncle.
That’s why simulators or trial scenarios are so useful:
- They draw on common experience. Many firemen have driven over curbs, failed to stop because of poor weather conditions, and been unable to negotiate tight quarters in firetrucks. The simulator lets those play-back for every newbie.
- They’re non-destructive. Screw-up a crosswind approach in a 777, and 400+ people will die. Do so in a simulator, and you learn without killing anyone.
- They provide consistency. Everyone who’s been through the simulator has been through the ‘Kobiashi Maru’ or whatever that equivalent is for your org. That builds peer confidence in the capability of the organization.
- They provide “impossible” situations. It’s unlikely an aircraft will have 3 major emergencies simultaneously without a SAM involved. However, one can simulate those trivially in a simulator, and help a pilot understand how to balance the workload to a good result.
The last is the big win for me. Critical customer issues happen, and they’re high-stress, and it’s difficult to balance your own physiology and the matter at hand. The insidious autonomous system looms, waiting to give you the “almost right” answer without you thinking.
Here’s the thing: I think we have the technology to build simulators right now, without extraordinary effort. Here’s what I see:
- Use Vagrant and Chef to codify the machine configurations in question. Maybe it’s a downlevel CentOS box with a wonky configuration. Let’s say the http_proxy isn’t configured and you want to show the trainee how the system behaves. That’s trivial to codify and check-in to your ‘simulator’. Destroying the box is no consequence; one can delete and rebuild with two commands.
- Your ‘development box’ is a set of tools similarly checked-in to SCM, with vagrant and chef creating a vm (either locally, on a VM server in-house, or on IaaS). Maybe you want to teach someone how to attach remote debug to a running JVM that’s dying because of MaxPermGen config issues. Easy enough.
- You can use ssh or the equivalent to monitor and remote control the development box, much like a simulator instructor can turn on rain, wind, or other environmental factors. Failing hard drive? Flaky Wifi? Throw the student some curve-balls, and make that cognitive system work.
Katas can help. Learning new languages and paradigms can help.
However, nothing would be better than a real scenario simulator.