A Blizzard Postmortem, Part 1: The Story
You may have noticed, if you read Control-Alt-Backspace regularly or if you pay close attention to the dates on my posts, that there was no scheduled post last Monday. That’s because, last Monday when I was supposed to be writing and publishing, I was stranded 45 minutes away from home with nothing but the clothes on my back (and, hilariously, my favorite hairbrush, which I had chucked in my coat pocket earlier in the day and forgotten about).
In IT, many organizations conduct postmortem analyses after their services go down, their systems are hacked, or they experience other kinds of disruptions or accidents. (This practice is particularly associated with the DevOps methodology.) The goal is not to find out who screwed up and fire them, or find a way to blame the problem on someone else, but to understand all the factors that led to the issue and determine some concrete actions that can be taken to prevent the same issue from happening again.
Like many IT and software development practices, this can be useful in other disciplines as well. This week and next week, as you might have presumed from the title, I’m going to apply it to my experience this weekend. The first step, unsurprisingly, is to explain what happened. That’s going to take a whole post of its own, because it’s a long and complicated story. Next week, we’ll do the actual analysis.
Note: Since this is a true story and we’re dealing with small towns where people might be identifiable, most people’s names have been changed for privacy.
Lead-in
Over the past few weeks, I’ve been rehearsing for a choir concert with the Austin Symphony. That’s Austin, MN, not Austin, TX; it’s a small town about 45 minutes southeast of Owatonna, where I live, best known for being the headquarters of Hormel, the company that makes Spam. (They even have a Spam Museum.) Here’s what the area looks like, including Owatonna and Austin:
Note the central triangle of highways: I-35, I-90, and US-218. These will be getting a lot of airtime in this story.
This February, southern Minnesota has been getting a lot of snow. We’re used to snow around here, as you might expect, but we don’t usually get this much of it, and certainly not all at once. This February is now the snowiest February on record and the second-snowiest month on record for the area. It has seemed like every time I have to go down to Austin for rehearsal, it’s snowing, or it just snowed yesterday and isn’t clear yet, or else it’s just incredibly foggy and slippery. One of my coworkers even joked that I needed to stop going to Austin so we would stop getting so much snow.
The concert was supposed to be last Sunday, but a blizzard warning was issued from Saturday night through Sunday night. That sounds pretty serious, but we’ve been having so many winter storm warnings and blizzard warnings over the past month that nobody thought much of it. It doesn’t help that most of the storms are pretty boring in the grand scheme of things (sure, it’s not too fun being out there, and you have to drive 45 mph on the freeway for a day or two if you don’t want to end up in the ditch, but that’s just a normal February up here). So the concert wasn’t canceled ahead of time, and I figured that I had best go down and stay the night Saturday so I could perform at midday Sunday.
This I duly did; I got a hotel room on the north side of Austin, brought a briefcase full of books and my new MacBook I was screwing around with (so there will soon be some more Mac expertise and screenshots on Control-Alt-Backspace!), and hung out in my room that night.
Sunday morning: Deepening
I stayed up late Saturday night and got up around eight-thirty Sunday morning, then took a shower and got some “free” breakfast at the hotel (you know, where they add $5 to your room fee and pay $3 for the food but make the cost invisible so everyone’s happy). The snow had piled on pretty thick during the night. Hardly any new snow was still falling, but as happens in blizzards, the wind was blowing enough to make it still look snowy. At around ten, I got an email that the concert was canceled, which was a bit disappointing, but we wouldn’t have had much of an audience anyway, so it wasn’t a surprise. So I had nothing further to do in Austin, I just couldn’t get home yet. Check-out was at eleven, so I cleaned up and headed out to my car a little after that. The blizzard warning was still on until six, so I was planning to sit at a nearby coffee shop for a while and figure out where to go from there. However, when I got out, conditions were looking surprisingly good, so as we do in Minnesota, I figured I would try the roads and turn back if they were too bad. Oops.
Sunday midday: Stuck on 218
The roads in town were not well-plowed when I started out, but the visibility was good, and the highways typically get salted and get more attention from the plows, since they have more traffic, so it still seemed plausible that the highway would be decent. Indeed, when I got over to it, the surface was acceptable. However, as I trundled out of town at around 30 mph in a 55, the visibility rapidly worsened. I had forgotten (or, likely, subconsciously ignored in my eagerness not to be trapped in a town I don’t know well during a blizzard) that this stretch of 218 is notorious for having nasty visibility and drifts at places; since it mostly runs through open fields, there’s little to block the wind. Moreover, one of the things that makes blizzards dangerous is that conditions can change very rapidly. One moment things look fine, and the next the wind picks up and you’re in a total whiteout (if you’ve never been in snow nasty enough to cause a whiteout, the amount of snow in the air and the sunlight combine so that you literally can’t see anything but white except for extremely dark objects, even feet in front of you). So, seeing that things were looking worse and worse, and after steering around someone who had gotten stuck on the shoulder of the roadway, I decided to turn back, exactly as I had planned. I found a safe spot, did a three-point turn, and headed back the other way feeling good about my choice.
About a minute later, with conditions still worsening, I drove right into a three-foot snowdrift sitting in the middle of the lane. It was shallow on my side and steep further ahead, so that with light snow covering the whole highway it was virtually invisible. I was only going about 20 mph and could easily have stopped and steered around if I had seen it, but I had no warning at all. The first sign I got was the crunching sound of heavy snow being packed against the undercarriage of my car. Don’t just take my word for it – here’s my dashcam video:
Usually when people get stuck in snowstorms, it’s because they slide off the highway, but here I was stuck right in the middle of the road. I put on my hazard flashers, got out, pulled my folding shovel out of the back, and tried to dig myself out. I have a Subaru Forester with all-wheel drive, so I typically have solid traction in the winter and can usually drive right through reasonable amounts of snow standing on the roadway. The problem was that the wheels were way up in the snowdrift and the car was no longer fully resting on the wheels – when I stepped on the gas, the tires caught fine, but they couldn’t move the car anywhere.
I was wearing a heavy down coat with a down-lined hood and a wool scarf and hat and gloves, so I wasn’t cold, but every time the wind came up, it would become virtually impossible to see, and if I had to take my gloves off to handle something like the pins that keep the shovel together, my fingers would go numb within seconds. The driving wind would melt snow onto my glasses and make my vision even worse, so I took them off and put them on my passenger seat (this will become very important later).
A man in a snowplow who spoke barely any English came by and tried to clear the snow from in front of my car, then offered me a tow chain, but even after clearing much of the snow away, the snow packed against the front of the car prevented me from reaching anywhere to attach the chain besides the bumper (which would of course have led to a stuck car with a missing bumper) – not to mention that it would have been nearly impossible to pull the car over the top of the drift anyway. We would have had to pull it backwards, which was impractical given the road conditions. So that was a no-go as well.
Sunday midday: Leaving the car behind
After maybe twenty minutes, I gave up on trying to get myself out and sat in my car and prepared to wait a while. I had a mostly full tank of gas (I never drive with less than half a tank in the winter for precisely this reason) and figured either the storm would clear in a few hours – as it was supposed to – and someone would be able to clear us out, or someone would come by and pick me up. As it happened, only a couple minutes later, several extremely large snowplows operated by the state came through in a caravan, with some cars and the county sheriff coming behind. The sheriff offered me a lift, and I promptly locked my car, went over, and got in. Big mistake.
The mistake was not riding with the sheriff, who was very businesslike but nice enough. Clearly the kind of guy who’s seen too many idiots stranded on the highway in his career to be upset at them anymore. The mistake was that I got in the car when the sheriff told me to without thinking about anything else. Like whether I could cram some of my luggage in the car on my lap, or whether it might be helpful to at least bring my smartphone, which I had left in the car. Or, you know, my eyeglasses which I had laid on the front seat while shoveling, without which I can’t see clearly more than eighteen inches from my face.
There was one other guy who had been stranded and picked up riding with me. I couldn’t figure out where to connect my seat belt, and was told not to worry about it (granted, we were going somewhere between five and ten miles an hour the whole way, since the plows were leading the way, but it was funny to be told by the police not to wear my seat belt). I’m fairly confident that we also stopped and waited right on the railroad tracks at one point, though admittedly it was not easy to see where we were. After somewhere between ten and fifteen minutes, during which time I tried to figure out what the heck I was going to do now and noticed that I had forgotten both my glasses and my phone, we made it back the two miles or so to the north edge of town and were dropped off at Wal-Mart.
Sunday midday: Stranded at Wal-Mart
Wal-Mart isn’t the nicest place to be stuck, but given the choice, it’s a pretty practical one, even when half the staff can’t make it in and the electronics guy is managing the grocery section and takes ten minutes to come when you ring the pager – mainly because you can buy virtually anything you’re missing. I have a personal saying:
Almost any problem you encounter when traveling can be solved with a credit card and a good attitude.
It was time to exercise the credit card here. (Final bill for the weekend: $520.71.) I had brought a lunch with me, but now it was freezing solid in my car out on 218, so I bought some M&Ms, a can of Pringles, and a bottle of sparkling water, and at least got a few calories. My clothes were also rather wet at this point, since I had been in the snow trying to dig my car out, and they needed an hour or so to dry out before I stopped feeling cold. (Getting hypothermia in relatively mild temperatures from having wet clothes is a real risk. Not likely inside a heated store, but it was still a strong enough effect to get me shivering occasionally.)
I was astounded how many people had come out to shop at Wal-Mart during the blizzard. And not just for milk or a flashlight or something; there were people looking at clothes, buying gift cards, whatever. Not only that, but the store phones were ringing constantly, presumably from people who wanted to know if Wal-Mart was open so they could come out to shop at Wal-Mart during the blizzard without having to go out in the blizzard if they couldn’t. I don’t think I ever saw one of the phones answered; the staff were quite busy enough as it was, and they also found it ridiculous that so many people were calling.
After enhancing my good attitude by getting some food and warming up so I was ready to solve my problem, my first order of business was to try to get out of Wal-Mart. I had remembered during my ride with the sheriff that the director of the choir I was singing in, Sigrid, had offered in advance to let people stay at her place if the weather was bad, so my plan was to get in touch with her. I could have tried to find my way back to the hotel and checked in again, but if there’s a time to take advantage of hospitality, it’s when you’re alone, don’t have any luggage, and have nothing to do and don’t know when you’ll be getting out. (Oh, also, the cable at the hotel had been out Saturday night. It was supposed to have been fixed in the late evening, but if it was still out I would really have had nothing to do, not even watch dumb TV.)
It was about here that I realized how few phone numbers I know nowadays. I had no problem with my parents’ phone number, which has been the same since I was six; I could remember my own three phone numbers; and I could remember plenty of numbers back in my hometown from when I actually still needed to know them occasionally. But none of those folks live in Minnesota, and people out of state aren’t usually much help in a blizzard. I didn’t even remember Sigrid’s last name, having just met her a couple of weeks prior and only talked to her for a few minutes. What I really needed was someone who knew Sigrid so I could get hold of her number. What I had was some out-of-state phone numbers and a laminated card with the phone numbers of my team at work, which I had shoved in my wallet before leaving just in case I couldn’t get home on Sunday and needed to tell someone where I was. That was a start, anyway.
I asked a young man on the Wal-Mart staff for help, and he graciously let me borrow his iPhone. I tried googling around for phone numbers, but online directory search services have gotten remarkably bad these days; I find anytime I actually find a hit on someone, the site asks me to pay some ridiculous fee to see their number! I studied my card and started trying the people who still had landline numbers listed on the card. I chose the people with landlines because I figured they would have phone books in which they could try to find other people they didn’t know. The second person on that short list answered, and I explained my predicament and got several phone numbers one degree closer to Sigrid. The second person on that list answered, and I got one degree closer.
Unfortunately, my search space here narrowed to one person, and she wasn’t available even when I rang her cell phone repeatedly. I left a message, but not having a phone, all I could give was the phone number of Wal-Mart to call me back at. (The young man who helped me didn’t understand why I wanted the phone number of the store, but did help me find it. I guess with everyone having a cell phone, the concept of paging someone in a public place is foreign by now. Like pay phones, you don’t need it anymore until that one crazy moment when everything is out of sorts and you really need it.)
As mentioned before, though, the staff weren’t really answering the phones, so I couldn’t count on anyone being able to page me. Besides, I didn’t want to keep borrowing random people’s cell phones, and I figured that I would need to deal with other business where having a callback number would be helpful before I got back to my car, so I decided to go buy one of those $20 prepaid phones that con men buy with cash to get an untraceable phone number and snap in half and chuck in a dumpster when they’re done.
Over at the electronics section, two young men, Dan and Penguin (obviously a nickname), had come to attempt to cash a check and buy an Amazon gift card and were waiting for the electronics guy to come over and help them. After ten minutes, he made it over (as alluded to previously, he was managing the grocery section today due to short staffing), told them that Wal-Mart doesn’t carry any Amazon products, and helped me find the cheapest phone and airtime card they had (total bill: $44 for a fully functional phone that lets you instantly talk to anyone in the country for 60 minutes – isn’t this world amazing?). We had to walk up to the front of the store to pay for it, since they hadn’t opened the registers in the electronics section that morning; then we had to walk back to the electronics section in the back again so he could activate the phone for me, as, hilariously, you need an already-functional phone to activate your phone. This took about ten minutes of swearing at the automated system (and he obviously did this regularly, so it wasn’t an experience thing).
So now I had a phone. At this point, Sigrid’s last name suddenly came to me, so I asked the staff for help again and we found that there was a phone book at the customer service desk. (Remember those?) And, amazingly, Sigrid was in the phone book, and she answered when I called on my new prepaid phone. I was welcome to come over, here was her address, and there were two other stranded people there to hang out with, but they weren’t plowed out yet so nobody could come pick me up. I dropped over to the Yellow Pages (remember those?) and found a taxi service, but unsurprisingly didn’t get an answer, so I sat down on one of those metal benches they have by the doors at Wal-Mart, ate some more M&M’s, and figured I would try again in a bit, maybe after the storm died down a little. It was starting to look unlikely I would make it home today as I had originally planned; even assuming my car was towed by the end of the day, which at this point I was still expecting to happen seeing as it was sitting in the middle of the lane on a major highway, the roads might not be safe to drive on until the morning. I certainly wasn’t about to get stuck again!
While I was waiting, Dan and Penguin showed up again, and Penguin spent the next twenty minutes doing “doughnuts” at 2 mph and otherwise screwing around on one of those electric wheelchairs with a shopping basket. I’m sure on any normal day someone would have told him to stop or kicked him out, but today it was really pretty funny. We talked for a while; apparently they had planned to drive to Wal-Mart in the blizzard, been unable to get out, and thus decided to walk to Wal-Mart in the blizzard. (I’m all for walking places, and I do it a lot myself, but in a blizzard? You’d think being unable to get the car out of the driveway would have been a clue, but I guess we were all young and stupid today.) After having little success walking, they ended up hitchhiking to Hy-Vee, a grocery store which was not far from Wal-Mart and which turned out to be closed due to the weather, then walking the rest of the way to Wal-Mart, so now they were stuck here unless they wanted to walk back in the winds which had gotten even worse since we all arrived.
The door we were hanging out by was stuck. I’m not sure if it was iced shut or just happened to break at the worst possible time, but people kept walking in the first set of doors, grabbing a shopping cart, and then being unable to come in the second set. We started yelling and gesturing through the glass as best we could and telling people they had to go down to the other door by the groceries. People walked in every minute or two and we had to repeat this, so we joked that we should ask Wal-Mart to pay us for being greeters.
After an hour or so, the taxi service actually called me back. I hadn’t even left a message, but I guess they checked their list of missed calls. There was one crazy but very nice young woman from the taxi company out driving a minivan, juggling keeping her van on the road, chatting with me about the weather, manipulating one phone with her GPS navigation, and answering calls to the company on another since dispatch was closed. She got to Wal-Mart about fifteen minutes after the last call, maybe two and a half hours after I had gotten dropped off at Wal-Mart, so not too bad all in all. I had to try three times to get her sliding door shut since it had ice on it, both when I got picked up and when I got dropped off. I ended up holding the navigation phone for her on most of the ride, which wasn’t quite ten minutes even under the conditions – though we did have to go around once and find a second route because Google Maps thought a tiny footpath was a road, something Sigrid says is a constant problem in the area. I thanked the driver profusely for being crazy and coming out in this weather and tipped generously, then tried to trudge through the snow up to Sigrid’s front door, only to sink into the snow up to my hip on the first step. Fortunately, someone yelled at me from around the corner that they were clearing the garage entrance, so I got in over there, having to go through snow only a few inches deep.
Sunday night through Tuesday morning: Stranded at Sigrid’s house
Krista and Anna had also gotten stranded. Krista was an alto in the choir, a former midwife and National Park Service ranger in her sixties who has also been to 17 countries and probably done an assortment of other crazy things we didn’t hear about. As you might imagine, she was quite the character and carried a large collection of stories with her. She lived basically in Austin but far enough out in the country that the plows don’t usually run there until the storms are over, so she wasn’t sure she’d be able to make it in for the concert and had stayed overnight here the night before. Anna was a homeschooling mother in her thirties, the only self-described extrovert in the group and yet the quietest of us, with a gorgeous clear soprano voice that gave me chills in rehearsal. She lived a bit further away up the highway, and since she had one of the big solos, she had figured she had best stay as well, as we didn’t have anyone to back her up and it would be somewhat disastrous if she hadn’t been there. And there was Sigrid, of course, who had been directing the choir; she was in her mid-seventies, a music teacher before she mostly retired a few years prior (she is still teaching piano and voice in a private studio). She had never gotten around to downsizing even though she was now living alone in a full-size house, so she had lots of space and three extra bedrooms, just enough for all of us.
None of us had expected to be here long beyond the performance, which was supposed to have been at two o’clock, so we didn’t have a whole lot with us. Nevertheless, we actually kind of had fun, or as much fun as you can have when stuck in a house for 30 hours. We sat around and told stories, looked out the window every hour and spent five minutes discussing how crazy the weather was (that’s the trick to surviving the winter in Minnesota), went through many cups of tea and several bottles of wine, slept much longer than necessary, read all the newspapers and magazines Sigrid had left lying around (Anna at one point realized she was reading a month-old paper only after coming to the weather section), and played an uncountable number of rounds of Rack-O. Sigrid made us salmon and wild rice and then chicken and wild rice soup with the leftovers the next day (the quintessential state soup).
Sigrid had a clock in her living room which chimed on the hour, but one hour behind the actual time displayed on the clock. So at two o’clock, there would be one chime, at six, five chimes, and so on. This was really rather funny once you got used to it, and the chiming to mark off the time we’d been stuck was a defining feature of the whole experience.
Krista made it out just after lunch on Monday, when the plows finally made it up the road to her driveway and her husband was able to come pick her up. Anna didn’t leave until shortly before six, when she went to do auditions at the local children’s theater, and she was able to get home afterwards. We kept checking back at the 511MN road conditions map throughout the day and were amazed at how slowly the roads were clearing, even though the snow had stopped and the winds were calm. The roads were all purple (Travel Not Recommended) in the morning, they were still all purple by noon, and they were still more than half purple by evening. Interstates 35 and 90 were closed throughout the area for the entire day, fully 24 hours after the blizzard warning had ended. The northbound lanes of 35 finally reopened late Monday evening, which meant I could have gotten home had I been able to reach 35, but 90 was still closed, and I wasn’t about to risk any smaller roads leading west to 35 if they hadn’t even managed to clear the freeway yet. So I stayed until Tuesday morning.
Tuesday morning: Retrieving my car
Throughout Monday, I had been trying to locate my car, which was going to be towed automatically by the highway patrol since it was sitting in the middle of the road. I first called the county sheriff’s office at 9:00 AM, was transferred to the state patrol, and was told they hadn’t gotten any cars off 218 yet. This was understandable since there were dozens of cars stuck on 90, which has much more traffic. At 2:00, I tried again and was told only one car had been towed from 218, but the county had been towing as well. Of course, the county had transferred me to the state in the first place, so I called the county back again and asked them, only to be told they weren’t doing any towing at all. I kind of raised my eyebrows and figured I’d just try again later. At 8:00, I called the state again and was told my license plate wasn’t in the system as having been towed by the state, but they had cleared the highway by this time, so my car must have been somewhere.
The state patrol office was able to dig up the number of the towing company the county usually worked with, but when I called them I was told they were towing so many people there was nobody in the office to check if they had my car. I called back 45 minutes later and someone had made it back. They hadn’t had time to update their records, so someone had to actually walk out on the lot – now holding several dozen vehicles – and look for my car, but they were able to confirm they had it. I decided not to bother picking it up until the morning, as I wasn’t about to drive on still marginal roads in the middle of the night in a car that had just been stranded in a blizzard for 24 hours and then towed without me present to supervise.
So Tuesday morning Sigrid drove me out to the towing place. The first time I walked into the building, explained what I wanted, and was told I’d have to go further down the road. This was a tire shop which had the same name as the towing company not five hundred yards down the road and was completely unrelated. I’m not sure who came first, but whoever came second needs to be fined for lack of creativity and creating confusion. At the actual towing office, I paid the $300 towing bill (the woman kindly removed the storage and impound fees figuring it was high enough already under the circumstances), then went out to get my car. When I turned the key, the battery was dead as a doornail; later I figured out I had accidentally left the hazards on when I abandoned it, which maybe wasn’t even the worst idea. Fortunately, Sigrid was still here, so I jump-started my car from hers.
Duh: I am astounded how many people living in Minnesota don’t carry jumper cables in their car and don’t know how to jump-start a vehicle. Most of us will see our car battery die in the winter at least every few years; by the time the temperature drops to -20 degrees Fahrenheit, which is a yearly occurrence up here, a typical car battery has only half its normal capacity, and cranking the engine needs a big burst of power, so even a battery in good condition that’s accidentally depleted a little bit (say, by leaving the overhead light on for a day) can prevent the car from restarting. You’re not likely to be in serious danger from being unable to jump your car, seeing as if you have another car to jump from, that car can presumably bring you to safety or get you help, but it can certainly mean the difference between having an inoperable car for hours or even days and being on your way in ten minutes. Buy some jumper cables for $20 at Wal-Mart and ask someone to show you how to make the connections, and that won’t be you. Roadside assistance doesn’t help much when the weather is bad and everyone needs their car started or towed!
I wasn’t quite done, because as soon as I started moving, I backed right into a boat trailer. A little explanation is in order here: due to the large number of cars in the lot, they had packed them in kind of funny, with the result that my rear bumper was less than a foot away from the hitch of this trailer. So I couldn’t actually see it from inside the car, even in the backup camera, and I had been so focused on getting the car started that I hadn’t thought to walk around the car to check for this. Fortunately, there was no visible damage to the bumper or the hitch, and I was able to get out of the lot without further incident. The car also seems fine in other ways, except that the passenger-side window won’t roll down, which happens to my car anytime the battery dies until I ask the dealership to magically fix it when I bring it in for maintenance. Of course there’s always the chance of hidden damage that will only become apparent later, but there was certainly no gross damage.
I thanked Sigrid profusely for taking me in and headed back up 218 again. The road surface still wasn’t anywhere close to clear, but the snow was packed and the wind was calm and I made a comfortable 40 mph back to Owatonna and got to work only a couple of hours late.
Next week, now that we have the story written up and available for reference, I’ll start applying postmortem analysis tools to see what went right, what went wrong, and how I can prevent any of this from happening to me again. It would be easy to say “don’t drive anywhere in a blizzard, stupid” – and I fully acknowledge that what I did was stupid – but when we make it that simple, we do ourselves a tremendous disservice. If it were actually that easy, nobody would ever get stuck in a blizzard! To make progress, I have to understand why I made the bad decisions I did so I can make sure I don’t end up in a situation where I make those decisions again. That’s where we’ll go next week.
#/media/File:Pole_line_(413157823).jpg){kind=link}