biggest website fiasco
Friday, August 15 2014
I had a number of web-related tasks to complete today before Susan and David and one of their two dogs (Darla, aka "Spawny") came over for a small dinner party. The most interesting of these tasks involved the website for a chain of alternative movie theatres in the Los Angeles area. Currently that site is administered by a difficult young developer (let's call him Brandon) who doesn't respond to his emails, doesn't complete work in a timely fashion, yet jealously guards the theatre website against anyone else who might want to work on it. I've seen this pattern a lot: websites that technically belong to some business, organization, or person, but are effectively held hostage by a lazy developer who refuses to let anyone else work with their code. Brandon is evidently very particular about the sites he administers, to the point where only he is qualified to work on them. Mike tells me that Brandon yelled at him over a simple robots.txt file he uploaded to a live web directory. Mind you, a robots.txt file has no affect other than to tell Google (and other automated web-scouring entities) what they can and cannot index. I'd been brought in to get a sense of how the site Brandon created worked and to also get a functioning development site working (there already was an existing one, but it was broken). All the sites were hosted using Amazon AWS, which I've only dealt with via an API. Trying to administer a site with the Amazon panel was initially highly frustrating. The panel is a mix of icons and text, which comes across as helpful until you realize that all of the text consists of proprietary Amazon-AWS-only jargon. Even someone as experienced as me has no idea what lies beyond those icons. Clicking on them is no help either; the next page is always some sort of marketing brochure with more undefined jargon. Sure, there are help pages, but they don't contain an avenue for answering the questions on your mind, which is (as always when dealing with web hosting panels) where is my stuff? I don't care about how many Gigagrix of Pytantostank you offer on the Blitzencloud, I want to know what the name of my database server is and how to reboot it if it comes to that. The stuff I wanted was, it turned out, all behind an icon labeled RDS (which is Amazon jargon for Amazon Relational Database Services). But mercifully I didn't actually need to go there, at least not initially. Except when extraordinary measures must be taken, you can ignore the panel and use conventional SSH-based tools; an Amazon hosted environment looks for all the world like a Linux server that can be reached via SSH and communicated with in ways that are familiar to conventional web developers. So there I was in Putty and Filezilla, exploring the web tree, downloading stuff via FTP, and then setting up a new development database. I created one and then thought maybe I should grant permissions to the web database user so it could access the new database, not noticing that I was logged into MySQL as that user. I issued a GRANT command and then noticed that I no longer had access to a bunch of databases. Hmm, that was peculiar. I'd never heard of a GRANT command revoking database priviledges. Wait, I thought, if I can't access the live database with the MySQL user I logged in with, and it's the only MySQL user there is, what does this mean for the live site? Sure enough, the live site, the place all the hipsters in Los Angeles go to buy their movie tickets online, was dead. It wasn't just a little dead either; it was showing an error about a bad database configuration. The pressure was on! I knew the phone would soon be ringing. None of this was pleasant, but for some reason it felt almost as comic as it did tragic.
There was nothing that could be done from MySQL within Putty. This was one of those extraordinary circumstances requiring me to wade into the proprietary-jargon-heavy Amazon web-based console. Eventually I found the database instances behind the RDS icon, and I also found a place where the "master password" could be changed. By now Mike had called my from Los Angeles asking why the theatre site was down and I conceded that I had somehow caused it to die by issuing a GRANT command in MySQL. Mike agreed with me that this shouldn't have caused a problem. He also found something online about how changing the "master password" is the way to restore all permissions in case they get screwed up. So I tried changing it. But it did no good. Eventually I saw the logs, where it said that somehow there were now two users with the name of the master user and that only one of those passwords had been changed. It seems (though it's not clear what happened) that when I issued that GRANT, another user with the same name was somehow created, and now I couldn't do anything to wrestle control back from it. Mind you, I was unclear about who exactly the master password applied to. The Amazon RDS pages do not display this information in any of the places you might desperately be looking for it (and help is useless). At some point somewhere, though, I discovered that the master user was the same user I'd logged into MySQL with, and the only user that Brandon was using for all the sites (both live and dev) to access all the databases.
In desperation, Marc, our contact at the theatre (he's Mike's friend whom Gretchen and I hung out with the last time we went to Los Angeles), put together a conference call between Mike, me, himself, and Brandon the difficult developer. From the moment Brandon opened his mouth, I knew exactly what kind of person he was. Had he been an actor, I would have thought he was overplaying the role. But no, it was just him, being a very tightly-wound asshole. He immediately started complaining (in a whiny voice that resulted in an occasion IM "haha" or "lol" to Mike). His complaints were many, mostly that he'd seen things done to the site that he never would have done, such as the creation of a new "gus" security group somewhere in the RDS pages (something I'd done as one of many desperate, though unsuccesful measures). He demanded to know who this new guy "Gus" was, claiming that the job was being somehow "outsourced" to me. (He wasn't actually wrong about that.) At some point he categorically said that there was no way that I would be allowed to work on the site. "Fine," I said, "I'll be very happy not to be working on this thing. It's been a very unpleasant experience for me." Mostly I found it best to just chuckle to myself in the face of all this bluster. If his whining and insults were what it took to get the theatre site back up, it was fine with me. Brandon didn't move beyond the complaints and insults until after I'd admitted that I'm completely incompetent and have no fucking idea what I am doing. Mike mostly stayed quiet, though at one point he said that he was tired of being insulted. As for Marc, all he cared about was getting the theatre site back up again. Ultimately, Brandon didn't have any better idea than one I'd had of restoring the database from a backup. I'd tried to do this from a backup that was evidently made after the fiasco, but with a slightly earlier backup, it seemed likely that the site could be made to work again. Brandon initiated the restore and, saying he had to go to a meeting, quizzed me about what steps I would be taking to restore the site after he vanished into his turdcloud. "Oh, you're being all Socrates on me, are you?" I said. At this point it didn't matter if I made Brandon angry since he'd already done everything he was going to do. Amusingly, though, his response (as Mike pointed out later) was to back down. Before long, I had the theatre site back up. It had been down for three hours on a Friday afternoon, which is never good for an urban theatre website. But the crisis was over by 4PM Pacific time. Restoring a backup wiped four or five ticket purchases from the database, but they could be tracked down via the credit card processor. All in all, given that it was the single-worst website fiasco I initiated, it wasn't that bad. And it was entertaining, at least in retrospect. Mike and I still are puzzled by how using MySQL's GRANT command had taken away permissions from the master user; we harbor a sneaking suspicion that Brandon somehow weaponized the site like a Hezbollah suicide bomber so it would explode the moment someone else started tinkering with it.
Soon after that crisis, David, Susan, and Darla arrived. I was on an emotional high from the relief of the afternoon's crisis being over (coupled with a recreational dose of pseudoephedrine that had intensified the fiasco's every detail). I told Susan and David what had just happened and when I mentioned that it was a Los Angeles movie theatre, Susan immediately knew which one it was. Later in the evening, when comparing our various workplace fiascos, she said something about once costing a boss $250,000,000 dollars, though she never gave me the details.
Dinner consisted of a homemade vegan mac & cheese Gretchen had made, along with various cooked vegetable leftovers from other meals. Susan and David had brought a spaghetti salad. The mac & cheese was a new recipe and has a somewhat unpleasant miso taste, but everything else was good.
Later, we all sat in front of the woodstove and had a wide-ranging conversation. The weather felt more typical of mid-October than mid-August, so I not only burned a bunch of cardboard, I also threw in a few logs of actual wood.
Conversation lingered for a long time on the recent conflict between Isræl and Gaza. Susan obviously wanted to talk to another Jew about it, having been traumatized by Hammas talking points issuing from the mouth of her recent houseguest Samira (who is Iranian). Gretchen isn't as reflexively pro-Isræl as she might once have been, and neither are Susan and David. Their main concern tonight was that too many people are talking about the conflict from the standpoint of blindered ideology, unable to see any shades of grey. Susan and David seemed to like Gretchen's synopsis of the crisis: "they're all crazy." This applies to both the pro-war Isrælis and the missile-hurling members of Hamas.
As for me, it's no surprise that the Gazans elected Hamas to represent their interests. They're a downtrodden, isolated people with nothing left to lose. And you can't expect Hamas to stop launching missiles when doing so is a political winner for them. Maybe Isræl has to take some sort of military action in the face of this situation, but I think in the long run it would be best if it didn't. Still, the politics in Isræl are such that the ratchet always goes towards more violence and war. There is nothing gained by peace, at least nothing measurable in the short-term timeframe that frames these things.
For linking purposes this article's URL is:feedback
previous | next