|
|
|
not been entirely of my own making
Monday, August 1 2016
I had a super stressful situation develop in my remote workplace this afternoon. Early in my workday, I was alerted to the fact that test emails couldn't be sent by the mass emailing system. I looked into what might be causing this problem, concentrating on the SQL queries, since that's always what seems to cause the problems. But the queries being run were executing quickly. So I went and did other things for awhile. Meanwhile the mass email server gradually bogged down and became useless. By a little after 5:00pm, I coud no longer use phpMyAdmin to administer the server. And attempts to fix things by restarting mysqld did no good.
At some point I noticed that there were no emails in the email queue. There should have been thousands, and the number of them should've been growing. Something was seriously wrong on the server. So I did something daring: I rebooted the server. I wasn't even sure it was going to come back up okay, but I was out of options. Happily, it came up fine and once again emails starting filling the queue. But none were leaving it. Something was wrong with the sending process. At this point I looked in the email log and saw that the last email had been sent at 12:10pm eastern time, which happened to be when I'd had Da deploy some code for me. That was too much of a coincidence to ignore, so I went on the server and looked at the files. I was astounded to see three or four with modified dates for today, though my deploy had only been for one file. I soon determined that the problem was coming from an experimental locking system I'd tried and abandoned maybe two months ago. Somehow it had made it into the git code repository when we'd switched repository systems from svn to git a couple weeks ago. And then, somehow it had been deployed to live when I'd changed and had Da deploy some unrelated file. Back under the old svn system, I could deploy my own code. But under this git system, Da has to do everything, and dealing with this crisis through an additional bureaucratic layer was like being in a nightmare and trying to run from a monster through a swamp in oversized rubber boots. But eventually Da deployed the code and the crisis was over.
I can't tell you how stressful that experience was. Not many people in The Organization know what I do, but it's generally known that if there is a problem with the server then I am the one to fix it. But in this case, for a terrifying period of time (which turned out to be somewhat less than an hour) I had been unsure if I could fix it. And if I can't fix it, then someone has to call up Meerkat from wherever he is (he'd been in Thailand, but now he's supposedly in the United Kingdom) and have him fix it. But I don't want to drop the ball that badly. I want people to remain confident in me. The more like a wizard I am, the better things will be in my virtual workplace. So you can imagine my delight when I managed to diagnose and solve the problem on my own. And though the mass email system didn't send a single email for six long hours, nobody in the Organization had ever become aware of this except me (and, to some extent, Ca, who'd been the one trying to send test emails). Mind you, it had been my code that had caused the problem. But it shouldn't've been in that repository, and even if it had been, it shouldn't've been deployed. So the problem I had fixed had not been entirely of my own making.
For linking purposes this article's URL is:
http://asecular.com/blog.php?160801 feedback
previous | next |
