|
|
|
incredible jolt of whatever
Thursday, January 26 2017
I'd set my smartphone to wake me up just before 6:00am this morning so I could watch how my new postfix instance allocation code handled a largish (over 200 thousand) mailing. It worked well, though I saw that I'd bypassed some code designed not to overfill the email queue, and I could see that being a problem with large mailings. The queue has to be fast, and for this reason it actually exists entirely in server memory (not on server disk). But if it gets too big, certain database operations will be slowed. And there's no reason for it to have more than, say, 50,000 items in it at a time. I made these changes as a dose of ambien started to kick in (though not before it had caused me to do the crazy shit I sometimes do under its effects).
I awoke at 11:30am and Gretchen was gone, off to various appointments and such (though I had no idea what they were). I was soon tasked in my remote workplace with a puzzling conundrum. People weren't getting emails after filling out forms, and nobody knew why. It was such a mystery that initially it was thought to be a frontend problem. But soon it was clear that problem lay in the queueing code I'd put live back on Friday. So I spent the entire day trying to figure out what the problem was. I kept creating accounts and looking to see if I got emails while looking at data in debugging logs. The results were sporadic, though it seemed I got about half the emails. At a couple points this afternoon I thought I was on the cusp of an answer, but then my hope was dashed by some further piece of evidence. By around 8:00pm I'd almost given up hope (not that I could allow that to stop me; if I didn't fix the problem it wouldn't be fixed at all). But then something in the debugging log suggested items in the email queue were being deleted correctly, changing my focus from the queueing code to the code that performs the actual write the queue table. When I started logging those SQL statements, the problem became obvious: in some situations, a value (for a column that wasn't even being used) was being set to null, though it couldn't accept null as a value. Fixing that problem fixed the entire issue. The problem had been a side-effect of the changes I'd put live Friday, but they'd been subtle and hadn't even been noticed until yesterday. If that's the worst of the weirdness from my new code, hell, I'll take it.
The process of solving programming problems is always stressful, and sometimes, when it's happening, it makes me doubt my choice of career. But I always end up solving the problem, and when I do I get an incredible jolt of whatever it is human body provides in its internal system of rewards.
For linking purposes this article's URL is:
http://asecular.com/blog.php?170126 feedback
previous | next |
