|
|
|
momentary terror from a fact in the data
Tuesday, January 31 2017
Before work this morning, I cut new two by six king studs and a 66 inch long two by ten header and then looked around for some way to score the backside of some drywall reachable through only a narrow 1.5 inch-wide slot. I wanted to break the drywall in the back and have it hinge forward so I could do some work in the back, and then swing it back, attach it, and eventually fix it without having to retape a seam. It seemed the best way for scoring it in the back was to make a modified tip for the oscillating tool. This tip would reach some distance forward and then double back at an acute angle, allowing me to apply it from below to an unseen surface. I have some old chewed-up oscillating tool pieces somewhere, and I had the idea of bending one into the right shape. But the metal was just too springy to bend, even when heated orange-hot with a MAPP-gas torch. Next I considered welding a couple bits together in the proper configuration. But I couldn't find the one extra old bit I needed, so I gave up. It turned out that I actually could just barely get my hand into the gap to score the drywall from behind, though the results were messier than I would've preferred.
Today a couple big mailings went out via the email system I've been working on for the past month or more. At first it seemed to work okay, but by around noon the mailing had ground to a halt, and Ca (the email guy) was freaking out about it. I looked into it and discovered the emails were going out at the background rate, which is a small fraction of the peak rate. Something was wrong. Further investigation showed the queue was stuffed with over 90,000 emails. It seemed my queue limiter code had failed because it was getting its number from a cron job that had also failed. I fixed all that stuff and had to wait for the queue to slowly drain. And as it drained, email sending went faster and faster. At some point the queue drained to below 30,000 emails and the mailing rate leveled off at a little more than 20 times what it had been when the queue had been three times longer. I'd never really had to think about or measure send rates in the past, so that seemed acceptable, at least initially. But then I had the idea to query old data to see what the rates had been before my code changes and before the implementation of the two new Postfix instances. To my horror, I saw rates four times as big. My guts tensed up and I began to sweat. Had I ruined everything with my shitty code? In a panic, I checked more recent data to see what the send rates had been during the one big mailing that had happened before the new instances were implemented but after my new code had been deployed. To my great relief, the send rates had been at that high rate. Clearly something bad was happening from the addition of the two new Postfix instances. So I had Dan turn them off. Immediately the send traffic rose up to the impressive rate I'd seen in the logs. Well, it rose to only half those rates, but that was because my queries of the old data had produced data across a span of two minutes, not one minute as I'd intended. Without the two additional instances, the amount of emails in the queue didn't seem to have much effect on mailing rates.
Later I was in the bathtub and Ca was talking about the relative importance of two different mailings, both of which had been trying to go out today. Only then did I actually think about the content of those emails. I realized some things I'd told Ca earlier had been unhelpful, mostly because (from my perspective) all emails are the same. From the perspective of The Organization, though, one class of emails is much more important than the other. But my mind was so full of technical thoughts that I hadn't allowed any such considerations to intrude. And Ca had failed to communicate this information to me, perhaps assuming that I knew or that these concerns would also be in my head. But they weren't.
In the end, it didn't actually matter that much. Both emails went out within an acceptable time frame. The President of The Organization did inquire at some point about the lagging delivery of the important email, but Da constructed a perfect explanatory email that was somehow both reassuring and factually accurate, while not revealing any names. He's good at his job.
For linking purposes this article's URL is:
http://asecular.com/blog.php?170131 feedback
previous | next |
