I made a silly mistake the other day.
A client had asked me to review the memory usage of a suite of programs running on a shared host. His account had a low hard memory limit, and when multiple programs ran simultaneously, the Linux kernel's out of memory killer killed one of the processes.
I ran the program on my machine and saw about half the memory usage. "Ah," I thought to myself. "I have a 32-bit Perl 5 installed and he has a 64-bit Perl 5 installed. "There's lots of little data in the program, so the double sized pointers on his installation compared to mine account for the memory difference."
This is where I should have realized that profiling such an assumption would have taken only a few minutes and could have saved an afternoon.
Installing a 32-bit Perl 5 on a 64-bit machine takes some work. You have to convince your C compiler to compile in 32-bit mode, and this must take place during the point at which you configure Perl 5. After a few failed attempts and a couple of patches to perlbrew, I had a new custom installation of Perl 5.
I installed the dependent modules and ran the application... and the memory use was only slightly better.
Then I thought for a bit. Then I instrumented some code for a quick profile and realized the problem. The application fetches a list of tasks out of a data store it uses as a queue. This queue has an iterator interface, as the list of tasks can be arbitrarily large. Within the code which performs the fetch, I'd assumed that the dataset would be reasonably short at any given time, so I'd flattened the iterator into an anonymous array and kept the whole thing in memory.
The program now processed more data than I'd ever planned, and the problem had only grown worse as more and more tasks piled up, as the consumer couldn't keep up with the producer thanks to OOM killing.
Five minutes later, I'd switched back to the iterator interface, and the process used a fraction of the memory than it did before. If I'd asked myself "What within the program grows over time?", I'd have figured out my mistake before I recompiled Perl 5.
So it goes sometimes.
