(I am not a programmer, feel free to submit this to the dailywtf, or coding horror, or make fun of me and call me ugly if you want to).
There was a huge huge huge file that I was trying to run a perl script against. The [text] file was about 3 million lines long. What my perl did was split it out into a CSV file so that I could actually work with it (the person sending to me sent it as a space delimited file that didn't really follow its own specs, or have any explanation about what was going on...blah blah, it sucked)...
Anyhow, my perl (stupidly) was doing:
foreach $line (<file>)
{
do_something;
}
The problem was that it kept coming back with an out_of_memory error. The other problem was that it wan running serially (is that a word) on an 8 core box....it would peg one proc with the perl, but the other 7 would just sit there being lazy.
My (very hackish/duct tape) solution was to split (which should have been a big hint) the file into 8 pieces, then call the perl for each one of the 8 chunks...8 cores, 8 threads...should be full of win, yah?
I used wc -l $filename to find out exactly how many lines were int he file, and then gave my new perl script that as the number of lines for each file. The new script did something like:
foreach $line (<file>)
{
print FILE1 $line;
}
(there was a couple of if statements so that after it hit $num_of_lines / 8 it would go to file2 file3 file4 etc.
After that mess was all finished, it would call a shell script that did:
magicscript file1 > file1.magic &
magciscript file2 > file2.magic &
etc. etc. etc. until it was done, then it would do
cat *.magic > fixedfile
which I could then work with...
I put all this into another shell script that did
echo "FULL POWER TO MAIN DEFLECTOR (this could get messy, look out!)"
perl engage.pl
by some miracle it worked on the first try!!
I have since fixed this mess of code. At the time it was a "we don't have time to do it right, just DO IT!" moment, so...whatever :).
When some large piece of fresh code works on the first try, I call this a "hole in one" :)