While we're sharing horror stories, a shining example from my experience is a friend who spends days analyzing western blots from experiments: it comes down to identifying a set of 10 rectangular black patches on clear white background and measuring their average brightness. The standard solution is this software package that lets you open each .jpg file one by one (there is a folder with hundreds), and you draw little rectangles and it tells you the average brightness inside. You write the number down inside an excel spreadsheet one by one, alt-tabbing back and forth.
After a brief facepalm moment I opened MATLAB, wrote ~30 lines of code and processed all images in the folder automatically, dumping all average brightnesses into a giant matrix in 1 minute of runtime. Several days of work saved that could go to curing cancer instead of filling spreadsheets.
The problems these people are solving though are definitely things that myself and a lot of my friends would find interesting problemsets. I think its just a matter of making it accessible.
I don't know for sure that there are repetitive tasks that the graduates do there, I am pretty certain that there are some.
But there simply is no structure in place that would allow me to work for them. I could do it pro bono, but honestly I don't have the time for it. On the other hand they simply don't hire programmers to automate tasks, that's just not part of their budget.
Have lunch with some of them. Schmooze with them the next time they call you for a hardware/software fix. Point out a single, simple way they could do something more easily by scripting it. Then recommend that the next time they want to someone to do some research, they instead hire an undergrad who can write code. Undergrads are cheap, and will get them thinking about what they can automate.
This just slows everything down.
Another shining example of this sort of thing is audio engineering. You have programs that are designed as GUI's that look like physical pieces of gear. You move the mouse around to adjust turn knobs, switches and sliders and adjust settings. Extremely cumbersome. But this is how they expect users to be comfortable. Then companies started selling physical consoles with real knobs, switches and sliders to control the software. It's comical.
The touch screen trend is not going to help people get comfortable with the command line. And the key to realy fast computing is using a command line and batch processing. There is no way around it. When you are pointing and clicking, you are just sending a signal that began as a text instruction in the code anyway. Why not just start with the text command and skip the point and click?
Only to someone who's never mixed a song.
Professional audio engineers do their work by FEELING the position of the equipment, making adjustments, and LISTENING to the results. And they expect the feedback to be instant -- just like playing a musical instrument.
Now put them in front of a command line and imagine if they had to type out their intents, listen to the results, and then make tweaks to their typed-in textual program to effect some change in the audio output. You'd end up with a bored, pissed, unproductive audio engineer. The sliders and switches made digital audio professionals MORE productive, not less -- even more so when they became controls on a physical console with digital outputs, as it put controlling the digital gear back in the analog realm a musician is already familiar with.
I find it also interesting that lab environments and music are where plenty of programming takes place by people who are not occupational programmers -- and much of it is visual, in environments like LabVIEW and Max/MSP. This is the future of programming. It is why we should listen to those who say we are too bound to a textual representation for our programs and that in order for the craft to evolve, a richer representation must be embraced.
On a later occasion the friend came to me again with a different problem to solve, but this time it involved spotting dead cells in a mix of cells, images taken with a microscope. They had to count the ratio of dead cells. The difference involved properties of texture and size that could be easily explained and I could do it with near perfect accuracy with little training, and yet it would be much much harder to automate this task. The cells were tightly packed, there were scratches and noise on the image, brightness and contrast variations, distortions,... It would involve training a model, computing features, cross-validations, etc. That did not seem obvious at all to them and it took a while to explain. I'm not even sure if they got it. After all, both tasks are very repetitive and boring, what's the difference?
To address your point, you're right in that I wouldn't expect my friend to did what I did if they only read a "learn how to program in x days!", or even several similar books over few months, or if they knew how to write simple scripts in python or something. But what I would at least hope for is that they realize that this kind of task is very easily automated, and have a very basic understanding of what it would involve. Perhaps labs should come with computer scientist hacker floater technicians?
Once, when I was working on a group project in an operating systems class, we modified our kernel malloc and free routines to print the relevant address every time they were called, to do some simple leak-checking. The result was a file of a few thousand lines like "malloc 0x94583900", "free 0x34739A4", etc. One of my partners looked at the file and said some variant of "oh no - this is going to take hours to match up". Apparently it hadn't occurred to him, someone who'd been programming professionally for years and who only minutes before had been tweaking the internals of a process management function in a multithreaded x86 operating system, that there was an easier avenue available to him.
I was also the first one in class to actually finish the project.
Note that this only pays off if that task occurs frequently enough to offset the upfront programming cost.
But I'd add that even tasks that only require a quick-and-dirty solution can frequently benefit from quick-and-dirty automation. A few throw-away lines in an interpreter environment like irb, for example can often shave several minutes of certain tasks, and throwing a lit bit of code into such tasks can certainly make some otherwise dull processes more interesting.
* print enormous text files with tables showing survey data.
* put a long list of .gif files each one on its own .ppt file (yes, as ridiculous as it sounds).
For the first problem I did two little Pascal programs. The first measured the files and calculated the best settings for the printer (it used to be a try and error process). The second, I did another little Pascal program that added page numbers to the files, so when the printer got stuck (or the pages got mixed) it was easy to resume instead of starting over.
For the second problem, it did an MS Office Macro.
I stopped programming at the office because one of my coworkers asked me to stop because "we are all going to be out of work" (sic).
That is when I learnt the importance of knowing how to code.
This is the argument of the anti-technologist, sometimes called the "Luddite fallacy" .
Somehow these people think that the time freed by automating menial and repetitive tasks can't be used to do more productive or additional tasks, or heighten the ability and/or education of the workers.
Who got benefited from the increase in productivity I generated with the automation I made? The owners of the company.
I even significantly improved timeliness and accuracy. And I absorbed a major change in inputs that likely would not have been accommodate-able under the old system. Turned one of the high-priced consultants onto a reporting package that greatly improved their life feeding upper Management endless varieties of ad hoc reports. Etc.
And... this senior management, brought in from outside due to the share price, set up separate reporting channels using their people, and eventually laid me off.
These days, with so much top-down control, it's often a matter of who you know more that what, and how you are perceived. I was very good at what I did, but from their perspective, I was a "grunt" and a replaceable cog.
In the mainstream work environment, you can indeed work yourself right out of a job, especially if perceptions of you don't match a profile that Management is used to respecting and promoting.
I know I often come across as cynical, here. And certainly, I bring my own faults to the table. But... experience has shown me that these days, still, image often outshines talent in the job market.
When I showed him a related tool that could hunt out and tag terms of interest using the same concept (phone numbers, email addresses, etc.) it completely changed his life. He ended up using those two little kernels of wisdom to mostly automate a previously entirely manual research process and now leads a team of 20 researchers in his field.
He's not a programmer or even a scripter, but he uses regexes pretty much every day in his work.
But for what it's worth, I think a working knowledge of the gnu utilities is worth more than most programming languages to most people. (If you're on a system that supports *nix utilities.)
Ideally it's not an either or thing.
With my comment, I wasn’t trying to contradict your original comment. I was just trying to reword the concepts to be more specific. “Everyone can script” was meant to be a more specific version of “Everyone can program” that might convey the concept better to those who shared or at least understood my definition of “script”.
The best way I know short of immersing yourself in a community of *nix geeks for a long period of time.
If you were looking for something more grandmother or time friendly, I'm afraid I can't help you.
Historically there was a meaningful distinction between scripting and programming: A script is interpreted, so you trade off slow execution for no compilation. A program is compiled, so you get fast execution but slow compilation.
Today (as in 2001) both compilation and execution is so fast, for most use cases it doesn't really matter which you do. JITs further washes out the distinction.
It's all still programming though.
By that, I meant that a “script” is a specific type of program. “Scripting” is a subset of “programming”. Any type of program is a “program”. To be a “script”, a program must be for an ad hoc (one-time) task, and small relative to other programs. At least, according to my idea of the definition – though it is unclear what the “official” definition is.
In my second job out of college, I was running the sound board for a national talk radio show. We broadcasted three hours a day, and I was tasked with getting recordings of each day's shows uploaded as a podcast with the commercial breaks edited out. Being far too lazy to want to do this by hand, and recognizing that we broadcasted on a "hard clock" (a commercial break would run every day from 2:06:55 to 2:10:25 for example), I decided it was time to learn how to let a computer do the work for me.
Fortunately, a friendly coworker who was a hybrid of broadcast engineer and sysadmin pointed me toward bash scripts and cron jobs. Figuring it all out was a frustrating couple of weeks, but I certainly didn't regret it.
A few years later, among other duties, I was responsible for the audio archives of a very old musical organization. Mostly this meant knowing how to make decent transfers of DATs, LPs, 1/4" reel to reels, etc. We found ourselves with over 20,000 tracks of original recordings in .wav format and the metadata about those tracks in an unconnected database.
After far too many searches in the library database, followed by finding the desired file by hand, I got busy scripting (and perusing posts on HN that have helped immensely along the way). That project has evolved into a searchable website containing audio, video, photos and print materials of everything in our catalog.
Next week, I'm on my way to Silicon Valley to interview for a programming gig at a company doing very exciting things. I definitely didn't see it happening this way, and maybe it's not the best career move for me, but my life wouldn't be the same if I hadn't realized my laziness could be rewarded by learning how to code.
So I went to the Chemistry lab, got some mechanical stirrers, plopped them in, and played video games all summer. Good times.
My first internship after college had me taking technical tables in old Word documents, and rewriting them as old-style HTML tables. These documents were often thousands of pages of tables, and there were hundreds of such documents.
It took me a while to realize that, over a few dozen tables, I had come up with a manual system that was just a series of copy/pastes of various HTML snippets in and around the copy/paste of the table contents in a text editor. At the rate I was going using my copy/paste method I was processing tables at about 2-3x the rate of the other interns who were hand typing the markup in the table (the average was about 20 tables per day, I was doing a solid 50-60).
Thinking for a minute I whipped up a script to do the work and my metrics went through the roof. All I had to do was copy/paste the table contents (I couldn't come up with a way at the time to automate that bit of tedium), then run the script. In a day I went to doing 6000-8000 tables a day. My boss, who fortunately thought it was pretty cool, had me train the other interns and we finished the project months ahead of schedule.
So even if a complete solution wasn't to be found, automating even part of it was a massive boost to productivity.
I mean, when you start clearly overusing a tool, if you call yourself a manager you must realize it. You can't keep saying that your job is not IT because at the end your and others productivity depends on it. Somehow I start thinking that the way you do things is as important as the things you are doing.
Bottom line is, we will come up with a budget, everybody will work a lot over the 40 hrs per week, there will be lots of input and logic mistakes (most of them will be corrected, others not). But they will be able to check the 'budget done' radio button. And they will be proud of this achievement. I won't, because I will be feeling stupid spending time doing this kind of things in 2012.
A few years ago I had a situation while working on a circuit board design. The board used huge FPGA's with over a thousand pins each. Creating and managing the schematic and PCB symbols for these chips took days of agonizing work. I finally had enough and wrote a tool that used Excel for data entry and editing. After about a month of work we could take an FPGA from data sheet to EDA files inside of 30 minutes. The tool most-definitely paid for itself many times over as it saw use over several years.
That's the one thing that is outstanding about MS Office: The ability to automate across tools is fantastic.
Early on these guys figured out they could recreate the missing lines using the data they did have.. the problem is that it's the most horribly tedious process imaginable, involving lots of calculations and comparisons.
One day one of the guys runs into a problem where 3-4 of these transactions hadn't been logged properly. I didn't have much to do that day so I opened up Visual Studio and got to work. Field by field I figured out what the calculations were supposed to be and then put them down in code.
Saved this guy hours of work that day, and we save ourselves time every time the problem has cropped up since.
Oh to be a programmer.
That was the beginning, I never looked back.
While we could easily process upto 400 small orders a day (because the warehouse was well organised) we maxed out at shipping around 6 large trade orders per day. Each one took an hour and the courier collection was at 3pm.
After getting through a particularly manic afternoon, I decided to have a look at why they took so long.
I realised it took 25mins just to turn the mailorder spreadsheet into an invoice to mail out with the parcel.
I was dying to automate it! I made a macro to remove the empty product lines, format the layout, background & colour, position the delivery address, insert the company logo and information and generate an invoice number and mark it as paid.
I even future-proofed it so that it would still work even if we added new products to our order form.
I stuck a big smiley face button on the tool bar. I tested it out, invoices now took around 4 seconds. The screen sat there with a beautifully formatted invoice, cursor flashing patiently for me to enter a payment reference and hit print.
Basically that one small change doubled the trade order capacity of the business, which was a LOT of income. It's an awesome feeling :D
That was about 8 years ago, I think they still use it now!
Also amazing that "normal users" consistently don't even want help with automating their IT tasks. Maybe they are worried about losing their jobs?
My take on this latest Jeff Atwood fiasco is that yes, I certainly agree with him that "coding" should not be lumped in with the same concepts as general literacy and numeracy.
But certainly automation using a computer should be, and there are a range of ways available to teach people how to do that.
I think something like "basic automation" provides the same type of introductory computer literacy as algebra does for maths, and "learning how to program" is similar to a continued education in maths.
"Automation" may well be a good bait word for luring in unsuspecting non-nerds, but it's still lightweight programming, a subset. I think at this point we're in agreement.
This was really manual work. You needed to do different mixtures with the colorant and the paint to get an even spectrum of colors from light to dark. Everybody just started to do them randomly, calculating approximate mixtures, doing draw-downs, measuring and repeating until the result was good enough. I started playing around with excel macros and studied how Kubelka-Munk formulae worked. I even realized that there are lots of colorants using the same pigment, but with different strength.
When getting these numbers and some old measurements I built a big excel file with macros, where you just type the pigment data and it would print you instructions what to mix and how much. This saved several days in the process and finally they let me learn Delphi and convert the excel file into a real program.
I wouldn't want to read that code nowadays, but I think they're still using it in the laboratory.
a guy used excel, put a formula into cell A1, and drag it to A10000. file-print ... presto ... wait for 200 pages or so
i wrote a one liner matlab and printed the result in about 20 pages
i just happened to know little bit of matlab -- and that's all needed to 'beat' the competition by 1000%
Many people do not think about whether they can automate certain things. Even if you tell them too, they will often not have the right mindset for it and be unable to recognize that something is automatable.
This is not the fault of those people, it’s a fault of education. Automation is not intuitive, it’s not something humans understand instinctively. To know what works and what doesn’t, to know what’s possible, people need to learn the basics about how to code. (Maybe some will even be able to do some of it themselves, while others can at least ask around for an actual implementation.)
No, it’s not the solution to the failing education system in the US. No, it’s not the best thing since sliced bread. It’s not the savior. What it is, though, is just a good idea.
Actually, I think a large number of people who are good programmers figured out automation on their own, meaning the education system is almost entirely useless when it comes to this. We all have our own stories for this moment of enlightenment - mine was writing TI calculator programs to "automate" problem sovling in middle school algebra.
That way people learn what’s possible. But you are of course right, in most cases even a naive implementation is better than manual labor.
Update: Actually it's this part: "Well, double fuck that."
You can't be afraid to tell your boss you think a task is ridiculous, but you'd better be prepared with a better solution.
Another secretary realized that he could make it a lot faster by just copying and pasting the data into different sheets for each person and then highlighting it in Excel. Then he wrote a macro program that would automate the highlighting once you'd done all of the copy and pasting. This cut the total work load down to a few days.
Once we had the phone records available on the first day of the month, demand suddenly rose for even more detailed and different kinds of analyses. So it turns out we still spent more than two weeks on making reports, but only because new ones kept being demanded since they could actually produced on demand.
*credentials/disclaimer: I'm a programmer and I worked as a clerk for law firms back in college too.