Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How did you automate your day job?
96 points by amerf1 51 days ago | hide | past | web | favorite | 48 comments
Curious to hear stories of people who automated their 9 to 5 jobs

I was a high school teacher in a school that worked primarily with underserved students, who often had transcripts with many academic gaps. So we had to spend more time than usual examining transcripts to make sure students were taking the right classes to graduate. It was a small school, with three teachers and a few support staff. We literally spent hundreds of hours each year poring over text-based transcripts with students, parents, and each other.

I spent one staff training day writing a script that parsed the transcripts and generated a visual transcript for each student. It had gray bars showing how much credit was required in each subject area and subdiscipline area, and green bars showing how much credit students had earned. For students close to graduation, it also generated a list of the classes they needed to take. What used to take 10-15 minutes for each student, and was error-prone, now took 30 seconds to 1 minute.

The really interesting and satisfying part wasn't just noticing how much time was saved. We ended up spending just as much time with each student and parent/guardian, but the conversations focused much more on how students could get through to graduation. What used to be a frustrating period of analyzing past failures became a focus on what students could do now that they were focused on graduation.

These small, simple things are the reason why I truly love technology.

We need more teachers like this! Optimizing to spend time on getting their students to their goals instead of optimizing to go home early.

Thank you!

I once took a temp job to get some quick spare cash after a failed startup attempt that wiped out my savings, when I still wanted a bit more time before getting another coding job. I ended up helping with some general office work in a mortgage bank, in a department where some accountants would look at all the applications that fell in the grey areas where it wasn't automatically accepted or rejected. They entered the data in a spreadsheet, then looked at other pages of the spreadsheet to see if it was OK to approve or not. Ultimately, there was no grey area... just humans in the middle when it was close.

So I wrote a button (VBA, in Excel) for them that would just do the lookups, and send the correct email response back to the original loan officer. It worked well. They sent my button to all the loan officers, and that department was no longer needed, at least not for anything other than some basic communications asking applicants for more info once in a while.

I had a fun conversation with their IT department, too, when they found out that a temp had automated a department.

Was there a job offer / spot bonus?

Yep. I got a job offer. It was nowhere near competitive with other opportunities, though, so the talks didn't go far.

This checks out. Banks have no problem spending millions on poor solutions but zero interest in rewarding optimisations.

That's because you are a threat to the manager if you implement something like this without approval.

When I worked as a lawyer, one of my tasks as a junior was to check the court lists every day against the spreadsheet of clients we had who were involved in litigation. If their name appeared then I had to send an email to the relevant lawyer containing the information.

This took about 30 - 60 mins each day and was monotonous. Just copying and pasting the client name into the browser search tool.

I automated the process, making a small website which lawyers could add their client and would receive an email if the name appeared in the court lists. I ended up turning it into a small SaaS business for a while and sold a licence to my company.

Not a huge victory but it felt great at the time!

There are several businesses that grew out of people that worked in law firms or corporate legal departments and spotted the numerous deficiencies and opportunity for automation. I know this is true for most businesses but having worked in that area for a few years made me think they're specially lacking process automation considering how bug the Legal industry is in the US.

I deal quite a lot with JIRA at my job, as this is a corporate tool.

Often, I get questions about the content of these JIRA issues, by issue key: either orally ("hey, what about PRJ-12345 ?"), or in plain text format in email, i.e. "PRJ-12345" _without_ the hyperlink of course. That could result in going to a https://jira.domain.tld/browse/PRJ-12345, or even https://anotherjira.domain.tld/browse/QSK-54321 (yes we have two, separated, instances running).

Like dozens of time a day.

As the JIRA UI does not make it fast enough for me to lookup for such issues so often (and having two different URLs do not help), I wrote a little Hammerspoon script that listen to a keyboard shortcut key.

When I press it:

- if a JIRA issue key is currently highlighted on screen: it just points my browser to it - if no, an interactive auto-suggester appears and I can type in the issue key, so that I can browse to it immediately as well.

Only a very small automation, but I literally save several precious minutes per day. Mentioning here since the idea might be helpful for others...

Your method will be faster, but fyi, you can also put the issue key in the search and it will go directly to the issue. We typically configure search shortcuts in the browser so if you type jABC-123 it will auto expand the j to https://somejira.tld/browse/ABC-123

That's what I do too but I liked the idea of highlighting and hitting a shortcut, from anywhere. I also configured Sourcetree to match Jira IDs and hyperlink them.

I do this by using the JIRA search integration with a prefix, and I have one prefix per relevant JIRA instance I have to talk to. 'j PRJ-12345' is what goes in the search bar and I zip right there.

I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't notice it in time because you hadn't checked the spreadsheet.

So my agenda has been to move the analysis corpus into Python, directly talking to Bloomberg and other data sources. Then these things run automatically and can flag me when they're significant. Rather than trusting me to check scores of spreadsheets and wading through endless True Negative results.

When I started working, Excel was the powertool that allowed a junior to outperform dinosaurs using calculators, and make an outsize impact. My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.

> My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.

I'll second this sentiment, having built a crude automated trading system using Python and the Interactive Brokers API, for use in trading commodities, futures, and derivatives.

Would you be open to sharing more? What sort of things are you doing in Python that are new and unique, distinct from capabilities you had in spreadsheets?

isn't python used for automating? I guess excel can do autochecking if you really pushed it

What things are new/unique? Hmmm, I'll bet someone could implement almost all of what I do inside excel sheets one way or another, but it would just be super-cumbersome to do.

One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc.

Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart.

It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned.

I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'.

One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks.

I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why.

To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.

you trading your own money though? awesome answer btw.

Heh, no. I wouldn't want to trade my own money when there are people that will give me money to trade instead.

I actually got my first dev job by doing exactly that. While I was searching for a new career and vaguely interested in something to do with IT, I was doing short-term temp work as an office admin. One was in a startup - not tech-related - where I spent most of my day searching for customers' names and addresses in a spreadsheet, then copy-pasting them into a bunch of Word templates and printing them out. I already knew about mail merge, but in this job the particular set of docs to be generated would be different according to the specific circumstances of each individual case, along with one or two customer-specific paragraphs that came in an email and had to be pasted into the document.

Having been a hobby programmer as a teenager, I had a hunch this could be made more efficient. Once I discovered VBA (what? Excel has its very own programming language BUILT IN?!) I quickly threw together some macros and an Excel form that generated the whole personalised document set in a few clicks (what? you can write code in Excel to control Word? This is AWESOME!)

At the time I was sharing a house with a couple of CS students and one of them let me borrow a very academic textbook about this wacky thing called SQL. I fell in love with the idea of tables and joins and stored procs, skipped over all the difficult maths parts, and discovered back at work that the funny icon next to Word and Excel called "Access" was actually a SQL database. Feeling like I had all the power of the universe at my fingertips, I hacked together a custom CRM system which pretty soon the whole office was using, and pretty soon it started grinding to a halt.

Round about then the company hired its first IT Manager, he liked what I had done and said: maybe you should check out this new thing called SQL Server 2000, do you want to be our Database Developer? So I shifted the back end over to SQL then discovered .NET 1.1 and started writing better front ends and hired a small team to take it over.

So yeah, I kind of fell into business app development by accident just by automating my very repetitive office admin job. That was 15 years ago, and I'm still getting paid to automate other people's 9 to 5 jobs.

You should thank that IT Manager today. Most of them would just ask you to get approval to use a database on your own PC at this point.

I became a manager and delegated everything.

Did you delegate managing too?

Yes, but let me elaborate.

I quickly learned the way to get promoted is to be replaceable.

If you’re relied upon too much or if you’re the guy who always fixes things - it’s actually too risky for a business to promote you.

Throughout my career I always made sure I was giving people below me the opportunity to step up, which allowed me to step up too.

I quickly appointed team leaders as soon as a team I managed had more than 4 people for example.

I had to manage the team leaders but they had a lot of freedom.

Some days I had so little to do you wouldn’t believe it, but the department was running the best it had in years and everyone was happy.

Nice answer.

The bosses I have been most happy to work for had such systems in place.

Scrum/Agile (while sometimes derided) if well executed can help with this. If the team is 'self-organizing'.

Although you had little to do, I imagine you probably have to nudge things in the right direction every now and then.

Another aspect I think is important is hiring good people. Hire bad people and you will be a busy manager!

Absolutely, if you truly want to automate your day job, you need the best people propping you up.

Luckily I did, and if I didn’t I removed the rot fast - that can spread and really make your life difficult.

> it’s actually too risky for a business to promote you.

What if the person leaves the org?

In the longer term that is usually a good thing. If they’re the only person who fixes things, that also implies the correct knowledge sharing isn’t happening.

We strive to remove any single point of failure from a hardware point of view, and for me, people are no different

The only exception to this is sales, it’s usually your top biller who you can’t promote - because you want them out there selling. Luckily sales people usually respond to money rather than titles so it’s important to reward those efforts. It’s why some sales people earn more than their managers and 6 figures+ is the norm for good sales people.

>> We strive to remove any single point of failure from a hardware point of view, and for me, people are no different

Thanks for stating this explicitly.

In my next System Design interview, I am going to list single points of failures as: HW, SW, People :-).

This seems to me like a real failure of late stage capitalism. Similar to how factories in the soviet union were overstaffed with middle managers who barely lifted a finger.

I might be wrong but this just seems almost like a ponzi scheme of some sort - keep moving up the middle management chain and getting somebody else to replace you. No offense to darrenwestall, I'm hating the game not the player.

This quote might be relevant: “I divide my officers into four groups. There are clever, diligent, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and diligent — their place is the General Staff. The next lot are stupid and lazy — they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the intellectual clarity and the composure necessary for difficult decisions. One must beware of anyone who is stupid and diligent — he must not be entrusted with any responsibility because he will always cause only mischief.” — General Kurt von Hammerstein-Equord


No offence taken! I don't disagree - but it helped feed my family well for a long time.

I'm on a start up journey now and much happier.

In short you have human bots working for you.

Last year in school I also worked for a car dealership one day a week. One of their employees transfered a task to me. She showed me a spreadsheet in Lotus 1-2-3, in which they had a list of cars with the day the received the car and they day the sold the car. She was counting the number of days in between and entered that number into the next cell. Took here 2hrs each week. During the explanation by her I already showed here how to do this with a simple formule. She was ecstatic.

Second thing I automated was automating their invoices. There was AS400 system that generated a CSV file which they printed. Then they opened a template Word-document and manualy copied the data from the print into the document, printed that and continued with the next. With some simple marco's I saved them about a days work every week.

I work in compliance for a major bank. Our process to investigate odd spending/payment patterns could not have been more manual. Essentially, cases alert according to a variety of rules and you investigate accordingly. Everything starts in Excel and you know what to look for based on why the case alerts. I automated everything (analysis) required in Excel using VBA. Did not know any programming prior to this but I did know this was the point of using computers ("bicycle for the mind"). Learning VBA was a little painful but eventually I figured out how to control a web browser with it so I would pull the required documents from the web. From there, I got Excel and Word to talk and auto-generated our summary memos. So, pretty effectively automated myself out of my position. The lesson is, while I thought it was all pretty straight-forward, many of my colleagues found my new work flow challenging to use and were pretty concerned my path would put us out of work eventually, which was only kind of the goal. Have since rewritten everything in Python, which is a comparative joy, and I'm now working on using machine learning, NLP, NLG to replicate some of the same work to be more intelligently automated.

Not a day job, but while looking for a day job, I came across a process, where a state run department has multiple openings; you upload your resume things like name, details, previous jobs, address etc to a standardized form, & then at job search page, just click the listing, review the text, click Apply, attach your standardized auto generated resume by click of a button, agree to their t&c with a checkbox, & click Apply, & note down the information from thank you page about job number, city, position, department, type etc. All of these 6-7 pages had unique URL, & all html was nicely generated with proper css id class names etc.

After spending about 5 Sundays doing this manually, I quickly wrote a JavaScript bookmarklet where it gets the current URL, & switch case calls the required function. Each page had different function to look for element, focus, highlight for me, & click it. The last page all required information was loaded & sent to a Google script for spreadsheet.

A major part of my job is to create reports. These are mind numbingly created in Powerpoint / Word with graphs from Excel. I've been told that automating any of the reports I do is basically impossible. I'm expected to do overtime as needed to get reports done on time. This happens most weeks.

Unfortunately scripting and devops aren't a thing where I am even if they could be applied. I'll need to find another corp. The next position I'm going for in is DevOps. But I haven't got enough of the silly certs / experience yet.

If only I had time off to study and wasn't creating reports all day. Then I'd be able to build my project portfolio and do the exam preparation. I've been here 18 months doing nothing but reports and a little system administration. My plan is to be here for no more than 36 months as this level of manually creating reports is too mind numbing.

Also me:

I generate a very important report for presentation each Monday afternoon and it takes around 4-6 hours of data gsthering and preparation. For the first four weeks I did it manually then realised I could automate some parts and then gradually up to all of it. Now it takes 30 minutes of script runtime with no manual intervention.

There's a patchwork of many other 1-2 hour reports I do across other days. They are now all fully automated by me. They take around 5 minutes of script runtime but no interaction.

I have another big report about to be assigned for Fridays. Its expected to take around 5-7 hours of manual collation. My estimation is that I will be able to run this in around 20 minutes.

In all I generate multiple word documents and powerpoint documents using excel spreadsheets with most of the heavy lifting being done using python, powershell, R and an ecosystem of little elixir critters.

As soon as I reach 24 to 36 months I'll have everything lined up and I'm out of here. I've done enough networking with various people to know that my portfolio of experience is relevant and useful. I've gotten my interview skills up. Plenty of my contacts agree.

I was a VMware admin primarily for years in engineering shops and everywhere I worked constantly had issues with resource allocation, which team had the most resources deployed, who had the most idle vCPU and memory running...

At one company pretty much my sole job for 2 days a week was to look over utilization by team, shutdown idle VM's and check for any that were provisioned with too many resources. I'd have to e-mail the team, power them down and resize the resources and boot it back up, etc.

I automated all that with scripts, from the team e-mails with a nice HTML table report of the pending changes to actually running the operations to resize and power off the machines. I was pretty proud of all that work.

I think, before automation, konmari method is more important. If the job doesn't spark joy, we should get rid of it

I don't automate my job specifically (I'm a tech trainer, currently) but when I have optimized myself heavily using AutoHotKey. I spend most of my time these days between Slack and a bash prompt, and so I have created macros for myself. I lean on it pretty hard, and have cut down the amount of typing I do simply because I can now type "yyy" to get "Yup." and other silly optimizations that save me time and effort, mostly cumulatively. There are few big optimizations to be gained this way, but they add up. I also have entire phrases and paragraphs macro'd for the same reasons, as well as common typos. It doesn't sound like much but these things save me a lot of time.

I had to compile reports from Jira of tickets in a release that weren't "in standard". I wrote some Excel VBA that got the details via the Jira rest interface, formatted a report, and sent it round to the "offenders".

Haven't totally automated my entire job but a small portion of what I do is configuring a fresh aws account some core services (SMTP, NTP, LDAP etc etc) and the associated VPC configurations before handing it off to the devs.

Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.

I have been assigned a work in which data is uploaded every month into a web portal for hundreds of items, manually doing so takes 1-1.5 hour. But after automating the uploading task using python beautifulsoup and requests module, it gets done in few minutes.

My first job was doing restock for a small independent retailer. The only report we could print showing inventory was inclusive of all the items we had ever carried, regardless of current status. The software was very limited and there was not much in the way of filtering. The best you could do was break the report up into ranges by vendor.

So my job was to go through the alphabet printing reports (e.g. "every item from a vendor that starts with 'A'"). I'd take this report, which for some letters was over a hundred pages long (about 20 items per page) then, using a highlighter and a ballpoint pen I'd go through the report highlighting the items that had stock available in the "backstock" column and had stock in the "sales floor" column that was lesser than the "restock level". I also had to compute the restock amount by subtracting the items on the floor from the restock level and taking the max of that number or the available backstock. I'd write this number in the margin of the report, then I'd travel to the backstock area, pick the highlighted items, and restock them onto the sales floor. After this was done I'd move on to the next letter. The goal was to cycle through the alphabet at least once a week and during the off-peak season this worked fine, but during the busy season, even with extra help, this system was showing signs of stress.

The first thing I did was to write a small perl script which I placed at the path the POS software was expecting to find a line printer. My script took the print-formatted report as input and simply stripped out the lines with no available backstock before forwarding the remaining lines to the printer. This alone reduced the number of pages printed out by an order of magnitude in most cases (from ~100 pages to ~10). At first I would copy this script to the printer path, run the report, then re-link the printer, but later I modified the script to recognize that specific report and pass-through other print jobs unmolested.

Over the next three years or so I added onto the script a lot. First I made it compute the restock amount for me, then I had it start skipping lines where the restock was trivial (e.g. restock amount was 1 and there were still >10 on the floor). Eventually the script was maintaining its own parallel inventory database, tracking rate of sale for each item and only suggesting restock for things that had very low inventory on the floor or were selling very quickly. I got to the point where I could restock the entire alphabet every morning and have time leftover in the afternoons to help with receiving. By the time I left I had a dedicated terminal set up in the back during the busy season that ran my scripts every 10 minutes and displayed in near real-time items that required critical restock throughout the day.

I heard a few years after I left they had switched back to the old methodology because they couldn't get their POS vendor to support my setup.

Fascinating that they switched to a worse manual methodology.

I wonder if there is a big difference between: a) an automation system that only the coder understands b) an automation system that is transparent and accessible to managers

Many coders get to point (a) but the system can't actually be trusted or relied on if the original coder moves on.

I think they (management) viewed the job as a mostly grunt brute-force sort of problem. By automating parts of it I managed to save myself some time and hassle, but it's not like they told me to start taking half the day off or tried to leverage my talents elsewhere. Other than saving a lot of paper my system didn't really impact their bottom line in a way they could easily see. Even though I trained my successor on using my system neither they nor anyone else in the company was interested enough in computers to take on the ongoing maintenance of the system, despite being just a few hundred lines of perl that an amateur programmer could probably figure out (I was definitely an amateur programmer when I wrote it). When it came time to upgrade their POS software and the vendor refused to copy and reinstall my scripts they had no choice but to return to the vendor-approved process because they had no one technical who could own the system.

At the time I thought they were fools for abandoning such an improved process, but older-me sees the difficulty of the position I left them in. From their perspective they get roughly the same result from having some other cog do the grunt work and hiring a real-deal computer specialist at $$$/hour to be on call to fix problems has the potential to cost much more.

They may have been foolish after all, they ended up going out of business. I think their biggest mistake was allowing their POS vendor to basically lock them in (by being unable to export any of their data in any convenient way) and not demanding/being willing to pay for the sorts of changes I was hacking together to just be incorporated into the official system.

Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact