I spent one staff training day writing a script that parsed the transcripts and generated a visual transcript for each student. It had gray bars showing how much credit was required in each subject area and subdiscipline area, and green bars showing how much credit students had earned. For students close to graduation, it also generated a list of the classes they needed to take. What used to take 10-15 minutes for each student, and was error-prone, now took 30 seconds to 1 minute.
The really interesting and satisfying part wasn't just noticing how much time was saved. We ended up spending just as much time with each student and parent/guardian, but the conversations focused much more on how students could get through to graduation. What used to be a frustrating period of analyzing past failures became a focus on what students could do now that they were focused on graduation.
So I wrote a button (VBA, in Excel) for them that would just do the lookups, and send the correct email response back to the original loan officer. It worked well. They sent my button to all the loan officers, and that department was no longer needed, at least not for anything other than some basic communications asking applicants for more info once in a while.
I had a fun conversation with their IT department, too, when they found out that a temp had automated a department.
This took about 30 - 60 mins each day and was monotonous. Just copying and pasting the client name into the browser search tool.
I automated the process, making a small website which lawyers could add their client and would receive an email if the name appeared in the court lists. I ended up turning it into a small SaaS business for a while and sold a licence to my company.
Not a huge victory but it felt great at the time!
Often, I get questions about the content of these JIRA issues, by issue key: either orally ("hey, what about PRJ-12345 ?"), or in plain text format in email, i.e. "PRJ-12345" _without_ the hyperlink of course. That could result in going to a https://jira.domain.tld/browse/PRJ-12345, or even https://anotherjira.domain.tld/browse/QSK-54321 (yes we have two, separated, instances running).
Like dozens of time a day.
As the JIRA UI does not make it fast enough for me to lookup for such issues so often (and having two different URLs do not help), I wrote a little Hammerspoon script that listen to a keyboard shortcut key.
When I press it:
- if a JIRA issue key is currently highlighted on screen: it just points my browser to it
- if no, an interactive auto-suggester appears and I can type in the issue key, so that I can browse to it immediately as well.
Only a very small automation, but I literally save several precious minutes per day. Mentioning here since the idea might be helpful for others...
So my agenda has been to move the analysis corpus into Python, directly talking to Bloomberg and other data sources. Then these things run automatically and can flag me when they're significant. Rather than trusting me to check scores of spreadsheets and wading through endless True Negative results.
When I started working, Excel was the powertool that allowed a junior to outperform dinosaurs using calculators, and make an outsize impact. My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.
I'll second this sentiment, having built a crude automated trading system using Python and the Interactive Brokers API, for use in trading commodities, futures, and derivatives.
One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc.
Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart.
It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned.
I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'.
One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks.
I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why.
To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.
Having been a hobby programmer as a teenager, I had a hunch this could be made more efficient. Once I discovered VBA (what? Excel has its very own programming language BUILT IN?!) I quickly threw together some macros and an Excel form that generated the whole personalised document set in a few clicks (what? you can write code in Excel to control Word? This is AWESOME!)
At the time I was sharing a house with a couple of CS students and one of them let me borrow a very academic textbook about this wacky thing called SQL. I fell in love with the idea of tables and joins and stored procs, skipped over all the difficult maths parts, and discovered back at work that the funny icon next to Word and Excel called "Access" was actually a SQL database. Feeling like I had all the power of the universe at my fingertips, I hacked together a custom CRM system which pretty soon the whole office was using, and pretty soon it started grinding to a halt.
Round about then the company hired its first IT Manager, he liked what I had done and said: maybe you should check out this new thing called SQL Server 2000, do you want to be our Database Developer? So I shifted the back end over to SQL then discovered .NET 1.1 and started writing better front ends and hired a small team to take it over.
So yeah, I kind of fell into business app development by accident just by automating my very repetitive office admin job. That was 15 years ago, and I'm still getting paid to automate other people's 9 to 5 jobs.
I quickly learned the way to get promoted is to be replaceable.
If you’re relied upon too much or if you’re the guy who always fixes things - it’s actually too risky for a business to promote you.
Throughout my career I always made sure I was giving people below me the opportunity to step up, which allowed me to step up too.
I quickly appointed team leaders as soon as a team I managed had more than 4 people for example.
I had to manage the team leaders but they had a lot of freedom.
Some days I had so little to do you wouldn’t believe it, but the department was running the best it had in years and everyone was happy.
The bosses I have been most happy to work for had such systems in place.
Scrum/Agile (while sometimes derided) if well executed can help with this. If the team is 'self-organizing'.
Although you had little to do, I imagine you probably have to nudge things in the right direction every now and then.
Another aspect I think is important is hiring good people. Hire bad people and you will be a busy manager!
Luckily I did, and if I didn’t I removed the rot fast - that can spread and really make your life difficult.
What if the person leaves the org?
We strive to remove any single point of failure from a hardware point of view, and for me, people are no different
The only exception to this is sales, it’s usually your top biller who you can’t promote - because you want them out there selling. Luckily sales people usually respond to money rather than titles so it’s important to reward those efforts. It’s why some sales people earn more than their managers and 6 figures+ is the norm for good sales people.
Thanks for stating this explicitly.
In my next System Design interview, I am going to list single points of failures as: HW, SW, People :-).
I might be wrong but this just seems almost like a ponzi scheme of some sort - keep moving up the middle management chain and getting somebody else to replace you. No offense to darrenwestall, I'm hating the game not the player.
I'm on a start up journey now and much happier.
Second thing I automated was automating their invoices. There was AS400 system that generated a CSV file which they printed. Then they opened a template Word-document and manualy copied the data from the print into the document, printed that and continued with the next. With some simple marco's I saved them about a days work every week.
Unfortunately scripting and devops aren't a thing where I am even if they could be applied. I'll need to find another corp. The next position I'm going for in is DevOps. But I haven't got enough of the silly certs / experience yet.
If only I had time off to study and wasn't creating reports all day. Then I'd be able to build my project portfolio and do the exam preparation. I've been here 18 months doing nothing but reports and a little system administration. My plan is to be here for no more than 36 months as this level of manually creating reports is too mind numbing.
I generate a very important report for presentation each Monday afternoon and it takes around 4-6 hours of data gsthering and preparation. For the first four weeks I did it manually then realised I could automate some parts and then gradually up to all of it. Now it takes 30 minutes of script runtime with no manual intervention.
There's a patchwork of many other 1-2 hour reports I do across other days. They are now all fully automated by me. They take around 5 minutes of script runtime but no interaction.
I have another big report about to be assigned for Fridays. Its expected to take around 5-7 hours of manual collation. My estimation is that I will be able to run this in around 20 minutes.
In all I generate multiple word documents and powerpoint documents using excel spreadsheets with most of the heavy lifting being done using python, powershell, R and an ecosystem of little elixir critters.
As soon as I reach 24 to 36 months I'll have everything lined up and I'm out of here. I've done enough networking with various people to know that my portfolio of experience is relevant and useful. I've gotten my interview skills up. Plenty of my contacts agree.
At one company pretty much my sole job for 2 days a week was to look over utilization by team, shutdown idle VM's and check for any that were provisioned with too many resources. I'd have to e-mail the team, power them down and resize the resources and boot it back up, etc.
I automated all that with scripts, from the team e-mails with a nice HTML table report of the pending changes to actually running the operations to resize and power off the machines. I was pretty proud of all that work.
Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.
So my job was to go through the alphabet printing reports (e.g. "every item from a vendor that starts with 'A'"). I'd take this report, which for some letters was over a hundred pages long (about 20 items per page) then, using a highlighter and a ballpoint pen I'd go through the report highlighting the items that had stock available in the "backstock" column and had stock in the "sales floor" column that was lesser than the "restock level". I also had to compute the restock amount by subtracting the items on the floor from the restock level and taking the max of that number or the available backstock. I'd write this number in the margin of the report, then I'd travel to the backstock area, pick the highlighted items, and restock them onto the sales floor. After this was done I'd move on to the next letter. The goal was to cycle through the alphabet at least once a week and during the off-peak season this worked fine, but during the busy season, even with extra help, this system was showing signs of stress.
The first thing I did was to write a small perl script which I placed at the path the POS software was expecting to find a line printer. My script took the print-formatted report as input and simply stripped out the lines with no available backstock before forwarding the remaining lines to the printer. This alone reduced the number of pages printed out by an order of magnitude in most cases (from ~100 pages to ~10). At first I would copy this script to the printer path, run the report, then re-link the printer, but later I modified the script to recognize that specific report and pass-through other print jobs unmolested.
Over the next three years or so I added onto the script a lot. First I made it compute the restock amount for me, then I had it start skipping lines where the restock was trivial (e.g. restock amount was 1 and there were still >10 on the floor). Eventually the script was maintaining its own parallel inventory database, tracking rate of sale for each item and only suggesting restock for things that had very low inventory on the floor or were selling very quickly. I got to the point where I could restock the entire alphabet every morning and have time leftover in the afternoons to help with receiving. By the time I left I had a dedicated terminal set up in the back during the busy season that ran my scripts every 10 minutes and displayed in near real-time items that required critical restock throughout the day.
I heard a few years after I left they had switched back to the old methodology because they couldn't get their POS vendor to support my setup.
I wonder if there is a big difference between:
a) an automation system that only the coder understands
b) an automation system that is transparent and accessible to managers
Many coders get to point (a) but the system can't actually be trusted or relied on if the original coder moves on.
At the time I thought they were fools for abandoning such an improved process, but older-me sees the difficulty of the position I left them in. From their perspective they get roughly the same result from having some other cog do the grunt work and hiring a real-deal computer specialist at $$$/hour to be on call to fix problems has the potential to cost much more.
They may have been foolish after all, they ended up going out of business. I think their biggest mistake was allowing their POS vendor to basically lock them in (by being unable to export any of their data in any convenient way) and not demanding/being willing to pay for the sorts of changes I was hacking together to just be incorporated into the official system.