Hacker News new | past | comments | ask | show | jobs | submit login
Look for the duct tape (rachelbythebay.com)
616 points by fanf2 on Mar 24, 2018 | hide | past | web | favorite | 183 comments

For non-programmers, usually the duct tape is a spreadsheet. I have seen people who invent a CRM on a spreadsheet while not knowing what CRM means.

Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

You don't need a domain-specific tool for every problem your business has. If a spreadsheet works, great. If it ain't broke, don't fix it.

In a similar vein, there are plenty of small software teams that use nothing more than a whiteboard for project management. Yet there are teams tooled up with Jira/Trello/Slack with much less productivity.

I have to argue against this, because spreadsheets are a real pet peeve of mine. It can be hard sometimes to see _why_ a spreadsheet is broken. Spreadsheets simply don't scale (and I don't mean lolwebscale or whatever.)

If two business processes interact through a spreadsheet, or more than two people edit, you're going to immediately run up against the rough edges. Multiple people editing the same sheet inevitably ends in error. It can take a really gross amount of time to update the data when you're pulling/pushing from two sources. It may seem small on the surface, but even 2 hours a week adds up to nearly 3 weeks of work over a year. Can you replace this spreadsheet in 3 weeks?

I've seen this play out many times at large and small scale. The first example that comes to mind is a 10-hour day trip my wife took to go out of town and manually update a whiteboard because somebody messed up the shared spreadsheet. She was very unhappy, and this wasted a precious day during a _very_ busy period. Initially, the whiteboard and spreadsheet worked great, so they didn't "fix" it. It didn't scale to the busy times and really screwed them over.

At a much smaller scale, I wrote a quick SQL query to get some business metrics for the CEO. He asked if I could run it for him every week. Since I didn't have a good place to put this query, I just ran it manually and emailed it to him at first. Then he wanted to share it with someone on the opposite coast outside our network, so he asked if I could start putting it in a shared spreadsheet. My initial thought was to bang out a page they could pull up instead, but I figured it wasn't worth it. Then he wanted another metric. And another that required a bit of logic that couldn't be done in SQL. Pretty soon I was spending an hour a week manually running SQL and copying the results in to a spreadsheet. (We're a Google Drive shop. No way to connect Sheets to DB2.) That's simply not worth my time. I should've listened to my first instinct.

The initial investment in a spreadsheet isn't large, but the ongoing maintenance can really rack up hours. I think it's very often worth investigating any spreadsheet that users have been maintaining/recreating for more than a quarter.

> We're a Google Drive shop. No way to connect Sheets to DB2.

FWIW, Google has an API that can be used to get data into and out of things like Sheets. The documentation for it is abysmal and there are only a few libraries out there, most of which aren't complete. But, I've done it, and have some rough (but easy-to-understand) code that can "import" Sheets into MySQL. Going the other way should be a matter of tweaking the Google account permissions and calling a couple of different functions.

It's in PHP, but it's not gross PHP, so you can probably translate it into whatever you're familiar with.

Here's a freebie that I wrote in like 15 minutes and use in multiple prod situations. Treat any Google Sheet like a JSON data source. https://gist.github.com/chrsstrm/3fb0ce6820acecf62c5490d220d...

Yes, the App Script API is kind of limited, but also extremely powerful for simple services or to glue services together.

There's another really cool way that I've started using lately - doPost[0]. You can set up a js (appscript) function with that name and then publish it for anonymous use (such that the long endpoint url is the secret key).

You can for example set up a cron job to post some json to that endpoint and have the js validate it and update a sheet accordingly (or even create a new one).

[0] https://developers.google.com/apps-script/guides/web

I use pydrive. If you're doing something reasonable once a week it'll easily be fine. I had problems trying to push a few hundred thousand things in a short time, but otherwise I highly recommend this as a first step for sharing data.

What's not worth your time? You're being paid for this all the same aren't you? 1 hr/week would be 52 hours for the whole year, are you able to create something that much better in less time? And will that maintain all the flexibility?

There's a reason Excel is so powerful, but it can't magically solve for people lacking a process and most software solutions solve it by using complex permissions, versioning and collaboration features. You can just easily lock down the shared spreadsheet to readonly and have 1 or 2 people who are designated editors and get the same effect, or just have the changes done on a new tab and compare that way.

Even the most amazing software can and will get screwed up by users who don't have a process.

We programmers are paid for the time we do at work, but side projects like this very often aren't taken into account for the things we are expected to do during the regular work week. That is it's own problem, but this happens to a lot of people. You are paid for doing "the big projects", but then someone asks you to do this little thing on the side that gets bigger and bigger, but it's taking away time and focus on the big projects. Your boss probably doesn't think the side thing is what your job is.

Then take ownership and communicate so that they know.

If there's a better way then suggest it, or at the very least document the time and effort spent on these reports. If they are that critical (which they usually are for decision making) then it shouldn't be an issue when planning future work.

I think I understand the situation you described.

I’m not sure how “global level” organisations are, but mostly you just can’t convince a typical layman CEO that something looks easy, is really not easy.

Then when you serve the “small request”, next he’s in doubt why adding a small feature could increase the complexity of your work by a few order of magnitude.

Maybe today, but its going to keep growing until you hear it groaning under the weight of features that have more and more risk, and generally an unknown amount because those Excel methods are getting creakier and creakier over time as you try to accomplish more with the same tools.

The problem isnt Excel or any useful tool, its the misapplication of the tool and seeing people driving that same road already extremely well traveled.


This, a million times.

I can't count the many times a business process is done (poorly) by inputing lots of data in an excel file, running a huge (poorly written) macro, and spending the file via email to the next person and so on...

> Spreadsheets simply don't scale

This means that if your computing problem doesn't need to scale, a spreadsheet may just be the perfect tool.

However, I agree with you. The next Facebook might be of whomever builds a spreadsheet that scales.

Check our Air Table. https://airtable.com. I’ve not used it in production yet but it looks like a good excel replacement for database type info. The UI is very slick.

I tried smartsheets but it’s more calc centric. Asana is great if you are task centric.

It never needs to scale - until one day you find out that it has overgrown this stage many months ago.

> Spreadsheets simply don't scale

You've never seen panic like a multi million dollar business discovering that excel doesn't allow more than some fixed number of rows (1m?).

The issue is the constant need for new features. Much more cost effective to add a tab vs. modify code.

I think something lost in this conversation is how spreadsheets are largely a part of general primary school curriculum. Generally people have a feeling of confidence around them that is lost with other tools. Particularly "easy database" applications that attempt to replace spreadsheets.

They don’t scale well, but you don’t want to over engineer a solution until you really need it.

Check out Google BigQuery. Not sure if you can do exactly what you need, but we use it as glue between some dbs and Google sheets.

DBVisualizer can export any query you just did to a spreadsheet...

> You don't need a domain-specific tool for every problem your business has.

The opposite is also true: You can make a decent living creating, customizing or even just maintaining domain-specific tools, no matter if the customer's business really needs them or not.

> You don't need

It's very easy to criticize something by saying you don't need it. There are very few things in life that you absolutely need.

Yet there are things that add convenience, greater efficiency, etc, and people will pay for them.

Completely agree with you here, but I see it most of the time from another angle: If you're working on a project where you can't just plug in external tooling for certain things, there might be tooling that's rather subpar and slows everybody down. And yet - when you try to convince management that someone should spend a couple of hours bringing them up to snuff the reaction usually is "meh, that already works.", ignoring the manhours you throw away every week because the thing is organized so terribly. You'd think from an expenses standpoint alone it would be pretty easy to decide to "fix" things like this if you do a napkin calculation of how much of a black hole for money the whole situation is...

> If a spreadsheet works, great. If it ain't broke, don't fix it.

A spreadsheet works until it doesn't.

Maybe you load in past, say, 150mb of data.

Perhaps the last person to touch it linked it to a data or content source they only have access to.

Perhaps the person editing it has it checked out from a Sharepoint and forgot to check it back in.

Perhaps the person editing it left it open on their computer and forgot about it being hosted on a shared drive.

Perhaps the business critical sheet is hosted on someone's desktop.

Fundamentally, spreadsheets are cocktail napkins. It's fine to sketch out a basic idea. It's a terribly risky approach to run a business from it.

I usually frame this as "there isn't any single problem for which excel is the right tool, but for surprising amount of problems it is usable tool"

And I've found out that having excel export of essentially any data in your line of bussines application is indispensable unless you expose some kind of excel-like querying and data manipulation mechanism to users as part of your application.

> Yet there are teams tooled up with Jira/Trello/Slack with much less productivity.

I work for a small company, we generally use Jira (which we all hate, some with more of a passion than others... ok mainly me).

More recently we sort of accidentally started using google sheets for managing issues on some long running projects: it's easier to get an overview, i can add and remove unnecessary attributes of issues and filter them appropriately and dynamically without having to define a whole new workflow and start over. It takes basically the smallest amount of time possible to log an issue... I can organise them however I want. Issue management is overrated, issues are simple, and having something as flexible as a spreadsheet is a surprisingly good fit for anything not customer facing.

Oh also, I don't need to _learn_ how to do all those things... it's a spreadsheet so it's obvious, but Jira - it's so hard to do anything unless you live by it, I don't want to live by an issue manager I want it to get the fuck out of the way so i can do work.

What scares me is the whole world runs on Excel. Name a data-intensive critical industry: deep-sea oil drilling? Power grid management? International finance? All powered by Excel at critical junctures.

It works until it doesn't work then you are Donald Ducked.

I one worked at a place where a key part of the finance system ran on a spreadsheet - turns out this SS had mess up or finances and was one of the reasons the company restructured.

Donald Ducked as in royally fubared?

>You don't need a domain-specific tool for every problem your business has. If a spreadsheet works, great.

If what you are doing is applying math formulas over small or medium sizes of data, then excel is OK.

Otherwise it will suck and will be a waste of man-hours at the Office.

The problem is external criterions like data recovery, data security, quality control, access control, auditing etc.


I can't believe I'm about to sink so low, but... criteria!

Criterions are what one adds to soup ;)

> Criterions are what one adds to soup ;)

You're thinking of crouta.

Let me introduce you to a cool website:


That something incorrectly oft-repeated can take on a veracity all its own is a wonderful aspect of the English language, but it doesn't make things any less twitch-inducing and, bluntly, incorrect.

Besides, I wanted to make a somewhat self-deprecating comment, rather than - as you have here - regurgitate some half-arsed passive aggressive form.

One advantage of spreadsheets is that data can remain local. Even air-gapped.

So can most other things - I'm aware that it's fashionable to have More Cloud, but domain-specific, local toolchains have zero reason to be non-local. "Spreadsheet xor cloud" is a false dilemma.

Well, there's local, as in your DC. And then there's local, as in your laptop.

Still not unique to spreadsheet.

What do you have in mind?

That custom solutions do not inherently require a server - "either it's Excel or it doesn't run on the workstation" is something that's actually easier to disprove today, with containers and whatnot. Custom solutions can be fully local, i.e. no network access required.

OK, fair enough. And indeed, I use MySQL with MySQL Workbench in Ubuntu VMs. But still, Excel on equivalent Windows VMs pivots much faster.

Pivoting is a feature, not a solution; plus it might pivot well on tiny datasets, but for anything over 100k rows, it scales about as well as a brick.

It wasn't that I had too many lines, but too many columns. As I recall, a few hundred columns and maybe 50-100K lines. Even after increasing the field/column limit in MySQL, the query wouldn't complete. But in Excel on an equivalent VM, it computed in a few minutes.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

Strongly disagree with this based on personal experience in "traditional" spaces that rely heavily on spreadsheets. There's a reason most of the financial industry uses Excel spreadsheets and it isn't because they can't afford to pay for specialized software. Excel spreadsheets are incredibly versatile. Your data format can change a little bit over time and you'll still be fine. With a custom software solution, you're kind of stuck within the confines of what the developers/product managers gleaned from your working process.

I'll strongly disagree in turn. I personally know some successful founders whose business, in one important facet, could be summed as "get rid of those painful Excel spreadsheets". It's not the use or not of Excel. As with most things, and as TFA discusses, it's about the pain factor. Spreadsheets are, far and away, the greatest success in end-user programming to date. But like all software tools, their usage can grow into painful places.

Do the users have massive, error-prone and/or time-consuming workflows built around those spreadsheets? Is it very hard to bring new staff up to speed, or to keep existing staff compliant with process, or to meet day-to-day business goals with those spreadsheets? Is there no solution between "painful spreadsheet" and "$X,000,000 contract w/ bigcorp" to solve those problems? If 'yes' is cropping up in answer to those questions, then lifting the logic from ad-hoc process around spreadsheets into software may be a good idea and/or a good business.

I worked at a mid-size startup where all of the customer-facing people were ex-finance---I was consistently amazed by the stuff they did in Excel using just pivot tables, rudimentary graphing, and (VBA? I don't think you could use Python at the time) scripts to do data cleaning, ad-hoc CRM, even rudimentary procedural generation of web content. 90% were one-offs, so it was totally appropriate to not throw developers at the problems (recurring things eventually got built out as microservices).

The main downside was that the functions and transformations weren't sharable across documents, and often there ended up being several mildly tweaked versions of the same functions across a bunch of documents. But I think I learned something about human tool use: when motivated, people will find a way to use the general purpose tools they know to improve individual efficiency.

I generally agree with you, but you picked sort of an exceptional example. As far as I know, Finance/Accounting is one of the explicitly defined use cases for Excel. For finance people, it's actually the best-in-breed tool for what they're doing.

GP is talking about people outside Finance who use Excel as a database. Though he's being a bit facetious, he's actually not far off. For the store data/retrieve data/share data workflows that most of these folks have, there actually is a good SaaS business in putting Excel on a server, wrapping it with a database, putting a nice UI on it and adding a few industry specific features.

Excel is the best tool in the world for quantitative exploration because the data flow and logic are incredibly easy to change. This makes Excel spreadsheets extremely versatile but also extremely easy to fuck up.

The basic counter argument to this is software engineering tools. If you have a more than trivial SS you end up needing tests, version control with branches, merge, and rebase, and an automated process for applying both.

OP referring to all use cases outside of pure financial. Basically where excel being used for managing a list.

For the several people reacting negatively to this: the advice isn’t “replace every spreadsheet you find” which would indeed be silly, but “look for spreadsheets as a possible smell that there’s a gap that could be filled”.

Yeah, and to that I would add another nuance. Even if one does find spreadsheets that "ripe" for a more robust tool-- that still isn't enough.

One also has to consider the cost of engaging the IT department or others who can do and/or manage development within the organization. That's NOT cheap. Many businesses can barely manage to handle their own shit, let alone successfully do even a small unplanned project to port an excel app running in somebody's cube to something more robust.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

I understand where you're coming from, but this is just not good advice. Somebody (I can't remember if it was pg, Joel Spolsky, or someone else) once wrote an entire essay basically saying "DON'T make a startup that's just pivot tables. I have seen so many SaaS startups fail because they didn't realize they were just selling pivot tables at a much higher price than Microsoft wants for Excel."

Also take a look at this HN thread: https://news.ycombinator.com/item?id=12450802 where a whole bunch of people express bafflement of and loathing for people who unnecessarily replace Excel with half-baked software. It seems like a good idea; it's usually not.

> It seems like a good idea; it's usually not.

Not for a startup, no, but in my experience it works well as a lifestyle business. Companies pay well for this kind of work and it seems the number of spreadsheets they want to have replaced provide enough work for the next few decades to come. Better the replacement isn't half-baked though...

It may also have been in a blog post, but Joel Spolsky makes that remark in his presentation "You Suck at Excel" https://www.youtube.com/watch?v=0nbkaYsR94c

That was Joel in his "You Suck at Excel" video.

> Looking for a SaaS idea? Ask random people in a target industry what kinds of things they do with Excel.

True, but you also have to make sure that you can improve on the spreadsheet way enough to be worth potential customers' switching cost and money.

I actually wrote a post about exactly this only a couple of weeks back: "Your biggest competitor is a spreadsheet" - https://medium.com/@hjalli/your-biggest-competitor-is-a-spre...

Specifically you need to whittle it down to those uses of Excel where...

- one spreadsheet is shared among more than about 5 people (or they each have their own version of it... shudder)

- the sheet has more than about 10,000 rows

That's the point where Excel truly starts to be not the right answer.

There's also the case where the spreadsheet contains a whole bunch of endlessly duplicated and error-prone cell logic that nobody really understands, partly designed to work around limitations in the underlying cell-programming model.

When you say "shared" do you mean the Excel file is kept on some network drive and that multiple people can access it there?

That would be the best case! There's also the horror of emailing them back and forth.

I was actually toying with the idea of replacing our custom-made, rather limited shop system with something entirely based on Google Sheets.

We have rather low requirements, which is why I'm resisting existing shop systems. They all seem like overkill to me.

But from time to time, someone wants a some statistics, or needs to send emails to "everyone who bought in the last four weeks". These are trivial for me to compile in SQL, but a spreadsheet would allow others to quickly work with the data.

Unfortunately, the Sheets API at around one second per request was too slow for my tastes when I last checked it out.

You could always schedule regular exports of your data to sheets if you just want it to be easier for others to answer simple questions.

I had a super slow sheet script but I found it had a batch-all-my-changes type of workaround in the API which brought it back into reasonable territory.

Excel spreadsheets are a lot more flexible and easy to use than domain specific tools, particularly for someone already familiar with Excel.

I agree this is a good place to look. I'm not sure a new tool will be an improvement as it will always have limitations and a learning curve Excel doesn't have.

90% of the people who invent a CRM on a spreadsheet are better off using that flexible spreadsheet they know very well and can change, add or remove features, store anywhere they want etc. than using a stupid CRM software with 3 features, designed around its creator limited imagination only.

Yeah. Until they leave for another job and literally no one else in the company can understand the ad-hoc process unintentionally developed over five years of "let's just add this here".

Which is not to say that whatever replaces it is guaranteed to be better, but it will at least have a good start by using the right tools for the job.

The issue for the spreadsheet is always around data-integrity, people start typing random things in their Excel and then it's a mess to try to make sense of the data. I would not mind Excel much if it was strongly typed instead.

But being strongly typed would remove a lot of the usability that makes Excel the most widely used programming language among non programmers.

I think there would be space for a middle ground solution where you could define a sheet "type" to something like "datatable" that would enforce data types on columns, only allow data entered a row at a time and if calculated columns were there, would enforce same formula in each row.

Perhaps there's something to try in that space.

Isn't that what Microsoft Access was?

Microsoft Access used to be the real test. By the time they've added a UI and started to get frustrated with one user at a time it's time for a CRUD app and a database

How is a spreadsheet scalable for a large organization? The team or person might be better off but CRMs and other tools help the organization standardize their processes.

What happens when this person leaves? What if business starts booming and you need to scale up? Are you really better off sharing a spreadsheet between a 100 or more people?

> How is a spreadsheet scalable for a large organization?

Just gonna focus on this part here. The last offices I worked in (and the last places I even dealt with tech stuff beyond teaching it to primary-schoolers), "Track Changes" was simply a godsend. One person "owned" the spreadsheet and could allow use or not; and we had it hosted on OneDrive - of course we all had the OneDrive client on our work PCs and it could tie into Excel natively, so we were all using the same document.

If the entirety of a large org needs the same spreadsheet, you're probably looking at a scenario where they really need multiple sheets, but haven't figured out how to do references to other files and sheets yet. But using Track Changes and having one person designated as Owner so that that one individual makes a decision to approve/disapprove all potential changes makes it much easier to let multiple people work together.

> What happens when this person leaves?

That's a management / succession-planning issue. Ad-hoc solutions shouldn't be kept secret, and allowing employees to just go off and do as they please in that regard is a major issue. If all work-related documents are required to be stored in the company cloud, and you storing a new doc alerts your manager, etc., you'll be able to deal with this much easier. One place I worked for a few weeks on a project did a reset on employee's terminals every Friday at around 7PM, and anyone who claimed they lost data or progress from that on Monday was in trouble, because they didn't save company-related data to the document & data server (which obviously did not get reset) - meaning they'd done something against company policy that they'd been told about the day they were hired.

The person that invents the CRM spreadsheet is better off. However, that spreadsheet is usually harder to share with the rest of the company. If the spreadsheet has more than 1 user than it might be worth a SaaS.

Just use a google sheet and you’re good for a long time.

It depends. Sheets with multiple users is a bad smell to me.

If you can get them to use Quickbase/Gsheets/AirTable or similar instead there's a big improvement. Still hacky, but you know what's out there and roughly what it's doing. And better centralized access control and backup.

If that’s the case, put me in jail and throw away the key. But I think the opposite: between a spreadsheet and something engineered I will take the (google) spreadsheet every time.

I’m allergic to the over-engineered solutions so prevalent.

I encountered situations in corporate environment where a couple of questions with an answer selected from predefined list (radiobuttons/dropdown) were made in Excel... of course completely broken without the latest office on windows and outside of the intranet...

Where I work, I big Word document with a table in it, is the duct tape. Using Excel would be the right way.

I actually had a gig based exactly on this. There was a very complex spreadsheet that was used by an actuarial department of an insurance company that they finally decided to automate. There were .net libraries that could access the spreadsheet components and produce external reports, and put values into a database. Oddly, they kept the spreadsheet component as a the input mechanism.

Oddly? Why is it odd to not want to change from an input method that you already know like the back of your hand? If everyone who’ll be using this relies on the spreadsheet program for most of their work anyway, being able to keep a lot of the process in there makes sense. Learning a new tool is friction.

It was pretty clumsy in its realization. There were like six or seven regions on the one sheet and it was hard to work with. So they kept that part of it.

To be honest, I first learned about using spreadsheets because my mother worked in insurance - not as an actuary but as one of the people under the actuarial department. Lotus 1-2-3 was the norm not only for that office but the industry. As she was retiring, Excel was becoming big (because the standard OS moved from DOS to Windows), and Excel was just starting overtake Lotus. When it's been an SOP for more than 30 years to the point that they can reliably teach the basic use of these sheets in colleges/universities? You're not highly likely to shift it in an established corp.

A hundred times this. Many times the apps which we are building tend to be a simplified version of the spreadsheet. Business points is spreadsheet that is online, which is controlled by authentication and authorisation and has some customizations.

We sometimes even propose to build a native Excel based VBA app. The users can always use something like Dropbox sync or box sync on Google Drive sync to share their work with others.

My most popular project to date started out as copying videogame player names from a website running "SteamQuery", and pasting into Excel.

> Ask random people in a target industry what kinds of things they do with Excel.

Think of all the would-be SaaS businesses that have been disrupted by Airtable.

I run all my business logic on a computer that I built in minecraft.

When did spreadsheets, a tool that is generic and versatile, become worse than a n$/month SAAS with fancy bootstrap inputs?

When they became effectively impossible to audit. That is, at their inception.

I've once worked with a trader whose magic sauce was encoded a spreadsheet he had been developing and carrying around for 10 years. It had tens of thousands of formulas and references cross ~50 sheets.

He suspected he had introduced an error somewhere, but gave up after a couple of months of trying to, you know, verify the whole thing.

Yeah I'd say that If you have a hundred formulas and more than 10 sheets, it is time to change to another tool. But when someone is this far, they'll have accumulated so much technical dept that they will probably have to hire a programmer because the typical SASS won't solve all of their needs.

When each user has a different copy, and nobody knows what the most up to date version is.

When you have error-prone processes that you do hundreds of times a day based on those spreadsheets, that a bigger application can automate.

When ten of those spreadsheets could really be replaced with one.

As with most things, its strengths are its weaknesses. Versatility is great, but it's also the enemy of things like standardization, data security and data accessibility. The more people are collaborating, the more those things start to be important.

Perhaps google trends might be useful (is there some API?) for example "how do i sort a list of customers in excel?" How do i track shipping containers in excel?

Love this...

This is an excellent tenet in general. A landscape designer at school was re-doing the engineering quad and there was a lot of debate about where to put foot paths, and the designer said, "We will put down grass everywhere, then after a year come back and see where the paths are, then we will add stones for the foot paths." It was pretty effective.

I think if you were a product designer you would do well to have users send you their customization files to extract things that were common to all of them to put into the product.

And of course Windows was pretty famous (infamous?) for taking what ever shareware thing people were giving out to make Windows work better and then incorporating it into the base OS. I read a review of Windows '95 that called it "Windows 3.1 with Norton Desktop pre-installed."

> We will put down grass everywhere, then after a year come back and see where the paths are, then we will add stones for the foot paths.

The old desire path that is based on user experience not design [1][2]. Never fight the desire path, it is a losing battle.

[1] https://i.imgur.com/fAmGY2A.jpg

[2] https://www.reddit.com/r/DesirePath/

https://www.imgur.com/r/desirepath (I used to work at Imgur...)

Thank you for sharing desirepath. I spent 30 minutes engrossed there.

And yet i keep seeing FOSS projects that are doing just that over and over and over, with some flimsy excuse about user friendliness as their cover...

This I heard a few times when I was small (30-odd years ago). I always thought it was great advice, the users will find their own way anyway so just figure out what it is and then you help them. Good advice for programmers too :-)

Maybe you don't been need stones once the paths have hardened up. Also better for the joggers if you don't.

Literally an example of pave the cow paths.

As an engineer-turned-programmer/sysadmin, working with other engineers, I've built a career out of doing this. Unfortunately, as a person with only-marginally-better-emotional-quotient than other engineers, I've expected my solutions to be widely regarded, based purely on merit. After 25 years of this, I'm once again hitting roadblock after roadblock because I didn't "kiss the ring" of the people in charge.

I've made them look stupid by building something pretty good where they have been failing for 10 years with an entire team. Then I went and wrote another application in a strange new framework that had been invented in the past 15 years, and this startled and frightened them.

Don't be like me, kids. Play the political game. If you don't want your managers being told to dismantle the things you build, simply because you didn't kiss enough butt, learn how to raise your "EQ" and ensure the success of what you build. Otherwise, like me, you will have just written a bunch of stuff users loved, but management hated, and forced them to stop using.

It's too late for me; save yourselves.

I agree with this as I currently find myself in this very situation. I was a Unix admin for a decade and then found myself having to embrace Linux and then Windows Server to have a job. I've been a sysadmin for 20 years and one is who quite happy to work as a "coder" when needed.

I took a new job about a year ago, one that found me in a group of people who were struggling to do "simple" things, but doing them the "hard way". I came into this job, and because the core infrastructure is Windows Server, I learned PowerShell in about 6 months and started "solving" their problems by automating this and that. Things got easier, but the existing people thought me strange that I would resort to the "mere command line" for a solution rather than go about things using a GUI. My rule of thumb has always been "if I have to do it more than twice, script it."

These people have poo-pooed my every effort along the way and the boss has even called me out on it, despite me showing him that we are making strides where the team once did not. Sorry, I'm a command line guy at heart, and this will never change. I'm not a method man by any stretch. PowerShell is the orthodox way of handling Windows Server issues and that these people I work with don't want to embrace it is not going to slow me down.

We engage in useless meetings to have meetings. We talk about senseless things to jabber. I want to write scripts to automate certain tasks, but no. "What happens if you get hit by a bus, bro? No one knows how to code in PowerShell." My response to them was "I came here not knowing PowerShell, but I took the time to learn it and I'm productive in areas where you could be, too. I'll be happy to show you everything I've learned and coach you when you have time." They look at me like the proverbial deer in the headlights. I'm toying with leaving this year, as I want to work around like-minded, always-curious people, not people stuck in yesteryear.

When upper management see you are 10 times more effective, doing the same job as 10 other people, they might become interested. But if there's not enough job and you get paid for just sitting there, then use that free time to contribute on open source or something. Or maybe there's someone else there who has a pet project you can work on. If this is the case you should consider yourself lucky. At most jobs there's never a shortage of work. Eg. there are always more tasks that you can possible do and pressure from management that it needs to be done today, if not yesterday.

Also if you are that much more effective you could start your own business, hire people and train them, then have 10x bigger margins then your old employer.

Thank you for your comments. I have toyed with the idea for years of going into business for myself, but have no ideas. I'm a sysadmin who knows *nix, macOS, Linux, and Windows very well. I can only write in "scripting" languages, as my jobs all the years required only those. I never really branched out into Python (I can read it), or C++, etc.

I have no idea what to do, what to offer, etc. It's funny, because when I'm at work, I don't really have to think too hard about solving a task. I'm generally the "smooth over" guy who fixes the things other people don't really see as issues, but having been in IT for so long, I can see the problems in the distance for a given job, as I've encountered them before. Most people just fly by the seat of their pants, but I love doing things once and well.

Maybe you can be one of those solo entrepreneurs where everything is automated. Pick a niche that is very boring and are currently done by mundane manual work. For example accounting. Kick start by taking an existing product and sell the patchwork/scripts as a module/plugin. Maybe something like SAP. Don't quit your job until you are profitable though, and can just kill the project if it doesn't work out. It usually takes a few tries and a lot of learning until you figure out what works and what doesn't as an entrepreneur. eg. selling something people want (a faster horse) vs trying to sell something you think they desperately need (a car). Eg. pick the low hanging fruit, and don't be tempted to solve the hard problems until you have enough resources.

I've struggled with the same thing. Here's the problem: How do you farm yourself out for this sort of work, when the kind of people who need it done are the kind of people who can't see the need, even after you've fixed it, shown it, and explained it?

That, and I don't have the temperament to deal with the politics of the business side. I tried it, decades ago, and found it tiresome.

I feel like you're stuck in a job where people don't appreciate what you do. I'd either go look for a better work environment or make one myself by starting my own business if I were you.

How many people are there in this organisation?

[I work in an educational organisation with around 1000 employees and around ten thousand students. There are something like 1 500 client PCs and the usual collection of servers. It is a microsoft shop. Powershell much in evidence.]

Check out Tanium, you'd fit right in

I've been at this cross roads, and it is about "EQ".

It's about building relationship and trust, and I feel it is less draconian as "kiss the ring".

> What bugged me about the (X) environment when I was there? What would I change?

I recently started as lead dev at a small company experiencing some real growing pains. The CEO wisely had me spend a couple days my first week going around talking to team leads. It was very productive.

I started with almost the exact same question Rachel suggests here. My version:

> What's currently frustrating you?

I deliberately left it open-ended and found many of the non-technical answers worth passing along to management. "Let the fools have their tartar sauce!":


The other question I came up after my first couple meetings. I called it the Magic Lamp question:

> If, as the magic genie of this company I could grant you three wishes, what would they be?

By day two, these were the only questions I needed to ask. Following up with the CEO, we came away with a very clear vision of what our development priorities needed to be. Most of it aligned with what I already sensed coming in, but there were still some useful surprises for us both.

(And, yes, found a few clever spreadsheets functioning as de-facto business applications.)

I asked a similar question in my last round of interviews: "if you could wave a magic wand and change one thing, what would it be?". I thought this was a good way of getting an insight into what the team did, and how the people there thought about it.

The boss of the team i eventually joined said something like "i'd like to be able to make all of our performance-critical programs fast enough without spending weeks tuning each one".

Having worked there for a year, i can tell you that this is nowhere close to being a significant problem for the team.

So yeah, ask, but don't take the answers at face value. People don't know what the real problems are.

This was posted a few weeks back...

Career Cold Start.


That Simpsons clip leaves out the punch line, where Smithers attributes the decrease in the accident rate to the promotion of Homer.

True. I noticed that, too. And I think that lesson is as important as the "listen to your grunts" message: beware false metrics.

It's a dense passage to be sure.

Moment of appreciation for the title of this post. This is a rather “sticky” way to share this idea (in the spirit of Made To Stick by Chip & Dan Heath). I get the gist of Rachel’s idea without even needing to read the article.

Really hard to read blog, because the context is often missing. But through reading a few articles it gets more clear.

(probably from another article than the one linked above)

> First thing, do not stick your neck out.

This advise always frustrates me. I know it's true, but feel it's really, really hard to implement.

a) What do you do if you don't get any direct command from your manager? If you then choose to do nothing you get the negative feedback that you didn't do enough. If you go left, you get the feedback that you should go right. If you go right, you get the feedback that you should go left. If you go straight, you get the feedback that you are too direct. With everything you do, there's automatically, intrinsically something you don't do, and that will be used as negative feedback.

b) I don't know about other people, but becoming more successful is really a necessity for me. If you always get negative feedback, how could you progress? So I start hinting at the left and when the negative feedback to go right is incoming I show that I actually moved right. You can believe me, I speak from expeirence, that's when the real shit starts to hit you. Suddenly you are not just a stupid employee, you are also disloyal and an intriguing schemer. You are dangerous and need to get rid off. So apparently that's sticking the neck out.

But then, what else to do? If (a) and (b) both is not acceptable, then what is (c)?

PS: Also this article. Feels so much like my current situation: http://rachelbythebay.com/w/2012/01/09/rigor/

First step is try to have an honest conversation with your manager. You cannot really win if your manager is your opponent. If you cannot have a positive relationship with your manager, you should look for a position elsewhere.

If your manager continues to be ambiguous and you haven't managed to leave, then I would try to go orthogonal. Achieve something unrelated. It is hard to blame someone who succeeds even if that success is unrelated to the main quest.

We had someone do that, then brag about it during end-of-the-year reviews. The managers were quite upset that the person had spent valuable time working on things that not only hadn't been assigned to them, but also hadn't been brought into the process for scheduling them or even brought to the attention of anyone else in IT.

So no, it's not hard to blame someone for working on the wrong things when they should have been working on something else.

Absolutely nothing to do with the topic at hand, but Rachel had a really cool scanner project online for a while that paired well with my temporary obsession with all things RF - http://scanner.rachelbythebay.com/

Back then your options were basically Ettus transceivers for >$1K or the little RTL dongles for $20 that didn't have sufficent bandwidth to do much good. Since then a whole crop of SDRs have hit the market in the <$500 range that could do this kind of thing.

I really like this project. Not only did the radio audio have great quality, the way that the snippets are arranged makes a very good gui.

Thanks! We were really lucky in that the Santa Clara city system was still using analog voice channels with lots of bandwidth (20K0F3E). If you check out the SVRCS system that replaced it (and I also logged for a few years), the audio quality is a lot worse. Super-compressed audio which then gets crammed through /another/ codec (MP3) isn't great.

I will admit that I actually listened to the original system a lot with headphones, so I had every reason to fine-tune it until it sounded as awesome as possible. If you go back far enough into 2011, you can find a bunch of calls with terrible audio before I figured out the finer points of squelch, tuning errors, and 48 vs. 44.1 kHz :-)

I haven't decided whether it's worth building another system now that I have time to actually listen again. Hmm...

When I lived near the airport I made an ACARS monitoring box with the B200 and GR. Had a bunch of fun analyzing engine performance data and the funny messages going back and forth. Travelling with my rtlsdr has saved my butt a few times. Always nice being the first to know that a flight has been cancelled.

I would vote for a new one!

(I love all your posts!)

When it was up and running they would just pop in live. It's easy to miss but the cool thing about this is that it's essentially doing multi-channel recording so you don't miss anything within the bandwidth of the receiver. They would just pop in serially and you would catch up during the dead air.

For a little side project she did a fantastic job (very good write-ups of the trials and tribulations in her blog as well).

All of her posts are well worth reading. I feel like doing so kicks my IQ up a couple of points.

Nice project.

BTW, isn't posting captures of radio transmission online illegeal, because it qualiifes as "rebroadcasting" under the FCC regulations?

>§ 73.1207 Rebroadcasts.

(a) The term rebroadcast means reception by radio of the programs or other transmissions of a broadcast or any other type of radio station, and the simultaneous or subsequent retransmission of such programs or transmissions by a broadcast station.

Internet websites are not broadcast stations.

So I can record anything that's not commercial or encrypted or in a protected/regulated band, and put it on line?

Anything in ISM band or anything said on unencypted 'walkie talkie' bands e.g. Family Radio Service?

There are plenty of sites already doing this with air traffic control broadcasts.

If it's not illegal then yes. If it pisses off enough people laws will be made.

Yes, great site. Made me create one for my local area ;)

I enjoyed reading this as well as some the other blog posts on her site. I was curious about the following:

>"I've been having some wonderful conversations in the real world since resigning and jumping off the merry-go-round."

What merry-go-around or where did she resign from?

I’m pretty sure she used to work at Facebook, but she’s never explicitly mentioned it on her blog. Previously, she worked at Google, where she also resigned.

Yeah I gathered she has spent some significant time in the valley. I hope she might consider writing a post on her question of:

>"... and how to succeed in the "2018 version" of the culture which now exists"

She wrote briefly about it in "How to look out for yourself inside a particular company". http://rachelbythebay.com/w/2018/03/21/next/

Thanks for pointing this out. Cheers.

Agree, but you have to be willing to invest your time and effort into a task when you look for it that way.

Sometimes you can be better off doing things the simple aka duct tape way, when it just works.

Anyone who's not read tacobell programming should take the 2 minutes to do so now because it underlines my point very good.


Incase my point is not clear: you have to differentiate, the concept of work with the concept of problem solving.

Taco Bell Programming can lead to duct tape.

You start with a simple script to do a simple job. It works - well. People want more features. You keep going until nobody can understand what you've created.

Then you leave the company.

Any software that works will be extended given enough time. To adapt the Peter Principle, every piece of software rises to its own level of incompetence.

every piece of software rises to its own level of incompetence.

Which, from a simple review of the contemporary software landscape, is obviously false.

i think the landscape has changed quite a bit in the last 10 years, but its not clear we're in a golden age of simple, robust and composable software either.

maybe you can paint a more detailed picture?

The thing about Taco Bell programming is that sometimes Taco Bell increases the number of available ingredients. What was 8 in the 1970s is at least a few more now. Gorditas, for example. And once upon a time they sold zero varieties of nachos. Don't forget to set up a Red Hot Cheeto cobranding partnership if you want some more variety.

Almost the entire Unix software ecosystem is simple, robust, and composable. I don't know where "golden age" comes from, though the list of existing software that has not risen to their own levels of incompetence stretches back decades.

Fantastic article! This is precisely why founders that are novices in their industry sometimes come up with the really big ideas. As counter-intuitive as it is, it’s easier to see the duct tape when you’re looking with fresh eyes.

I wonder if fresh eyes don’t see the “stigma” of a duct tape approach; rather seeing a good-enough solution for now that’ll work. Instead of agonizing over a better way to do it, they implement and get on with it.

I think more senior engineers fall into the trap of seeing the maintenance burdens ahead and not wanting to, or not being able to, make the call to use the duct tape.

IME, Jugaard is a rare skill.


One tool that I would like (and is probably) out there is tool that would easily let me compare and merge directories. I think this goes beyond Beyond Compare. But here's my use case:

I have videos, images of our children in various places (my phone, one drive, dropbox, maybe usb sticks) that I want to go into the target directory (probably one drive). the tool would go through all the other directories, checksum them against all the other directories, and then present them as "these are the files that should be merge into your target directory".

Can regular diffing tools do something like that in an easy manner?

Maybe off-topic to your original need. But I've always wanted / needed a utility that can de-duplicate media (images, music, videos) that I have scattered around which are the same thing just at different qualities/resolutions.

I get that for music and video this is actually quite challenging but images - - I would've thought quite easy? I've tried a few tools which claim to do this over the years but they all fail rather miserably / require a paid upgrade to do it better (although what is demonstrated in the free version is never impressive enough to consider paying).

Might be a good project for myself in my spare time, but wondered if anyone knew of anything obvious I was missing? FSLint and stuff is great for general filesystem de-duping, this is a slightly more complex case.

I work a lot with image and video processing - this sounds like the kind of thing I'm interested in. Would you want to chat some time? Email is in my profile.

You might want to try digiKam[1], it has a fuzzy search feature for finding similar images[2].

1: https://www.digikam.org/

2: https://docs.kde.org/trunk5/en/extragear-graphics/digikam/us...

I use rsync for something similar to this; it might help you too. I use ‘-P’ which makes the destination be like the source (so you never accidentally delete files in the source) and --dry-run will list what changes would be made without actually changing anything, effectively givingg you a directory diff. I’m sure there’s a flag you could use to say “I only want to create new files in dest if they’re not in source, but not delete or change anything” but I don’t know what it is off the top of my head.

I use a tool that I bought called Resilio Sync that has good usability, but not the flexibility that something like rsync would have. Thanks for the advice.

rsync is definitely capable of doing what you want. Check out the man page and use '--dry-run' to see how each command would work.

Also, the -P option (at least on the version I have installed) is for enabling both --partial and --progress. --partial keeps partially transferred files in the event something goes wrong, and --progress shows you the progress of the transfer.

You will probably want to use -r for recursing into your directory structure.

Total Commander

Thanks, I'll take a look at that. I'm assuming it can recurse into subdirectories and you can use wildcards/regex?

A related observation I would offer is that not every environment / system can be fixed with a layer of duct tape like this.

Unintuitive defaults, annoying long command-line options, needing to take many separate steps to complete one task are all problems that can be trivially and permanently solved with shell macros and wrapper scripts.

Performance problems, using the wrong data structures, random crashes, on the other hand cannot be solved this way.

This is exactly what led me to build Dependabot - dependency management was a minor pain at my old job that we'd written some scripts to handle better.

Another group of friends started Personably from exactly the same strategy - we had a script to create dozens of calendar invites for new joiners with all their onboarding details, so things didn't get missed.

And is it going well?

A qualified "yes".

Dependabot has ~100 paying customers, and makes just over $2,000 a month. It's still quite a lot of work adding new features, but if I wasn't extending it's functionality it would be passive income.

Personably have raised money and landed some big paying customers - I think there's scope for them to be solving a relatively large problem, having started from a duct-tape fix.

The only proviso I'd add to the article is that sometimes the problems you find like this are small. Dependabot will never be a full-blown startup - it's scope is too small. That's not necessarily a bad thing, though - I think the problem has still been well worth solving thoroughly.

https://dependabot.com https://personably.co

Don't any of the langauges you support have options to keep all dependencies up to date? Eg. By putting wildcards in version numbers

That’s normally an option for your manifest file (e.g., a Gemfile or package.json), but most production applications also (sensibly) use a lockfile. Updating the versions in that lockfile is what Dependabot is great for.

This is why open source software works so well- layers of duct tape transition seamlessly into the thing itself.

Always a good approach, but also recognize that not every duct tape solution is worth productizing.

I took the scripts concept as a metaphor for the broken aspects of a company’s culture, and how people work around these broken aspects with “scripts” of behavior that they’ve learnt over the years.

Curious to hear about people’s duct tape fixes for team/company culture issues.

I once has to log time on everything. It was a tempting workaound to log time to the wrong ticket (still one of mine though) so that none of the tickets go over estimate, so I wouldn't be taken to task. I even thought of writing a tool to keep the mapping of real ticket to fake ticket.

Also better to try and get more nebulous work that can carry a higher estimate/ buffer in the first place. And never refactor in the same ticket. Create a new one. But that won't get done so just let it go.

- A nuance: Consider a hammer with a cracked handle.

What's key is not the hammer crack (bad enough?) or the duct tape repair (good enough?).

Instead it's the generality (portable, abstract concepts behind it like 'gluing' 'joining') and availability (common, cheap, simple) of the duct tape that enables the (w)hack.

People don't buy duct tape to fix cracked hammers.

- Afterthought : A Stack Exchange user answers: "why use duct tape when you could just use super glue" (or rubber bands, zip ties...).

Me, I tend to use super glue, some duct tape to reinforce, and finally heat-shrink tubing for neatness and durability.

This is why I was so frustrated with a co-worker a few years back -- he set up his home directory to be unreadable, so I couldn't go look at his ~/bin directory!

this post helped me realize I had a whole open source project in my collection of shell scripts. great post to help you look right in front of you to find something good to work on.

"a whole bunch of dumb little shell scripts"

This is definitely my duct tape. I am neither a developer nor a sysadmin, just a dumb end user. When something cannot be written using only shell builtins, I use a relatively small set of external programs in my dumb scripts, utilities that would be found on most UNIX install media.

If I had the skills I would write many of these in a compiled language, i.e. C or in assembler. As a non-programmer, I have found fasm to be the most useful e.g. its ability to easily call C libs, but I also keep coming back to inline asm as I like the extra bit of control over registers that if offers. I use a small, old assembly level debugger that I prefer over gdb.

History suggests more than a few well-regarded projects originally began life/were prototyped as shell scripts. (I would bet most readers are using at least one today.)

Even though I lack the knowledge and experience of a career programmer, I do not feel ashamed for using Almquists shell to test ideas. Though I certainly understand the use of the word "dumb". However the truth, no matter how "dumb", is that shell builtins are faster than most of the other scripting languages I have tried, and not just in theory.

For some of these dumb little shell scripts, I have managed to replace them with compiled programs (often with the help of flex/yacc), but most are beyond my C/asm "implementation skills".

The point is that if a C programmer came to me, an end user, asking for ideas of what "new" or "missing" utilities they could write (not sure if thats what the author is describing), I could supply a lengthy list. Double digits, maybe even triple.

As I understand it, programmers want people to use their software. Given that I use many of these dumb little shell scripts every day, I could guarantee at least one loyal end user.

Care to share your top 5-10 missing utilities? Maybe I’m that programmer

I'd like to see the whole list as a gist - name, description, sample input, sample output.

Eg: "tac", reverse lines in input, sample input: "first\nsecond\n", output: "second\nfirst\n".

There's already the "standard" gnu coreutils, as well as suckless tools, and 9base from plan9. And projects like https://github.com/uutils/coreutils

So there's plenty of places to draw inspiration from.

Examples of your work?

Does it matter? Just post your list. Worst case, none of them get implemented.

I take "dumb" to mean a simple program as opposed to a product, as described in The Mythical Man Month by Fred Brooks.

I interpreted the snippet you've quoted to be scripts that have /usr/bin/human as the shebang

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact