Excel doesn't have a compeling, standard way of doing version control that I know of, and most Excel files mix freely data and code in the same sheet, which makes debugging pretty hard. Excel files go from assets to liabilities when they start boxing over their own weight, which happens pretty fast...
All of that is forgivable because Excel has done something very few other tools have ever done; which is allow 'non-programmers' to effectively program their computers. Excel is empowering software that lets normal users solve novel problems. With excel, a non-technical small business owner can create software(a spreadsheet) that automates the solution to a problem nobody has ever had before. Solutions to programs the creators of excel never even considered.
A lot of software is like single-use kitchen gadgets. So called "unitaskers". It has one purpose envisioned by it's inventors, and you'll be lucky if you ever figure out something else it's good at. Sometimes they're pretty slick, but still limited. My mother has a device that skins, slices and cores an apple all at once. With that device, you can process enough apples to make a pie in a minute or two... but the chances of you ever finding anything else that tool can do are slim. Maybe you could use it to slice curly french fries out of a potato, but that's about it. It saves your time if you're operating in the narrow problem space envisioned by the inventor, otherwise it doesn't have much of anything to offer you.
Programmers love creating unitasker software, particularly for end users, frankly because it takes much less thought. And when they do create powerful general purpose highly flexible software, their target audience is most often other programmers. Excel (and a handful of others like hypercard) are proof that this status quo can be broken.
In my experience, oftentime Excel is also used out of desperation from not being able to install proper tools, it's not necessarily the only programming solution the users know, it's the only one IT let them have (in the case of technical users, e.g. engineers).
Restrictive IT policies are the root cause of some of the gnarliest Excel sheets I've seen.
Most businesses using excel don't even have an IT department. I'm talking about the LONG tail of small business where the owner is using excel and nobody, except perhaps the size of their wallet, can tell them some other piece of software is off limits.
Consultant, so I see lots of different customers, mainly industry.
Many Excel sheets are a form of shadow IT, no proper way to get the app they need in a reasonable time, so people jury rig something with what they have.
I suspect I see more dysfunction than average still...
I work in industry I've seen some of the same things. The IT policies at my work do tend to be overly restrictive but I think it is somewhat understandable there is maybe two thousand people that work at the plant I'm at. 10 to 15 years ago someone made the decision to outsource IT support (before this every department had IT specialists) there is just no way external support can scale to the level its even feasible to think about giving everyone on site admin access or similar. So we've ended up in enterprise like situation with creeping bureaucracy where everything is super locked down and if you need some software installed submit a ticket get a manger to sign off on it (and be prepared to argue with them about why it's necessary to get this software installed) and maybe 1 or 2 weeks later ticket will get actioned (if you are lucky and it doesn't end up sitting for a month in purgatory somewhere).
Excel is part of standard environment so it is present on every fresh installation here so there is a tendency to reach for it. The "I can open up excel and start working on my problem now, or I can submit a ticket and wait two weeks" attitude is a real thing. The other issue is "I know I have excel installed and I know Bob has excel installed - who knows what else Bob has installed already so if I want to collaborate with Bob I better just use excel."
Personally (I work as an Engineer - the non software kind) I think excel is fine for rapidly prototyping something and as a first pass solution for when you need to quickly answer something but once you've reached the stage a spreadsheet is needed to run process it should be rewritten in something more robust - preferably before it reaches that stage.
> The "I can open up excel and start working on my problem now, or I can submit a ticket and wait two weeks" attitude is a real thing.
Yes, one of the major problems for which Excel is a popular solution (and perhaps the biggest in enterprise environments) is IT service request friction.
If you really want to displace Excel, you don't need alternative software (I mean you do, but not anything novel), you need IT to not be an isolated distant mystery group which can only be invoked by time consuming, arcane rituals to which they give unreliable and untimely responses.
> Programmers love creating unitasker software, particularly for end users, frankly because it takes much less thought. And when they do create powerful general purpose highly flexible software, their target audience is most often other programmers. Excel (and a handful of others like hypercard) are proof that this status quo can be broken.
Exactly. If we view personal computing more like a "medium for literacy," then it becomes clear that computing power is wildly imbalanced. This is especially egregious when we consider the original goals of personal computing. Spreadsheets, Hypercard, and things like them were a totally different direction for computing that has largely since been abandoned in favor of market-dictated "wants" and shrink-wrapped "solutions." The upshot is that the people who should be most in the know -- computer people, developers, etc -- are stuck in a rigid world of epistemic closure (where, for example, Unix is more or less the "End of History")
I have yet to find a better definition of programming than "telling a computer what to do." When put this way, a lot of things that the trade-school approach consider "programming" are really only a subset -- and perhaps the most regressive and uninteresting to boot.
There's a saying: do one thing well . Software doesn't weight or take up space. Then there's high level programming where you sew together high level modules.
The problem is that not everyone has the same needs, and there aren't enough programmers to write special purpose software for each person on the planet. Making software customizable is a way to give users a powerful tool that can be adapted to their needs.
I met a guy at a wedding reception last year, where we were talking tech, and I was lamenting similar issues (we use fairly large multi-worksheet spreadsheets for configuration). He mentioned that his company makes some tooling that might help with this:
Aside from Excel's built-in "Track Changes" features (which stores the revision log within the file), the "Spreadsheet Compare" tool that ships with Excel is a decent diff engine that can be paired with any VCS. Sharepoint is another "official" solution.
Unstructured use of Excel is a completely orthogonal problem.
Google Sheets lets you download the workbook as an Excel file which can be automatically saved to a Google Drive folder. Hourly (?), daily, weekly, monthly snapshots of a shared workbook can all be easily scripted and stored for later retrieval. It eliminates any confusion over which is the latest workbook because the Sheets version is always the most up-to-date.
Google Sheets is the main reason I gave up on Google Docs and bought an Office 365 subscription for myself a few years back. It's just nowhere near as good as Excel. Frankly none of the Google Docs offerings holds a candle to Microsoft Office[1], so the latter is well worth the money to me.
[1] I am tactfully not counting GMail as part of GDocs here because, honestly, Outlook is the one tool that wouldn't come off well in that comparison, and reason #1 is that Outlook's search is terrible; reason #2 is that Outlook doesn't perform at all well in the face of the modern "I want to keep all my mail so that I can search for and find relevant messages whenever I need them", which can be key when dealing with tricksy customers/partners/co-workers, and is handy even in mundane day to day use.
"Frankly none of the Google Docs offerings holds a candle to Microsoft Office"
Word and Excel are great for individual productivity. If you use documents and spreadsheets as tools for collaboration, though, Google Docs is much much better overall, even though it's missing many features to which power users have become accustomed.
Sure, you can save a file to OneDrive and have multiple people working on it at the same time. But:
- In my experience, simultaneous editing via OneDrive (whether using the browser or the desktop apps) is more laggy and buggy than with Google Docs
- The commenting functionality is missing lots of features that are essential to my desired collaboration workflow:
i) Ability to give someone access to add comments and suggest changes, but not to accept changes or edit directly
ii) Comment authors are identified, and only they can edit comments they have written
iii) Comments can be received and replied to by email.
The way commenting works in Google Docs encourages collaboration in a way that the simple comments in Word doesn't. If you've not worked somewhere that uses Google Docs for collaboration, it can be hard to know what you're missing.
> If you've not worked somewhere that uses Google Docs for collaboration, it can be hard to know what you're missing.
I have (for a previous company), and it still hasn't proven worth it at other companies, including my current employer. We use Office 365 exclusively: it's not perfect but, as I say, Excel is streets ahead of Google Sheets in every other way, and that matters for our use cases.
So Google is "much much better overall" for multiple users, as long as you can dumb what you're working on down enough to fit Docs significant limitations. That's totally fair, but isn't a trade off everyone can make.
Yes, there's a trade-off. Note, I didn't say Google Docs is better for every use case. I said it's better "If you use documents and spreadsheets as tools for collaboration".
The key difference I'm talking about is between:
A. Writing a document, and sending a version of that document to one or more other people, so that they can do something with it. (This might involve sending you back feedback or suggestions.)
B. Sharing a link to a 'living' document, with which people can interact in different ways (comment, participate in comment threads, add change suggestions, accept/decline changes).
In my experience, B is much harder to achieve, and unlikely to happen organically, if you use Microsoft Office.
Has anyone here witnessed an organisation that uses Microsoft Office, and where significant progress is made on people's thinking, through their online collaboration on docs/sheets?
We use Google Sheets as a team for its collaboration functionality. However it still has a lot of room to improve with its visual customization for charts and formatting and such.
I realize some people don't care about how beautiful a spreadsheet looks, but in presentations or sharing complex information, it matters. And formatting anything complex with charts and such is a major headache in Sheets (like my latest struggle to get annotations to properly appear, or move a single peak data label so it isn't cut off from displaying).
The thing about Google Docs is that I haven't seen any changes or updates to get toward feature-parity since I was using it in high school a decade and a half ago. I assume it has been exiled to Google's island of misfit toys that they can't quite take out back and shoot just yet.
Have you tried using Google Sheets? I have. You can't even cut and insert a bunch of rows. That's a primitive functionality that is very much needed that was available in Excel circa 1988 (I'm exaggerating, but it's been in Excel for at least 2 decades).
While sheets is nowhere near my idea of ergonomic, I've figured out this one every time I needed it. And, despite working for Google, I'm nowhere near a regular user.
Maybe your browser, or some extension, got in your way?
Sheets is way less powerful than Excel. Losing a lot of the keyboard shortcuts also makes it way less useful for power users. It also doesn't do well at generating macros and its chart generation capabilities are much less flexible.
Everyone commenting: Google Sheets is not an exact copy of Excel, but it's the closest (I know of) that exists.
For plain-old Excel with version control, save your Excel files in Dropbox. MSFT might also offer a cloud version with version control but I have no experience with it.
Are Google Sheets "easily machine-readable"? I've tried to add support for it to my application, and I could not get it to work at all. Maybe it's technically possible but it's sure not easy. You need to use Google's proprietary APIs, and the documentation and support are terrible.
I've pulled some data from a sheet into a python notebook by copy-pasting some code I've found. Sure, these are proprietary APIs, but how else would you talk to a proprietary web service?
Disclaimer: despite working for Google, I don't know that much about the suite.
IIRC, Excel XLSX files are just ZIP archives with a bunch of XML inside. Therefore, if you can come up with a way for a version control system to transparently look inside ZIP archives, it ought to be possible to version-control XLSX files using the same tools you'd use to version-control any text files, no?
(Though this only solves half the problem, the other half being that even if you can wrangle Git or the like to version files inside a ZIP, Git is far from being a tool that could be considered appropriately usable to dump on the Excel audience...)
It isn't as simple as that. There are numerous equivalent representations of the same underlying content (same data, same styling, same charts, etc) that have wildly different XML representations. You need a content-aware diffing tool.
A file that is edited by more than one person, of which at least one doesn't quite now what he does? It's pretty hard to see what changed from one version to another, and a destroyed formula can go undetected for a while...
And obviously, merge would be nice if multiple people can legitimately edit concurrently.
In spreadsheets (excel) and notebooks (ipython/rstudio), having the data bundled with the code is a feature. It provides reproducibility.
> a destroyed formula can go undetected for a while...
That is a problem in methodology, not with the tools. There are many solutions that do not require abandoning spreadsheets.
For the destroyed formula example, you need tests. Simple example to do that with a checklist: if you are doing a SUM(), require that the employee ticks a box saying "all the number were highlighted when clicking on the SUM formula".
For the out-of-sync version, you need a central repository and another box "I retrieved the latest version from the xx repository, and this version was: ... "
Then require that to be printed and signed (accountability), and you'll see mistake disappear.
Is it any different with code? Developers don’t always stick to the procedures either.
I have ‘unit tests’ in my excel files on the last tab. Eg, the sum of all lines in the Data tab must be the same as the sum of the annual revenues in the dashboard tab.
With a bit of conditional formatting it’s easy to see if a test is failing as well.
I like programming for lots of things, but ask me to do a business case or some one-off analysis and I’d use Excel in most cases.
After they received training when they learn that this step is important and not just a formality, and why it is done, I found that people do observe procedures, especially when they have to sign the checklist they filled themselves.
Human beings observe procedures when they understand it's part of the job and they are held accountable.
Version control is extremely useful in software development because code modification by multiple authors is a primary use-case that happens multiple times throughout the day.
Maybe your experience is different, I haven't seen a use-case where multiple authors are changing excel macros or formulas that often, or even often enough to where this is an issue. In fact I think adding version control would make things very very confusing for most people.
And like any tool they can be abused - I really like Excel but I've seen it used for things that were unwise.
I've spent months reverse engineering a vast, complex, costing application that had evolved within a spreadsheet and needed (for very good reasons) to be turned into a proper application. The really entertaining part was trying to convince people that their spreadsheet didn't actually do what they thought it did....
Edit: It was costing a complex heavy industrial process so it had logic that was basically physics right through to finance.
I think the solution should be to have better excel reverse engineering tools. I have seen plenty of crazy complex spreadsheets that had hit a wall and were hard to refactor but the solution would never have happened if they hadn’t had excel enabling regular users to create it.
I don’t think Excel is ever bashed for its primary use case. It’s when someone starts building complex, multi user applications things start to go wrong. It almost always becomes an entangled mess. And good luck to anyone taking over when the creator leaves. I’ve been both the creator and maintainer of such Excel applications, and I do not miss it.
One thing that frequently comes to my mind - rather than trying to convince people to move to a general-purpose programming language (and ignoring all the reasons that people turn to Excel in the first place), is it possible to turn an Excel-style spreadsheet into a general-purpose programming environment that is powerful enough to do that sort of stuff?
People really underrate Excel as a programming environment, honestly. Yes, it's crippled and leads people to produce massive gross un-debuggable hellsheets ... but there's reasons (beyond just "it was the only usable software that could be run on office computers") that end users with a problem to solve keep turning to it despite those flaws. The combination of reactive programming, data-first visibility, no hidden state, decent approximations to structured programming by way of click-drag-and-copy-paste, and being able to reference variables and values without needing to name them has some kind of magic to it.
There's quite a bit of interesting research I've seen from Microsoft on how to take something like Excel and turn it into a non-crippled programming environment - spreadsheet-defined functions (including recursion, lambdas, and higher-order functions natively in the spreadsheet environment, without having to drop into VBA!), dynamic arrays, an alternate computation-first textual view that exists simultaneously with the data-first spreadsheet view, first-class complex data structures ...
Between HyperCard, Jupyter/Mathematica notebooks, and Excel, there's a lot of common features that seem to point to a vision of a truly useful end-user-accessible programming tool, something that would let people write real, useful programs intuitively and quickly enough to be worth building them themselves for whatever they needed them for, without forcing them into condescendingly-simplified drag-and-drop "visual programming" codeless clunkfests.
Excel is a general-purpose programming environment, it has a full programming language inside it (VBA) with the ability to call out to the operating system and load functionality through DLLs.
No, the way to tame Excel is to make it less general, to impose principles of structured programming on it. Separate code and data and presentation. Put some of the data in a database. Impose types on it, so Excel doesn't interpret genomes as dates etc. Make the control and data flow visible. Split it into pieces with defined responsibilities.
Excel is really a "two dimensional assembler": you put an operation in each cell, and they can read any other cell, and it all executes together.
In my first job out of Uni I wrote some VBA to make requests to the "online diary" to pull out some of the information into some template Excel files we had to create.
Some poor soul is probably still maintaining that.
After switching from technical to a product role, I have seen a complete change of perception towards excel/google-sheets. I still observe the passive hate towards excel by developer companions, but for me, it's a goto tool, the same as it was the case with Python, the goto language.
I like to call it a fundamental or building block technology that enables you to develop things on top of it, my favorite being a finite-state-machine we modeled in a sheet. Won't be surprised if we end up talking about Google sheets being one of my favorite products.
Because lots of people have lived through a spreadsheet that had grown too large and too important, and the original author might be gone, and reverse engineering a spreadsheet that's hundreds of megabytes in size and has a dependency graph that's longer than a page quickly becomes... burdensome.
because the one you got emailed last year is out of date vs the one your boss got emailed last month which is also out of date from the finance team's version
Not only that, but Excel can tether back to a traditional rdbms for pull/push data. It's often the preferred way for PMs to interact with, for example, VSTS/TFS. There is absolutely a LOT that can be done with Excel and other spreadsheets, and it shouldn't be underestimated.
I'd rather not work with them more than basic/classic usage of a more well formed data dump. I use spreadsheets for monthly finances, some historical information that fits well, and a few other use cases. I'm not big on VBA and have little desire to work in the space.
I have, however, seen some amazing usage of Excel and other spreadsheets for everything from charting and pivot data, to multi-user applications tethering to other data stores.
Agreed. The bashing does not really come from the end users / business people who benefit from the spreadsheet but those that are either competing against it (engineers) or having to take on risk through misuse (compliance).
As other comments have noted, it gives rise to the new problem of the proliferation of stale, incomplete, or otherwise incorrect data.
Google Sheets, on the other hand, is truly compelling, and a lt of the work I've been doing lately has been using Sheets and Apps Script to build really powerful "apps" on the super-cheap. This is the stuff we used to do with VB years ago, and it would take weeks. Now I can get at least a PoC of a fully multi-user system in front of a client in a day, and they don't pay anything to host it.
I would wager that 10 times (perhaps 50 times or more) as many people can make a simple Excel worksheet as could program the same functionality in a mainstream, general purpose "programming" language.
The quotes seem unnecessary. Programming isn't just an expert activity. Many applications offer macro recording and other facilities to create programs without scripting. Dataflow programming is often used in graphics and media applications. Spreadsheets are one very successful implementation of dataflow programming.
What Excel doesn't let you do (without heroic effort in support toolchains) is software engineering...
It also allows domain experts to program. So while they may not be expert programmers, they are experts in the problem area and will come up with expert solutions.
Spreadsheets really are the fullest realization we've seen of functional-programming-without-code, and with a built-in (rudimentary) database too. It's no wonder they kicked off the microcomputer era and remain essential to this day.
Also the realization of the hardest trick in industry, how to scale :)
Excel works very well for single users with small datasets but if the business works it breaks in various ways. Sharepoint was the solution to scale excel but it didn't seem to have been successful as a tool.
Tons of businesses continue to function with excel without difficulty. Of course if you're shooting for a SV "unicorn" growth curve you're going to leave excel in the dust pretty quick, but a great many businesses in a wide range of industries find excel continues to meet their demands for a lot longer than you might expect.
Out of curiosity. Did it stay that way or did you manage to migrate to another solution? If so, what did that look like? Interested because I’m in a similar situation.
A lot of things fell through the cracks with Google Sheets and Zapier handling everything. I eventually just moved to building a personal dashboard using Next.js and connected with the APIs directly.
Because of excel's swiss-army-knife powers, it will always be present somewhere in most organisations, I agree.
But if you start your business with your stock and order tracking in excel, your customer list in excel, and your accounts in excel, it's not unusual to outgrow excel in some of those areas fairly quickly.
Right, Amazon's inventory system certainly can't be run out of a spreadsheet. But even for something like a very successful restaurant with half a dozen locations across the state, a few spreadsheets often sit comfortably in the middle of it all.
Startups hardly bother with licensing excel/365 these days, and just pop into sheets. From my experience, you don't usually start bringing excel into the equation until you hire that one sales manager that insists on it, or the "Microsoft guy/gal" that just keeps asking "why aren't we using excel?" over and over again.
Or until you're at the scale where you're working with datasets/queries simply too big or too complex for Sheets to handle effectively.
At my current job (late-ish stage SF startup), most of us default to Google Sheets, but there have been enough use cases for "real" Excel alone to warrant setting up Office 365 licensing/accounts and defining a flow for that at the corporate level instead of just one-off licenses here and there.
Our finance team in particular (or at least specific key members thereof) has to use the Windows version of excel because both the macOS version and Google Sheets rapidly buckle under the datasets they're manipulating in their monthly reconciliations. My team has been working on various tools and systems (mostly as extensions to our ERP) to better automate some of these things, but Excel is still a reality until that happens for all their use cases.
> Or until you're at the scale where you're working with datasets/queries simply too big or too complex for Sheets to handle effectively.
At which, again from my perspective and view of things which may be wrong, startups are more likely to automate/productize that so they can better maintain and dogfood it if possible instead of letting someone's Excel workbook grow into a pet ML tool that only they understand.
I think this is where our respective experiences diverge, at least momentarily. Yeah, eventually a company will (hopefully) try to build better tools (as I and the rest of my team are trying to do in our case), but there will be an often-long gap between some employee creating a spreadsheet-based monstrosity and some engineering team turning that into a better tool/system.
I worked for a clearing company that mainly dealt with commodities market and option exchanges. It was a fight to get the individuals in finance to use the software we developed because they would just use Excel with macros. I had even been asked to look at one of the workbooks they were using to fix an error they had been having.
The problem with the software you developed is that they have to give up control. Every change is a change request which needs to be approved and implemented and tested and released.
The upside is higher quality, the downside is way reduced turnaround speed if they need to change something.
I understand why people using Excel are often reluctant to give it up.
Quality often goes down. That request process results in you getting what you ask for, not what you want.
Think about anything that you do in Excel. In what scenario do you roll with exactly what you initially start with?
I just did a preliminary capital forecast in Excel. I could have used something like Google sheets as well, but beyond that, the lack of flexibility makes purpose built solutions less able without specialized people that I don’t have or domain knowledge that I don’t possess.
This. Excel democratizes computation in a way that code has yet to do. If something breaks, non-technical people can just jump in and fix it. If they need new functionality, same thing. They don't have to bring in (or retain) a specialist. That's an incredibly powerful proposition.
>>Tons of businesses continue to function with excel without difficulty.
Without difficulty? No, definitely not. The difficulty is always there, it's just internalized, ignored or worked around in various (often crazy) ways.
Without difficulties induced by scale. Some sorts of businesses will hit scale issues with excel, but many will not.
The baseline difficulty of using excel is not what I'm talking about and I don't consider it relevant since experience has proven that baseline difficulty is not much of an obstacle to real adoption.
Sharepoint was the solution to scale excel but it didn't seem to have been successful as a tool.
I've seen these trends around "Unbundling" Excel:
- Improve UX by specializing UI
- Increase automation with software integration
But given that Excel is acting in the guise of software, doesn't it follow:
1) Running business of Excel has many of the same problems as other software
2) Excel has some key advantages over other software
(By 1), I mean, QA, testing, deployment, etc...)
The article mentions that Excel doesn't need much onboarding. I don't think that's its only advantage. Also, to solve the general software development problems around Excel, one has to match the near-zero onboarding barrier of Excel, otherwise those solutions will get left behind.
Sharepoint had some questionable synchronization corner cases that, in a poorly configured environment, would just lose some people's files. There are plenty of international horror stories where a teammate in Japan checked out a file and forgot to check back in, effectively blocking teammates in Europe and the US from working on the data.
The main advantage of Excel, which leaves us pessimistic on the long-term viability of the niche SaaS solutions, is the "hackability". There's an incredible depth of tooling and market expertise around Excel, from mundane formula structures to macros to addins and COM automation. Thousands of office workers who otherwise have little programming experience are extremely adept at solving problems with Excel.
The SaaS solutions might individually solve different problems, but are not really designed to play nice with each other (nor are they really incentivized to work well together).
That was quite a while ago. My company has been using the O365 for the past two years and it is very reliable, plus you get the full experience and not a half baked Excel such as Google docs(talking about backwards comp, extensions...).
There are plenty of international horror stories where a teammate in Japan checked out a file and forgot to check back in, effectively blocking teammates in Europe and the US from working on the data.
Then don't have people check out files and check them back in. Only programmers like that kind of nonsense! 99% of the time, let people "nominate" a particular copy of the file as authoritative. If something happens to that particular person's copy, let's say their machine gets smashed, then the latest version can be downloaded by their coworker, and an admin can nominate that copy as authoritative. People then go back to the 99% case of just using Excel and saving their file.
The main advantage of Excel, which leaves us pessimistic on the long-term viability of the niche SaaS solutions, is the "hackability". There's an incredible depth of tooling and market expertise around Excel, from mundane formula structures to macros to addins and COM automation. Thousands of office workers who otherwise have little programming experience are extremely adept at solving problems with Excel.
Therefore, a SaaS solution needs to let people do what they've always done, only better. Don't try to make them act like programmers. Don't change their behavior one iota, unless they're getting a payoff of nifty functionality in exchange. As much as possible, just let them act like they've always done.
The SaaS solutions might individually solve different problems, but are not really designed to play nice with each other (nor are they really incentivized to work well together).
One SaaS solution which can solve dataflow and versioning with ultra-low friction could be the one ring to rule them all. All the other SaaS solutions could be built on top of that. The problem is keeping everything super low friction.
FWIW, one of the biggest fast food franchise chains in my country uses a custom Excel sheet as their main backoffice app across the organization. From what I can tell, it's far from a single user app with a small dataset.
Yes, it can be done with a lot of care, but perhaps needing more care than a "real" CRUD application would need. Data normalization, ACID (transaction integrity), indexes, server-side validation, roles (user access levels), etc. take more effort to put into a spreadsheet (or add work-arounds for) than a regular CRUD stack (such as LAMP).
They are great for small projects and prototyping, but just don't scale well unless you want to pay for expensive babysitting.
Note a good CRUD app allows exporting of data to Excel for searching or experimenting with budget scenarios etc., but the "original" data should be in a real database.
The eternal struggle to find balance between spending all your time type checking, compiling, linting, Rusting, testing, unit testing, CIing, etc and actually making working code.
Data normalization, ACID (transaction integrity), indexes, server-side validation, roles (user access levels), etc. take more effort to put into a spreadsheet (or add work-arounds for) than a regular CRUD stack (such as LAMP).
All those things take less effort if you already know how to use Linux, Apache, MySQL, and PHP or Python. If you don’t, but do know how to use a spreadsheet, it takes vastly less effort to do those in a spreadsheet than to learn how to do those with a LAMP stack.
And there are orders of magnitude more people in the latter group than in the former, and their time generally costs much less on the open market.
Which, to be clear, I don’t think is a good thing! Doing those things with a spreadsheet is worse in every way—effectiveness, maintainability, you name it—except this one way. But it’s usually the only one that matters.
My dream, coming soon to an Internet near you, is to “democratize” the things that come easy to us “real programmers”.
The tools to scale excel are essbase or hyperion where excel sheets can all points to a central database/cube of data that can be refreshed and updated centrally and then the sheets can do local analysis and processing; sometimes even pushing data back to the central location (for controled superusers). The finance groups of most fortune 500 companies use these tools.
One of the keynote speakers at Lambda Days in 2018 said something that I've agreed with for years: Excel is arguably the best programming language, largely because people are programming without even realizing that they are programming.
It’s because instead of seeing a set of instructions inside a loop, you see the instructions left to right and the loop iterations top to bottom. It’s fucking fantastic.
Yeah, no argument here. I tried getting my wife to learn how to program in Go, and while I think she's certainly smart enough to do it if she stuck with it, it wasn't really her thing.
About two weeks later, I showed her how to use Excel, and she took to that immediately, and uses it for nearly everything. It's to a point now to where I wonder if she'd pick up Go better as a result of her experience with spreadsheets.
Spreadsheets aren’t without code, the code is just in cells.
Spreadsheets are not functional.
Spreadsheets are not a database for any sensible definition of database. If you losen the definition enough to include spreadsheets, then random scribbling of data into a .txt file is also a database “because the data is put into lines” like spreadsheets puts data/code/comments into cells.
Excel combines database principles of keyed list values with the non-database principle of locational calculation and canvas page layout.
This advantage shows when using Excel positionally rather than as a simple data table system. For example, =B7*C6 is virtually impossible to express in SQL, and constructing positional lookups using concepts such as “the previous row” or “the next column” are virtually impossible - yet, easy in Excel:
INDIRECT(ADDRESS(1+ROW(H229),COLUMN(H229),3)
Excel’s ability to write non-tabular data and calculations alongside tabular data, without needing to split it onto another tab/pane/page, is another critical benefit that databases cannot match. While they have stored procedures and linked functions, they have no concept of “positional” with which to model the theory of canvas that underlies Excel.
Finally, due to being a canvas model with calculation functionality, Excel meets the basic needs of a page layout and data graphing engine, with the ability to configure and position reporting in various UX ways that are considered out of scope for nearly all databases. HyperCard, FrameMaker, MS Access all clearly understand this issue and work to solve it in various respects.
So while the written definition of spreadsheet may seem to be encompassed by the word “database”, that is true only if the word “database” encompasses canvases that store both tabular data and other content, and not just key-value data (such as a MongoDB or anything SQL).
If you want to break into the Excel market, you won’t do it with a database alone. You’ll need canvas UX to even begin to be relevant, a point many startups discover the hard way.
The structure in a spreadsheet is “all the stuff is in cells” and in a raw binary file it’s “ all the stuff is in a binary blob” in a txt file it’s “all the stuff is in lines in a file”.
As I wrote initially, if you losen the definition that much, then a txt and literally everything is a database.
There are many different definitions but my personal completely subjective perspective is that the bare minimum is that whatever structure you claim the data has must be enforced and be translatable to a semantic connection. For most sql databases you have a schema enforcing the structure and at a minimum there is a connection between different pieces of data in the same row.
In a spreadsheet having different sets of numbers/text/code on the same row or column might mean something but doesn’t have to. So whatever “structure” there is, is not enforced.
This is in the same sense that a txt document that contains 4 book chapters have in them 3 email addresses is a “structured list of emails” sure you could find them and there is a structure to it, but calling it a database is a stretch in my book.
What about a document database without an enforced schema? For example, you can just install CouchDB, create a new database and start inserting random documents.
>random scribbling of data into a .txt file is also a database “because the data is put into lines”
There are plenty of computer programs that use text files as databases, reading and writing lines. /etc/passwd is one very common example, the passwd file is just a text database that you can read with any text editor.
Yes, but by that definition every group of bits on a computer is a database. The word sort of loses any value.
I mean a HN comment is also a database, a text is a database, a picture of rocks arranger i the shape of names is a database.
In the world of stupid internet arguments, sure we can agree you are right txtfiles are database, .wav files are database, everything is a database. Talk this way about it at a job interview and you’ll get cut by your second sentence
I understand what you say. I mostly agree, but just wanted to push the semantics to the limit ;)
>> Yes, but by that definition every group of bits on a computer is a database. The word sort of loses any value.
Not exactly. We need to define the atomicity we would like to deal with. That is:
- if a bit is what the atomic unit of data is, then a byte is a database
- if a byte is what the atomic unit of data is, then a collection of bytes in properly defined structure is a database. For example, a JPEG file or little-endian or big-endian datatypes.
>> I mean a HN comment is also a database, a text is a database, a picture of rocks arranger i the shape of names is a database.
If a word is the atomic unit, then a comment can be a database. If the comment itself is an atomic unit, then the thread can be a database.
As the author of a specialized inventory management application (a small ERP/MRP for electronics production, https://partsbox.io/), I definitely agree. Spreadsheet is my biggest competitor, but also a great opportunity: while you can start tracking inventory using a spreadsheet, you always eventually outgrow it, which is my opportunity.
A lot of those SaaS end up with end users dumping data down into Excel at some step, but they definitely can be wins for workflow standardization and error reduction versus trying to do it all in Excel.
Definitely lots of truth to that. The goal to riches is just finding a niche thats big enough to create a software company for, but small enough that someone like salesforce won't show up and crush you.
Excel is not a competitor to our SaaS - it is an interface. Original our service was web and email only (energy market financial data). But we quickly learned our clients want and need that data in CSV so they can build their own tools. So our API integrates well with Excel.
And that is the power of Excel - a client can build their own solution, backed by our service. We do the parts that Excel can’t do or do well.
This frees us up from the pressure to build customization for any client - we won’t do customization only generalization.
We will create new features without any UI - just the API for JSON and CSV. This lowers our costs and development time - UI can be expensive and time consuming.
This is exactly how we use Excel where I work (a large multinational contact-center provider). Data is stored in SQL Server and then consumed in Excel in several ways:
- Direct query from Excel to SQL Server (This is fast, reliable but users need to know SQL)
- Cube pivot or formulas to SQL Server Analysis Services (Lower user skills needed, but depends on IT/Dev to setup correct measures and processes in Analysis Services)
- Using MS Access as an intermediary between Excel and SQL Server (Using MS Access in the middle is a great way to allow people that don't know SQL to build some simple joins. Comes at the cost of query performance though.)
- Using MS Power BI to connect to SQL Server data (this is in its early days at my company, but is proving to be a great solution for historical reports, where Excel usually shows its limitations)
I'm surprised neither the article nor any of the HN comments have (so far) mentioned Airtable.
Airtable seems to have the main advantage of Excel (it can be customised by a non-technical user) whilst solving some of the limitations (e.g. multi-user, permissions, workflow triggers).
Someone who would have built their CRM tool in Excel in the past, might well choose Airtable today.
Airtable looks great, but seems incredibly expensive at $10-40 / user. There are a lot of situations where pricing like this makes no sense, e.g., if you want every employee to have access to your data, but most of them only check or enter data once every few months.
That’s my impression as well. I use Airtable, as well as Notion, personally in a single user environment and could see countless use cases for our team at work, but Airtable is simply too expensive and even in its highest tier the number of allowed records per base seems limited, or “increased” according to their pricing table.
I don't know why nobody mentions this, but if you DO unbundle a feature from excel, it is exceptionally hard to compete on ubiquity. Excel is cheap, and even if you replace part of the functionality, nobody will ever actually beat the price.
Personally, I wish I had an excel-like interface for Python/Pandas. Instead of sheets, give me modular data frames, and give me a nice GUI for data import/export/connection.
But, the best feature of excel is that the interpreter is built in. No version control, no wondering which version of whatever library is installed. If it is excel, it (mostly) works when I send a file to someone.
Most of the time, I use excel because the SaaS tools that I have that do a better job have a per/seat cost that is MUCH too high for me to bring casual, one time people into the projects. If companies were more sensible about licenses for casual users, maybe excel would be less relevant.
> Most of the time, I use excel because the SaaS tools that I have that do a better job have a per/seat cost that is MUCH too high for me to bring casual, one time people into the projects. If companies were more sensible about licenses for casual users, maybe excel would be less relevant.
This is the main reason why I don't use products like Airtable or Notion, even though they are otherwise very compelling. Charging $10-40 for every user, even people who may log in once every few months (and maybe don't even change data) makes no sense. At that price, even excel files on a network drive is better.
FWIW, Airtable has sensible defaults for this. you can share a table for free (password protected) with non-airtable users. (I do use that for specific team projects)
I actually think there's an opportunity in the opposite direction: convert your SaaS apps to spreadsheets.
Instead of having to log into 10 different SaaS apps, you simply make a few menu selections from within Excel / GSheets and your data is pulled in, ready to be analyzed.
The spreadsheet becomes the UI, and the SaaS "apps" are basically databases accessed via an API.
This was my thought as I read through the comments. Email and spreadsheets are great, they’ll be around in five years. Almost nothing else is so dependable (besides having to pay taxes).
The company that makes many of the Google Sheets add-ons (https://www.ghostwrench.net/) that I use at work has made this their bread and butter. Google Sheets IS the interface to their software.
Spreadsheets are good and fine until they're not (fine-grained permissions, collaboration, referential integrity and validation enforcement, multiple data source aggregation and ingestion, easy visibility into history and auditing, features that Excel has but people don't want to learn, etc).
You have to convince customers that what they have going is not actually working, which requires having a discussion about how just because poor data management practices have thus far failed to completely destroy your company, they are wasting tons of time and money. Many of them will have in the back of their mind that they can always go back to how things were before.
So the behavioral change/change management aspect of a product that really does directly compete against Excel is very challenging.
Spreadsheets are good and fine until they're not (fine-grained permissions, collaboration, referential integrity and validation enforcement, multiple data source aggregation and ingestion, easy visibility into history and auditing, features that Excel has but people don't want to learn, etc).
All of the things you just mentioned could be handled using just two mechanisms:
1) An extremely low friction way of flowing data to/from spreadsheets
2) Automated publishing/versioning
If a particular piece of automation for flowing data between and in/out of spreadsheets is the lowest friction for users, then that will be the way they use by default. Then that software will be able to provide collaboration, fine-grained permissions, referential integrity, validation enforcement, multiple data source aggregation, ingestion, and auditing. Tracking versions takes care of the rest. The trick isn't providing all of those facilities. We've known how to do that for many years. The trick is to provide all of those with near zero friction.
What's needed is something like Dropbox, plus dataflow. Just let users save their files, and it gets versioned and uploaded into "staging" for the dataflow part without their having to do anything. Then, as people start running into issues of scaling, enforcement, etc, enable them to start incrementally adopting more administration features -- all while preserving the transparent "it just works" magic of their just saving the file.
It is extremely hard to convince someone they're doing it wrong, when what they're doing is working for them and is making them money. Its pretty much ingrained in our human nature, I think.
Instead of telling them their solution blows, its better to gain trust by first addressing the pain points without any major rework, and then slowly change the system bit by bit, with each bit providing an instant usable benefit to the end user.
I think there are intermediate steps possible in some workflows. For e.g. A business app that accepts an excel file as input for further processing. So now you can let them keep the excel file with their existing macros/features, but give them additional features via this workflow.
Does the author mixes up "spreadsheet" with "database" or "data entry" or even "line-of-business app" in general? There's more than spreadsheet in many of SaaS examples.
I actually think this article is late and we’re already seeing the next evolution of this trend: flexible SaaS.
Right now SaaS products are mostly single-purpose workflows and aren’t super customizable (until the start-up ages and enters a few enterprises who demand customizations). This is a step-up from Excel which was general purpose but didn’t facilitate workflows seamlessly.
However what Airtable is finding is almost any specific SaaS can be recreated with general purpose spreadsheet-like products. Essentially they merged the ease of use of Excel with additional shared workflows of SaaS.
How is AirTable not mentioned in this article? If Excel has any major startup competitor my money is on them. I use spreadsheets constantly and work with a lot of people who aren’t the savviest when it comes to technology. AirTable essentially lets them “program” even easier than Excel but also has sharing/snapshots of changes over time that everyone can see. It’s not quite a replacement for Excel yet but I absolutely see it heading that direction.
FWIW, I have seen (and have recommended) designing a number of applications with the spreadsheet as their front end. This works particularly well with google sheets, given its reasonable API. It is way richer than anything you will quickly build, is designed for mucking with tabular data (and makes it relatively easy), and it turns out a ton of applications are basically interfaces on tabular data. Hey hey!
Feels like a ton of business could just benefit from automating various spreadsheet tasks via Zapier. I mean this stuff exists and it’s not rocket science but people don’t care to figure it out. I think part of it is people also not wanting to replace themselves out of a job. Does Betty really want to take that 1 hr daily task and turn it into 5 minute weekly task? I don’t know... maybe not.
The problem with SaaS alternatives to Excel business processes is the difficulty of linking data between apps. Let’s say you want to answer some topical question, like “are code reviews for women and men written with a different tone?” You have a HR system that has people’s gender, and a code review system, and some Google API that measures tone. Good luck stitching this together between vendors. But dump it to Excel and make the API call from VBA and it’s easy.
The problem of course is that this becomes an important business metric, making the spreadsheet a critical business application, but without any of the tooling used for successful software development (version control, code review, test automation)
My anecdote.
I work with mining (resource) companies and am mining engineer myself.
I reckon one large company coming into our office and spending 2 hours describing their new software to assist with blast design.
They gave us a trial and while it was slick, it was not much better than our spreadsheet, that everyone can use and the few extra features they showed we easily implemented ourselves in the spreadsheet.
Hate to think how much money was spent developing this software that literally was a pretty GUI over a spreadsheet.
for those who feel the need to version control of excel sheet content (only values, not formulas) I published a tool (only VBA code) I wrote to export the content of every rows of selected sheets (a text file for each row of data, using a unique id contained in the table as filename).
Ironically, I work in a regulated industry and every software we use has to be compliant with certain government regulations. We would actually love it if we got to use excel in our business workflow, but we're forced to deal with some horribly ungodly mess of softwares that cost tens of thousands of dollars, that do nothing more than have the magic "compliant with X" sticker.
There may be a good reason for it. Excel may be quick to set up, but is not known for data integrity. You don't want important data being subject to willy-nilly changes.
Just because I can quickly make a boat with duct-tape and 2x4's doesn't mean I'm ready for an Atlantic crossing.
Spreadsheets could also be understood as just a fancy pen/paper for a manual business process. Looking at it this way, the competition isn’t excel but rather doing things by hand (i.e. a non-solution vs a competitor).
The spreadsheet is certainly one of the best tools ever invented.
[0] Joel's Spolsky must-watch talk on Excel: https://www.youtube.com/watch?v=0nbkaYsR94c