No it isn't. This is MY take and I'm not an elitist software engineer.
> People building on Excel isn't a result of failure to educate, it's humans doing what humans do best—automating their own workload.
No, it isn't. It is human behaviour alright, but it's cutting corners because it feels better to do what you already know instead of doing what was instructed. This is what we see in businesses from 100 to 1000 people happening time after time.
Example: if you need to visualise your data from multiple sources company-wide, we'd have Tableau (and training, access, templates, portals etc) for example. That means you do it there as per instruction, and you don't go trying to setup some home-brew graph in excel that you manually regenerate every day and forget to generate when you are on vacation.
> It's people using general-purpose computers as general-purpose computers. It's the closest to the personal computing dream that we've come, and likely the closest we ever will.
Sure, do whatever you want at home, no discussion there. But since I already specifically wrote that, I suppose we don't need to keep repeating that.
> The alternative in the real world isn't "everyone learns Python", it's "we lock normal business people out of computing and keep it in the hands of the trained and very expensive software engineers".
No, it's not. The alternative is follow the workflows your teams and BUs have established and don't go off on your own. It works, it's proven, it's highly effective and attracts talent as a bonus.
> That's not going to happen, and it's frankly not something we should want to happen.
Maybe not where you are, but it's definitely happening here. And we want it to happen because the amount of data and the types of data don't work with excel. And the way people have tried to work around it by doing sampling and then saying "but it works on my machine" when it inevitably fails is a waste of time.
> I think this kind of Excel denigration comes up so often in software forums because we're usually called in to rescue a business when their Excel workflow gets completely unmanageable.
Perhaps, but what I wrote isn't Excel denigration, or denigration in general, it's real world scenarios where Excel wasn't the solution and people found out too late because they didn't know any better and they thought they were doing the right thing. Heck, replace excel with spreadsheet or 'workstation-based' and you have the same issue.
> We miss the decades that the company ran very successfully without any software engineers on the payroll and see the giant spaghetti mess that made them finally decide it was worth the cost.
You must have forgotten mainframes along the way, that's where the actual money was made, not spreadsheets. And if you create a 'spaghetti mess', that is something you can do perfectly fine with Excel, Access or just plain pen and paper. Creating a mess is at the core of end-user deficiencies when it comes to using a spreadsheet for non-spreadsheet problems.
> But it's important to remember that these same businesses reached the point where they could afford to pay us to build something custom by building a successful business on top of Excel.
No, it's not. It's important to remember that older software was written to emulate the physical world, but that was also what created boundaries and limitations. Desktops, folders, spreadsheets, word processors, rolodexes, telephones, paper mail etc. are not the 'best' solution, it just happened to be what was used when software was written. The software emulated the processes that existed to make it easier to understand what it means and how to use it for the people of that era. Practically everything has evolved beyond that, to the point where you have to educate new hires on what a file and folder structure is about if you're using those at work. Just like you have to educate older people that the way they used to do things is no longer how we do it today.
In some ways, Excel is the COBOL of our generation. And that is not a good thing.
>Example: if you need to visualise your data from multiple sources company-wide, we'd have Tableau (and training, access, templates, portals etc) for example. That means you do it there as per instruction, and you don't go trying to setup some home-brew graph in excel that you manually regenerate every day and forget to generate when you are on vacation.
Then your CFO calls up and says, 'I have a meeting in 30 minutes, can you tell me the total cost of pensions for people who work out of the Singapore office on sales admin' and the BI developer never pulled the measures through for that, because nobody asked for it before... so you export it to Excel, do an xlookup, a sumif and a filter and keep your job...or put a request into IT to have it added, wait 3 weeks, and get fired.
In the real world there are measures repeated week after week, and there are many more ad-hoc requests. Your mental model of the information required to run a business is way off.
Exactly. When a process is important enough and repeated often enough, it will move into a more formal system. But most of the time spent running a business is at the fringes of what has been automated and formalized, and this isn't an accident—the automated processes are there precisely because they reduce or eliminate the human time. That means most of the humans (outside of front-line customer support or similar) tend to be working on stuff that is substantially less well-defined than what OP is imagining.
Which again gets to my point about the weird perspective on business that we have as software developers. We're called in exclusively to work on those parts of the business that are deemed sufficiently repetitive to be worth automating. That dramatically skews our perception of what running the business should look like.
That's great, but has little to do with the 'excel is my hammer, therefore everything is an excel-shaped nail' problem.
Sure, like I wrote (twice already), you might need to have some malleable rows in front of you on your local workstation for some ephemeral work. But the actual workflow happens in the ERP system. Or the CRM. Or the PIM system. Or in Tableau. Or Databricks. And my beef is with people who think their personal flavour of alternate software should bypass that, even when they have been explicitly told not to. When someone needs to stream the truth (data-wise), that comes out of those systems, not some kludgy sheet on someone's network share. Those are the rules that we set up.
I don't understand why this is so difficult for everyone to understand (on HN of all places). These are general staff functions we practically have entire schools for, to make people work this way.
I love that you mentioned PIM systems. I've worked with product information systems from small 3-man companies all the way to Fortune 500 companies, and without exception, their PIM systems have been total dogshit: slow, extremely unintuitive, limited in functionality and borderline unusable. When I need a list of product names, barcodes, weight and dimensions, nothing beats a simple Excel table that can be quickly transformed into whatever format I need at the moment. With the intensity of a thousand suns, I hate the "workflows" of having to create an account, sign in, browse products, put them into a cart, submit an order, and then wait for an email with a download link to some non-standard XML dump or broken CSV that their multi-million dollar PIM system produces. These systems have wasted entire lifetimes worth of their users' time. When it comes to sharing product information, nothing beats the ease of use and flexibility of a plain Excel table.
Our live PIM database is over 2TB, we're not going to copy that to excel. On top of that, we're also not going to have everyone do a little bit of PIM here and there, we stream 80 to 90 PIM mutations per second across various systems. You get a PIM frontend with appropriate access and that's it. Technically, the only real PIM work the teams do is corrections, there is practically no manual entry.
Luckily for us, we put request-response time in the requirements documents, so our PIM users get sub-50ms responses for everything.
As for sharing data, our manufacturers and subcontractors use APIs (with the exception of some wanting JSON uploads via SFTP), most of them stream directly into their PIM or ERP (yes, some don't have dedicated PIM systems and just use their ERP for that).
Even if we could have the data flow through individual workstations, we'd still not do that as we don't want to legal and repetitional risk associated with that, especially since it is completely avoidable.
As written in a different reply, sometimes people want to do some local fiddling around and they will export some data. But that data is considered expired, and not part of the normal data flow. Which was my point: spreadsheets are never part of the main loop. And anyone who wants to build a system where it is, is not working for us.
I do notice a trend here, lots of people having bad systems (slow PIM, outdated ERP, missing CRM), which is probably something that should be fixed, rather than taking the shadow route. Escaping to a spreadsheet is just that: escaping.
Our PIM team disagrees with that take. They enjoy the PIM interface and Excel doesn't come close to the data hierarchy and relationships our systems provide. And that includes reporting, vendor exchange and internal integrations.
But perhaps you're missing the point here (like plenty of others): it is not about handling a few offline rows, it's about re-inventing established systems that have proven their value and risk mitigations. We don't want someone to setup a collection files to build a homebrew PIM or ERP or CRM because they feel like it, disconnecting themselves from the organisation and teams they work with. In most cases, single player data silos are bad.
No, they don't. Our finance teams, accounting, B&S, purchasing etc. also all disagree with your take.
There also is no "fireable offence" structure here, we aren't a US-company and we have strong labour protections here. They themselves chose how they want their processes to work, and none of them ended up with Excel in the main process loop, anywhere.
Perhaps this is something you cannot imagine, but over here, this has been the standard for years. And it pays dividends.
This does not mean excel doesn't exist or isn't used outside of the main processes/loops/flows. But that is what I wrote in my first comment here, which people seem to skip over.
In the real world (which is where my example comes from) the CFO expects it in Tableau, all day, every day and isn't going to waste time on the phone with some random people who were assigned to put it in tableau. Perhaps you don't like this example, but that doesn't make it less real.
Ok. My real world counterpoint. I work for a CFO at a multinational. A really big one. I have been the finance lead at a multinational myself, a smaller one.
Our business changes so rapidly that we are forever adding business units, acquiring, divesting, resizing. The ERP people are years behind us all of the time.
What should we do in the mean time with our new group of companies we just acquired? Wait for the IT people?
If your ERP can't keep up, then that is an ERP-shaped problem. Escalate with the CFO and CTO since the latter probably signed the responsibility/ownership (albeit probably indirectly) for delivery. In the mean time (since the real world is imperfect) you would of course rather kludge it than do nothing. But you would probably not rebuild the entire ERP in excel, forever, which is what I will fight against.
We don't have that problem, our ERP team can keep up. Perhaps that is because our rate of acquiring and divesting is maybe 2 or 3 cases per year. We are only active in a couple of countries per BU in the cases I write about, so that makes it easier as well.
On the other hand, if the ERP team isn't capable of producing the rate of change needed to support the business requirements, I really hope that you get that sorted since that is practically the entire reason an ERP team exists.
We do deploy changes (technical changes) about 200 times each day, of which the teams dealing directly with business workflows do about 60 (including some stock, pricing and WSSI teams). It's not a lot, but it allows us to have nearly no lag time between business needs and implementation delivery. It did take a while to get to that rate, and a 2-week feature freeze one time to shift all processes to a model that allows for this change rate. But it works well for us. (and reporting, visualisation, ETL, and ML are all done without spreadsheets for about 3 years now)
As for why we did that (in case that was a question you'd be asking since it is obviously not a change that costs nothing): we no longer wanted to accept the risk of doing it the old way, we also didn't want to have as much onboarding friction due to having many diverse sub-workflows inside teams, because even if you did learn about what someone DIY'ed in your local team, you will still run into the same problem ad-hoc when dealing with other teams, usually at the worst possible time. Same goes for off-boarding, we don't want to spend months trying to figure out what custom stuff someone put in place, especially since you get non-engineering solutions to engineering problems (the excel shaped one) but also engineering solutions to non-engineering problems (the python notebook one). We want our people to be able to spend as much of their time on the actual work instead of side quests that don't add the value we are looking for. It doesn't mean exploration or experiments are banned, but it does mean we want to facilitate as much as we can so you don't have to do anything extra by default.
See my other comment, we have over 1000 factories in many countries. 23 different ERPs in my division alone apparently. They are constantly trying to standardise, but acquisitions happen faster than factory ERP implementations. When you acquire a group that isn't standardised itself you have even more systems. That is one reason why complex groups use consolidation systems.
I'm guessing your company is somewhat more homogeneous than us conglomerates?
Edit: no it is not an ERP problem. It is a people problem. The team simply can't scope out the new factories quickly enough before the business changes something. Often the newly acquired factory has a different costing method for instance, and lots of data has to be collected before they can go on the master ERP. This may involve, for instance, timing every products build, collating and verifying it (in Excel probably), before the implementation can progress. This can take months. Especially when.... three main priority is selling stuff, rather than having neat BI
At the end of the day, everything is a people problem. I tried checking the other comment but I couldn't find it (I thought I replied to it actually but it seems it was a different one about non-integrated ERPs).
We are not a conglomerate of 10+ nation size (slightly below that; also not the best measure but trying to toe the line with sharing but not naming gets harder with detail) but I suppose since we had to pivot from financial services to CPG a while ago and B2C retail in our segment at that time really demanded extremely low lag&lead-times for user changes and onboarding of products and services (well, VAS.. but who's counting), we engineered our way out of it, including logistics, SCADA, web, mobile apps, people-onboarding and M&A onboarding.
Having neat BI is more a side-effect than a primary goal. That goal came much later since it was the last remaining driver of shadow processes as it lead to many opportunities of bad engineering, even if it was seen as valuable in the short term (it never was worth it in the long run). It wasn't even about spreadsheets (or excel specifically), it became a real-time streaming issue where you couldn't have a fully manual process in the loop any more and we couldn't hire extra people fast enough to keep the existing processes running at the rate we needed. We don't have data entry positions anymore for the same reason; it's not that we don't receive new data, it's just that we had all B2B contacts agree that they will not send or receive files between humans if it's regarding an automated streaming process.
Having our data architecture being based on real time streaming (and some periodic sanity checks and shadow reconciliation as that is mandated by some record keeping laws) means we beat practically everyone in the same market segment on speed and delivery.
But perhaps this ties in to us being more homogeneous than some other conglomerates. We're not doing more than 5 countries and don't have our acquisitions result in extra divisions as we tend to aim for competitors in the segments we want to grow in (we want the data, the people and the customers, not the systems). This means that the people that operate the business (be it the business processes or the actual product processes) don't really get 'extra' software as a result.
No it isn't. This is MY take and I'm not an elitist software engineer.
> People building on Excel isn't a result of failure to educate, it's humans doing what humans do best—automating their own workload.
No, it isn't. It is human behaviour alright, but it's cutting corners because it feels better to do what you already know instead of doing what was instructed. This is what we see in businesses from 100 to 1000 people happening time after time.
Example: if you need to visualise your data from multiple sources company-wide, we'd have Tableau (and training, access, templates, portals etc) for example. That means you do it there as per instruction, and you don't go trying to setup some home-brew graph in excel that you manually regenerate every day and forget to generate when you are on vacation.
> It's people using general-purpose computers as general-purpose computers. It's the closest to the personal computing dream that we've come, and likely the closest we ever will.
Sure, do whatever you want at home, no discussion there. But since I already specifically wrote that, I suppose we don't need to keep repeating that.
> The alternative in the real world isn't "everyone learns Python", it's "we lock normal business people out of computing and keep it in the hands of the trained and very expensive software engineers".
No, it's not. The alternative is follow the workflows your teams and BUs have established and don't go off on your own. It works, it's proven, it's highly effective and attracts talent as a bonus.
> That's not going to happen, and it's frankly not something we should want to happen.
Maybe not where you are, but it's definitely happening here. And we want it to happen because the amount of data and the types of data don't work with excel. And the way people have tried to work around it by doing sampling and then saying "but it works on my machine" when it inevitably fails is a waste of time.
> I think this kind of Excel denigration comes up so often in software forums because we're usually called in to rescue a business when their Excel workflow gets completely unmanageable.
Perhaps, but what I wrote isn't Excel denigration, or denigration in general, it's real world scenarios where Excel wasn't the solution and people found out too late because they didn't know any better and they thought they were doing the right thing. Heck, replace excel with spreadsheet or 'workstation-based' and you have the same issue.
> We miss the decades that the company ran very successfully without any software engineers on the payroll and see the giant spaghetti mess that made them finally decide it was worth the cost.
You must have forgotten mainframes along the way, that's where the actual money was made, not spreadsheets. And if you create a 'spaghetti mess', that is something you can do perfectly fine with Excel, Access or just plain pen and paper. Creating a mess is at the core of end-user deficiencies when it comes to using a spreadsheet for non-spreadsheet problems.
> But it's important to remember that these same businesses reached the point where they could afford to pay us to build something custom by building a successful business on top of Excel.
No, it's not. It's important to remember that older software was written to emulate the physical world, but that was also what created boundaries and limitations. Desktops, folders, spreadsheets, word processors, rolodexes, telephones, paper mail etc. are not the 'best' solution, it just happened to be what was used when software was written. The software emulated the processes that existed to make it easier to understand what it means and how to use it for the people of that era. Practically everything has evolved beyond that, to the point where you have to educate new hires on what a file and folder structure is about if you're using those at work. Just like you have to educate older people that the way they used to do things is no longer how we do it today.
In some ways, Excel is the COBOL of our generation. And that is not a good thing.