And so here we are now.
The much harder problem is data management and preparation. Anyone with half a brain and a decent visualization tool can create basic graphs - but that doesn't mean they should, especially if the organization doesn't have good data management processes in place.
Issues like data governance, data prep, and data modelling are the major pain points for me. And honestly, developing a useful BI solution is more about culture change than it is technology. If a company has poor data governance, it doesn't matter how whiz-bang their technology is, they're still not going to get useful insight from their numbers.
There is something potentially harmful, or perhaps that needs addressing, about end-user tools growing in expressive power. A good friend who does statistical genetics work once told me "but I don't want every user running their own regressions and drawing nonsensical conclusions from badly prepared data!"
BTW, that is also the reason I not only set alerts, but review data at a daily level when pulling any rolled-up reports of significance.
We're in the middle of doing this ourselves and would like to hear a random HNers thoughts on the subject.
The other problem is that most executives don't even know what kind of graphs/reports they need. Sure, they can have the classic sales by region and stuff, but the really important things are hidden and require a lot of digging and asking to get to.
When something needs to be regularly produced, then and only then should it be moved to a formal BI solution.
People have a lot of questions and Excel is an approachable tool for small analysis that users can self-serve with.
everybody knows the correct answer is jupyter notebooks, and totes everyone like uses them, that's what it said on my facebook the other day.
NLP interfaces in reporting work for simple things, when you want to start doing more advanced things they become incredibly verbose and inconsistent.
That's where a well designed UI really shines...and will continue to do so.
structured language > purpose-specific UI > natural language (processing)
Old adventure games were really illuminating with respect to limitations of natural language interfaces. Rather than allowing for the expressiveness of natural language, you simply had the feeling of groping around like a blind child at an easter egg hunt for the few valid operations that existed.
I feel like this using GUI tools for data tasks. It's like being forced to communicate by banging rocks together compared to, say, the expressive power of R or Python.
I wonder if natural language interfaces could leapfrog GUIs if we all up and learned something like Lojban...
Really? That's 3 clicks a way with a well-designed UI.
And what if I have those 3 fields in different tables? Even more verbosity...
P.S: My job has nothing to do with designing UIs, just to dispell any doubt :)
That being said, it's a start. SQL as natural language.
How about we just 'say' it ? At one hackathon, I had the idea of making an Alexa skill that would translate simple voice commands like 'graph column A' into matplotlib functions and then show the graphs. Teammates weren't too excited about spreadsheets though.
But I'm assuming lots of things, I can also envision your scenario playing out to great annoyance.
You don't necessarily need to make noise to speak to computers.
I started typing in a question but it couldn't guess what I was interested in. YMMV. Perhaps with a fairly simple spreadsheet you can intuit things? Back 10 years ago I built a google spreadsheet competitor called Numbler (well, I didn't know if was a competitor, google sheets came out a couple of months later). But one of the things I learned is that people use spreadsheets for just about everything, and it can be in the wierdest format.
That is so true. I used to do consulting for small businesses and was amazed at how non-programmers used Excel to solve so many problems outside of finance and accounting. The CRM/HR solutions were very common and interesting (e.g. Lead/prospect management, sales, timesheets, vacation schedules, etc.).
I built a private Add-on for my company that surfaces specific aggregates as Sheets functions (i.e. getSalesByDay(...)) and I have found so many bugs with that whole ecosystem. Deploys are completely manual and require copy-paste, you can't reliably tell what version is being invoked in a sheet, invisible cell-level caching that caches error state, concurrency limits that are too low and impossible to work around, and more. It all kinda sorta works but Google doesn't make it easy.
For our non-technical employees I write an endpoint that gives them a CSV with the most current info (often with a super simple front-end that they can use to query for specific date ranges or with filters). They download this CSV and upload it into their sheet manually to update the underlying data when they need new info.
It would be so great if you could just say "here's a URL; keep my data fresh from this source" and it would automatically do it.
This is one tool in our tool belt for communicating with data. We also have an in house SQL-in-chart-out platform that everyone has access to and we have Looker for deeper exploratory work.
I wish that Google would take the same sort of "embed" idea further in G-Suite. I find it amazing that I can't (as far I know) reference slides from another deck in Google Slides. The use case would be putting together a series of "core" slides that are updated across your organization as they change. Given the web nature of G-Suite, this, to me, would seem like a no brainer.
Also, inserting charts from Google Sheets into Google Presentations looks pretty terrible. I often revert to Excel because the charting is fair superior imho (though just as challenging to wrangle).
I don't understand the example, what's the difference between typing "Show me a line graph" and clicking a button in excel that does the same thing.
I found it immediately useful. They've solved some UX and discovery issues around creating charts. And it's not _just_ charts... they're answering questions and identifying trends within the data. e.g. I threw some pretty basic data at this and it told me “Flights” contains a yearly cycle: “Flights” increases until May 1, decreases until October 1, and increases until December 1.
That's pretty handy. YMMV, but this is awesome.
- Query data in Google Sheets from BigQuery
- Create virtual views in BigQuery that are powered by Google Sheets
- One-click export data from BigQuery to Google Sheets (< 20k rows or so)
- Using AppScript, build dashboards and reports in Sheets that query BigQuery for results.
(work at G)
If developers had a quick way to create properly engineered applications from a working workflow involving a spreadsheet, it would be easy to define most company processes on Excel or Google Docs and turn them into solid software.
But building an application that is functionally equal to the spreadsheet, just with robust engineering practices, typically involves several-months-long projects with many developers and managers, which is expensive.
I did come to really like the "Trace Dependents" and "Trace Precedents" features in Excel! :-)
At least they are functional reactive, so it's easy to trace the evaluation of expressions.
Google's "app maker" looks like their attempt to do something similar. https://developers.google.com/appmaker/ Though they seem to have left out the 'start from a spreadsheet' part.
can you explain this to me? i've never used spreadsheets extensively so i'm clearly ignorant but my impression was that the only thing you could really (i know you can bend the rules etc.) is data transformation. when you say application is what you mean? i.e an ETL app or do you mean more than that?
The more that's been done in Excel, the more pain it will be to maintain it in Excel or port it to a non-Excel platform, because Excel is seductive for write-once, but horrible for maintenance, and the people in an org who have it as their main tool are unlikely to be either cognizant of the problem or devoted to spending effort mitigating it.
You opened the "spreadsheet", which hid Excel itself and showed a form. You filled it out, clicked Send and it used your local Outlook client to email a copy of itself to your boss, who then opened it and was presented with another form for approval, etc, up through finance and finally to the person who wrote the check.
It was amazingly ugly and fragile, and of course, there was no way for anyone to see the state of the system (because the 'state' was distributed among the entire workforce's Exchange inboxes!).
Want to know the status of your check request? No problem, simply call everyone who might have been in the approval chain and ask them to search their inboxes. Hope you find the most current person!
(updated to correct the name of the software)
When I started, almost every process was based around Access, using queries made using the query builder. With some basic scripting I managed to cut a couple of man-months of work down into about an hour or two of work a month. The entire thing is more fragile than a stack of cards though.
At this point I could literally write a 200 page dissertation on all of the reasons why this is a bad idea. But it works, and it is cheap. So there you go.
An excel form / outlook workflow would have been an improvement.
I miss using machinations in MS Access back in the day. Now with Google Sheets, I am left wonting for a tool that enforces primary keys and is /easy/ to set up a view or query.
The definition of machine learning that I use is: an algorithm which improves its performance through experience.
So, if charts doesn't get better the more you use it, it's not machine learning.
Does it perform NLP on company documents?
My assumption is that it works on metadata coming from a relational schema with a rule parser on top.
Honestly I don't see how it would work well enough if it was based on NLP of unstructured data.
But the best, still-mostly-hidden feature I've found recently is App Scripting and especially the ability to do a UrlFetch.
I use it as an "API Runner" to run various batch jobs against APIs.
It's fairly trivial to get this to work with well-formatted data.
I'm currently evaluating BI solutions for my company and just about every single one has something like this.
Google used to say that everyone who did an RFP chose Google Apps/GSuite. The problem is that nobody does that!
I'm not convinced it's better just because it has machine learning on the back end, but if excel would learn how I want my graphs made from how I manually adjust the graphs (adding axis labels and a title, color preferences, never a 3d bar or pie chart), that'd be a nice enhancement. I'm sure there's a setting, but I haven't searched for it.
You would message the bot something like "sessions for this month", and it would send back a graph.
wonder if you could make a similar bot with google sheets if they provide an api
has anyone worked on something like this ? the big challenge is synchronization - between server and multiple clients - while being able to offload a lot of computations on to the client.
I wonder how is the security built ? if i maliciously change the formulas in my browser.. will the backend datastore still accept the data ?
Implementing 70% of the most basic of all functionality of Excel is easy. Implementing 90% is insanely difficulty.
I would say Handsontable reaches the mark of 78%.
I've tried twice, one with React + Mori and one in Cyclejs. In the first I may have reached 82%, in the second probably 85%. But in both cases the latest features I added started to conflict with the oldest ones and I abandoned everything.
We use HOT, but have been considering aggrid. Because of these tools, that's not the hard part any more..It's the synchronization and computation logic.
We need open source implementations for stuff like that.