"I've invented a program that allows non-programmers to input loosely-typed semi-structured data without needing to define a schema. It is automatically human-readable, and data can be simple code that can act on other data. It doesn't scale too well, but it can export to CSV and other standard data formats, allowing you to build real systems with it once the non-technical people have finished their logical prototyping."
People would be singing its praises and proclaiming the death of SQL for umpteenth time.
The one good thing why software and platforms are created is control. The only challenge is how can someone create a seamless web 2.0 spreadsheet almost similar to Google spreadsheets but definitely with more flexibility as Excel's and more security-enabled features like any other enterprise software.
Yes, I saw that startup idea below. If someone wants to do something about it, I'm in like Flynn.
They're an online app for you to build a database like creating a spreadsheet form. And the created app is used just like a spreadsheet.
This pretty much mimics the spreadsheet behavior of:
1. Create a spreadsheet and put all the field names in it
2. Have other ppl fill out your form
But it has all kinds of functions of a database, like creating relations (named differently) and arbitrary complex, nested data schema.
Let's take an example. If the user wants to stick some free-form text in between the end of sales records for one year and the start of the next, they are free to do that. In any less free-form application, they need to define a "comment record" or similar, and they probably need to get a programmer involved to do that. And although the SaaS web version of their "database" may have a better interface, in a lot of cases, having to get a tech involved to make that sort of change is not a compromise people want to make.
And they have a point. It's not a slight against programmers, it's just that when they need to make that change, you'll be 2 companies and 5 projects down the line, and it won't be possible. The article mentions a sheet that has been in use for 15 years - if that had been made as a proper program, at that time, it likely would have been done as, say, a VB application, with an MDB back-end, and it probably would have had purple buttons. The source code would now be lost, and if the business process changed at all, the choices would be a full re-code, or working around it. I would be surprised if in 15 years time, we don't look at today's pet technologies in the same way that you just did when you read VB and MDB.
For me, the direction that Excel, and other spreadsheet, need to take is the same route that browsers needed to take when IE6 ruled the world. We need standardisation, and innovation. I've written a couple of blogs on this, and for me, the way to go is a central repository of extensions (http://edparcell.posterous.com/how-about-an-app-store-for-ex... for more on that). In the case of Excel "databases", it might be sensible to create an extension to standardise the management and creation of such "databases". It could even allow features like sharing data, backing up etc, but for that to still happen where users are comfortable: within Excel.
Seriously though, the problem is that any real solution needs to accommodate housing data internally that's accessible concurrently with backups. Once you do that you still need IT involvement, which still makes it 3 months slower than excel.
+ Most companies have share-point, and the newer versions allow users to define spreadsheets with rules that can be edited directly in excel yet still upload centrally for all users to access with backups.
I think you're trying to find a sweet spot where technology will overcome organizational bureaucracy. It's a tight spot.
Best of Luck!
All the 'Excel-replacement' solutions I've seen are forms based, so there's a lot of pointing and and dragging going on. That's so much slower than just using the keyboard.
Just observe how true Exel ninjas use it... the keyboard calisthenics is mind-blowing.
Perhaps an Excel replacement will need to have an Excel-like interface, or find a way to get rid of excessive pointing and clicking.
Perhaps an Excel replacement will basically be Excel, with a background script/plugin that scrapes that routinely scrapes the data and feeds it into a database. (Hey! Startup idea). The first entry in each column becomes the data field. The program is smart enough to take a sample data from each column, and use that to auto-pick the best data type. (Also re-checks a random sample periodically and re-defines data type if necessary). Bonus points for creating excel templates for consistency ('actions template', 'to-do list template', 'known bugs'....etc. Last one was a joke) and for easier definition of relationships between different spreadsheets (like for like data comparisons and relationships as long as the user enters the right info in the right cell). Now that Office has migrated to the web, this could be entirely web-based.
People's workflow visually to them remains exactly the same, which would overcome the biggest obstacle - unfamiliarity and resistance to change. Non-techies who need to make a list of customers, processes or actions don't know or think about headaches they are creating for themselves or others down the line. It's only the technically literate who would appreciate the problems of consistent, clean data, version control etc. This solution would begin to answer some of those problems.
This could even be automated by screen-scraping Excel running on virtualized Windows instances.
Users of Excel see its great potential and its intuitive interface and use it for things that builders of this product have not anticipated.
What's wrong here is that builders do not follow users and don't try to improve their product to better fit what the users are using it for. Excel should turn into actual database many years ago.
Personally I'd like to see Excel migrate in this direction: http://dabbledb.com/demo/#play
Isn't that the kind of thing the OP is asking for?
It is not going to happen for several reasons.
One of them is that the company just got acquired by twitter.
I hope they release the code as open source...
Interestingly, I did not start out to build a replacement for Excel, but to offer customization features for my CRM app. Removing the default CRM entities (or "tables", if you would like to call it that), provides a slate wherein you can build your own custom database apps.
Like, a bug-tracker :
Or, may be, a property management app :
Ill chuck up a review my startup thread in the next day or so when we get the website / documentation cleaned up, but would be happy to hear what anyone thinks.
Also, I never thought schools, PCs or technical manuals did an adequate job of explaining what the difference is between a spreadsheet and a database. Usually they just say 'database is for large amounts of data' but as we know businesses have enormous spreadsheets anyway. I'm not even sure how I would explain it without referring specifically to relational databases and saying something like 'think of a database as like a 3D spreadsheet...' and mumbling a lot
Also, I'm building everything on top of Django with plans for an Export App feature that will give people portability. Build your app and tweak it with us, then export and host it on your own servers if you want (although you'll lose the ability to customize it without getting into the code).
All in all, I think there's room in the market for a lot of different approaches, since everyone's problem is a little different, and everybody's data is different.
Django already is an easy way to create web based database applications so what does AppRabit add to this?
I'd love to find a better solution but excel so far is still the cheapest/easiest to train.
Accessing patient data is tough because it's tough to model in a database. It's tough to model in a database because every single vendor exposes data in a slightly different way. HL7, the protocol that is used as an industry standard for this sort of data, has several flavors and everyone implements it their own way.
Also, the industry as a whole has no incentive to move forward. Healthcare providers (aka hospital systems) don't really want your data to be portable because it makes it easier to keep you as a customer. On the other end of the spectrum, FDA regulation has scared the big players into spending 5 hours on documentation for every 1 hour of software written, meaning it costs $20-50,000 (depending on the salesperson you talk to) for a hospital to buy a server that handles a few hundred beds' worth of patient data.
Needless to say, I'm glad I left.
The cool thing about that program, was that it was based on an OR framework, so all we had to do was to map it to a particular schema, and it could work with anything. It was implemented in Smalltalk running in a web plugin. It would be easy to do as a webapp in any dynamic language with a good OR framework and chart graphing framework.
From the Excel systems I have seen a lot of the business logic is coded at the VBA level. So without full VBA support you only have half the a solution.
We met with a group of spreadsheet support people once at a fairly large company where, as is typical, IT was trying to force the users off their spreadsheets. Here is how they said users were actually using the spreadsheet-replacing-proper-enterprise-apps that they had been mandated to use:
1. Open the enterprise app.
2. Paste the data into Excel.
3. Do whatever they want.
Unfortunately, I'm away from my notes, but my latest thought was this:
1. Able to import basic excel data, even if an MVP can't deal with VBA. Support the basic types of Excel, and be able to interact with the data in a spreadsheet sort of way. (Much like Access.)
2. Don't make IT afraid of this app. Imagine the technical support requirements as people lose data, do something weird. It's a burden to IT. So, it has to be dead-easy to use (Read: like Excel) with incredible documentation (video, audio, tutorials, dead-tree), and has to give buy-in to IT that it will make their lives easier, not harder. Judging by my experience working in shops with small IT, this is really hard.
Or, it has to be for very-small businesses without a major IT presence. Then, it has to be cheap, because they already have excel. This looks to be the biggest barrier! You have to convince the client they have a problem. To me, this looks like the wrong market, though the one I started to look at first.
It's a sexy market. I'd still love in on it, and the people who finally figure it out will be a billion dollar business. However, more people than you know have tried it. It's a deceptively hard problem and I've been thinking about it off and on for over a year now.
I'm starting to think it should look a little different. Take care of an enterprise app framework for small businesses that takes care of single sign on with easy LDAP/AD integration, connections to multiple database, connections to legacy apps (simple links with option for single sign-on integration, with options for new windows or frames) and THEN have an Excel/Wufoo-crossbreed add-on for creating small applications. This would have IT buy-in, as they could use it themselves for small things, it would have built in transparent security, and they would be able to delegate to power users who could define the problem and would be able to work with IT for solutions.
This would be a big app. I haven't found the MVP to extract, save for the framework itself. I'm looking for partners on this idea currently.
ObjectStudio Smalltalk could do interesting things with Excel spreadsheets through OLE. You could open a spreadsheet and pull data out of individual cells, etc... I'd see if there are other technologies in more popular languages with better licensing terms.
Edit: Clarified my first statement.
though I never tried it (or similar) -- my days of heavy Excel use are some years in the past and a few hats away.
If it were, then the winner would probably be Oracle since you are "using that database" every time you, e.g., make a purchase with your credit card. Probably multiple times since your purchase will end up in the databases of the credit card company, the bank, the merchant...
I'd like to see something that's more databasey as a desktop app. I mean, hell, just let me rename the columns, or something.
I'd separately like to see a "forms and workflow" webapp that you suggest.