

The world needs better data manipulation tools than Excel and Access - Zak
http://zakwilson.posterous.com/the-world-needs-better-data-manipulation-tool

======
3pt14159
Besides Dabble (which can kinda cut it for certain non-tech companies, but is
primarily used by tech companies) Excel is the only real option and is
probably Microsoft's best product.

Sure it may have a few strange date bugs. Its column/row order shortcomings in
vlookup() and hlookup() may be annoying. But you can really, really, really DO
stuff with excel.

The learn curve for the software is basically Y = x^.7 (0<x<1) so getting
started is as easy as you can get for a tool that can do all of the below:

Math functions (I could have individually listed them, but they are many,
including correlations for many statistical distributions) Financial functions
(Mortgages, Black-Scholes, Variance) Multi-Dimensional Linear Optimization -
seriously, this can (with a bit of tweaking) also give you all non-inferior
solution sets. VBA - I hate programming Microsoft languages, but VBA was
designed to be easy, and with excel it is SO easy. Relative cell references.
Stunningly beautiful graphs - Next time you are at excel and you need to show
data really dig into how much you can change. You can make these things look
extremely pretty if you take some time to learn how.

The nice part about all that stuff above is that the new/uninformed users
don't every really have to worry about it. I've met tons of young people who
got into programming because they started with macros (and later VBA) for
Excel.

~~~
avibryant
Dabble is "primarily used by tech companies"? Where did you get that idea?

------
phugoid
I'm don't agree with the article's premise, that "the world needs better data
manipulation tools than Excel and Access".

Most of my present colleagues are not programmers, and they manage to create
relatively complex applications with Excel and a bit of Access. There's no gap
between what they can do with their tools, and what I can do with a real
programming toolset. Also, I can't see one tool fitting everyone's needs.

I have also observed that most non-programmers have an allergic reaction to
anything that looks like code, even something as basic as VBA. They'll do
acrobatics with all sorts of weird macros, but that's OK because it takes
place in a GUI; there's very little syntax to get wrong, or structure to
create.

I think programmers underestimate the learning/confidence people need to code
even a simple program.

~~~
tortilla
What the world needs are more non-programmers who can think logically.

------
m104
See, I was going to agree with the headline until I read the article. I was
thinking along the lines of a data warehouse version of Mathematica with
distributed processing and web front end and RPC/REST and all of that.

No, the author wants to "integrate a (not yet developed) visual programming
language." On the JVM. What will this accomplish? Well, it would allow users
to "create fairly sophisticated apps by dragging, dropping and plugging things
together." But fear not, programmers: "Programmers wishing to make more
sophisticated apps would have at least one real programming language with a
DSL or DSL-like library for interacting with the system."

Wonderful, another do-everything GUI app development environment where users
can do more and programmers can do even more than that. Ugh.

"Does anyone who understands the needs of the corporate market have some
input?"

Sure, make sure you check out the billions that have been spent on this very
problem:

<http://en.wikipedia.org/wiki/4GL>

------
anigbrowl
<http://synthmaker.co.uk/> <http://synthmaker.co.uk/about.html> < pictures

This tool is designed for audio DSP, but is not limited to that. If you're
interested in this topic, I recommend downloading the trial, putting it in
advanced mode, and just futzing about for a few hours.

It's very accessible and fast to work with, and could be adapted to quite a
lot of non-DSP tasks. I would love to see it generalized: the modules within
are how I feel libraries should present. Draw graphs for data flow or complex
functions, open up a code box for tight procedural stuff. Perhaps not what the
author had in mind, but audio synthesis of this kind is event-driven,
asynchronous and features a high degree of interactivity, so it may be worth
further exploration.

Please email if you know about this kind of cool stuff in other contexts.

~~~
rjurney
Very impressive software. Unfortunately it is windows only and closed source.
If only there was an open system like this that could be generalized and
extended.

~~~
chancho
<http://puredata.info/> <http://en.wikipedia.org/wiki/Pure_Data>

I'm sure there's more out there. It's a pretty common paradigm for
audio/interactive software. cf. Max/MSP, Reaktor, Audiomulch.

------
jimbokun
R is now at the top of my programming languages to learn list. So, obviously,
I don't know R yet, but is it something that could work well for this? Is
there a way to skin it with a GUI/web front end that would make it more
powerful than Excel, but still make mathematically astute end users not feel
like they are "programming?"

In the theme of "ideas are worthless," I have a vague idea that a "Heroku for
R" could be a winner. Collaborate on datasets, pay for extra capacity for long
jobs on large amounts of data, good UI for quickly generating graphs from
datasets, integrate with existing data stores like Simple DB, Couch DB, etc.

Does this make any sense at all?

------
thehickmans
This idea has a lot of potential, especially if you were to focus on a few
situations that I've seen over and over in the corporate world. The most
common scenario I've seen is data coming in an extract format like CSV,
containing information required for another business process. You've got data
coming from an expense reporting web app that needs to processed before
entries can be made into an accounting system. A spreadsheet or database is
created by a computer-savvy manager, and a clerk uses it on a weekly basis to
do their work of posting entries. If the data format changes, the manager
tweaks macros and formulas to adjust, and life goes on. Any new clerks coming
on board learn how to use "the spreadsheet" and the business process keeps
chugging along. Change managers, and 9 times out of 10, a lack of
documentation and knowledge transfer will cause things to blow up in a short
period of time. If you were to introduce a tool that exposes the funky macros
and formulas in a more visual, understandable fashion, I think it becomes more
easily maintainable. Providing some data validation, error handling and
notifications would be valuable. Finally, provide some version control to
prevent accidental deletion or modification of important functions. The key
would be a much better user experience than the Excel function editor.

------
jaaron
There have been a number of competing products. Two that come to my mind
quickly are Lotus Notes and DabbleDB.

~~~
whatusername
:) If only IBM could work out what they were/are doing with notes/domino...
Some aspects of it were well ahead of it's time. (Shame about the interface
from 4 - 7)

I think that ship has well and truly sailed though.. (Notes 8.5 even has a
pretty nice interface - on a beefy enough box anyway)

~~~
nradov
What gives you the idea that IBM doesn't know what they are doing? Their
public statements about the product road map have been pretty clear.

~~~
whatusername
it's slowly getting there. But I think it's too late.

Area's where IBM/Lotus dropped the ball:

 __\- It looked too old for too long - pretty much anyone my age will find it
(at least < v8) pretty archaic. I think the focus for so long was on keeping
the interface the same (so as not to have to re-train any users) - it missed
the fact that anyone new to the corporate world is going to be horrified at
how old/slow/crap most of Notes 7 looks -- (and don't get me started on the
Workspace)

 __\- Desktop Email clients are probably in a slow death circle (note the fact
that I've had a web based email account for about half of my life now - 12
years). I love the 8.5 Sametime integration - but for an email interface - I
prefer my gmail account.

 __\- The sales pitch was always pretty confused. Email. Groupware. Databases.
There is definite value in some of the Notes/Domino offering - but getting a
sales rep to explain it to you is a fair bit of a challenge

 __\- Lack of end user marketing / interest. How many organizations in the
world today could could announce "We're deploying Lotus Notes" and have people
excited? Not many I'm guessing -- at least - most people I've talked to have
only ever heard the horror stories For most people -- their opinion of the
notes interface equals something
like:<http://homepage.mac.com/bradster/iarchitect/lotus.htm> or
<http://lotusnotessucks.4t.com/>

I think IBM are basically doing the right things with it now.. but I basically
see it slowly dying..

------
pasbesoin
I wasn't too interested in the linked blog post, but the title made me think
of Resolver.

<http://www.resolverhacks.net/resolver.html>

<http://code.google.com/p/resolver/>

I have no experience with it (I'd be interested in comments from those who
do), but the premise is intriguing and it appears to be non-vaporware, if
still beta. (N.B. It appears to be going in the opposite direction, in some
ways, from what the posterous page is proposing.)

\----

Separately, I'm reminded of a recent page I encountered where someone
describes (including how-to steps) using SQLite as a backend for Excel, for
handling large datasets that benefit from the Excel interface (without getting
into the whole Access thing/disputes).

Googling, it looks like the below is the page. I haven't read through it, yet,
but I'd made a mental note to do so sometime. I'll still mention it in case it
intrigues anyone else as it did me.

[http://blog.gobansaor.com/2009/03/14/sqlite-as-the-mp3-of-
da...](http://blog.gobansaor.com/2009/03/14/sqlite-as-the-mp3-of-data/)

------
joe_the_user
The world certainly needs better data manipulation tools for programmers and
non-programmers alike BUT ... neither group feels like strongly enough to make
the market for spreadsheets and user-level database tool competative again.
It's sad.

It shows that individual users drive markets rather than "the world" driving
markets - I don't mean this snidely. There are many things that people would
benefit from if they thought from the point of view of "the world" But that
thinking is not common.

~~~
Zak
I think the route in to a market like this might be a killer app of some sort
built on top of this platform. Users want their POS box tied in to their
accounting program - they may not care about user-level databases directly,
but they certainly care about making their businesses run more smoothly and
reliably.

