On: The Data Deluge

[blogimus]

Interesting but the article fails to even make a mention of information visualization or visual analytics (how are we going to sift through all that data to see the patterns to make decisions)

http://en.wikipedia.org/wiki/Information_visualization

http://en.wikipedia.org/wiki/Visual_analytics

[elblanco → blogimus]

It's interesting how many different kinds of tools there are in this space.

Everything from ESRI Arc Map products http://www.esri.com to Starlight http://www.futurepointsystems.com to Palantir http://www.palantirtech.com.

Each is as different as can be from one another, yet there seems to be significant overlap between the products as well.

[eru → blogimus]

They talk about this at length in another article in their `special report'.

[jodrellblank → blogimus]

http://infosthetics.com/ - a blog about visualizations from around the web.

[pasbesoin → blogimus]

Visualization and other summary tools remind me of my concerns vis à vis the recurring, seemingly endemic problems I've had over the years in working with same. Or rather, in others' working with same.

Some years ago, I was in an accounting position. We hired a newly minted junior staff member. Requests would come in for certain figures, and he would turn to our admittedly half-baked reporting (inherited, and faulty due not so much to original design as to failure to spend to keep it and its data structures current). He'd generate a a number, graph, etc. out of the system and spend zero effort considering or investigating its validity. If it came out of the system (aka black box), it must be correct.

When the number was used intra-department, I or someone else had an opportunity to double-check and to correct mistaken assumptions. When the number was requested externally, mis-perceptions could quickly propagate, and their correction became troublesome.

I have similar concerns with the data behind current visualization trends. How many people know what is truly going into them, and how much in the way of resources is being invested to ensure that the underlying data are correct?

Something I find particularly troublesome with the Obama administration's initially supplying only "summary information" in some if not all of its "open access" initiatives. If I have only the summary, and based upon my past experience, I have limited trust in what I'm being supplied.

Another example of this seemingly endemic "failure to verify", from a different domain. I've done some QA work in the software field, and all too often fellow QA'a have taken specifications as facts. They don't investigate whether the specifications reflect what was actually developed. Testing becomes a matter of confirming the most basic details, while ignoring real confirmation of underlying functionality much less seeking out and interrogating corner cases.

These testers have been people with four year college degrees and who have seemingly made QA their career. Nonetheless, the assumption is that what they are told is, prima facie, truth.