

Palantir: an operating system for data analysis - regs
http://blog.palantirtech.com/2009/11/06/palantir-an-operating-system-for-data-analysis/

======
daeken
Please, please stop overloading well-known terms like 'operating system'. This
is simply an application platform, much more akin to something like the JVM
than an operating system. The implementation details are sparse, but at the
lowest level this could be a VM+language, and at the highest level simply a
set of libraries. Calling it a "data analysis operating system" is akin to
calling Ruby on Rails a "web application operating system."

~~~
regs
I think that the line between "operating system" and application framework is
a bit fuzzy, to say the least. I would argue that the Java VM is exactly an
operating system to the code that uses it: a system of operation defined by
the primitives presented by the VM. In fact, a VM and a set of APIs is all
that a "real" operating systems is!

I can see why the title might irk you, but if you read the post, we're not
claiming that it's another operating system to list besides Windows, Unix, OS
X, and VMS, but rather that we've built an abstraction that has the same
power, robustness, etc. for analytic tasks that an operating system does for
simple processes.

The history of computing is one of building abstraction upon abstraction --
it's not like we built the OS and we're done.

------
elblanco
I have great respect for the Palantir team, they are bringing an interesting
approach to the space, but this post is nonsense full of buzzwords (Operating
System, Ontology, Abstraction, and then polishes us off with chit chat about
the information age and a non-sequitur graph of the internet from 2003) abused
so badly that they ultimately mean nothing in the end. It may as well say
"Palantir: a new kind of _physics_ for data analysis" or "a new kind of
_medicine_ for data analysis"

~~~
regs
Yeah, I don't agree. Being the author, I may be a bit biased. ;-)

The post is attempt to put into words an explanation of what it is that we do
that other engineers can understand. It's an attempt to explain by analogy.

Nothing in there is being used as a buzzword. We build actual ontologies. Our
system is an actual abstraction on top the data. It's operation follows the
same pattern as an operating system.

Not only that, I wouldn't call 'abstraction' or 'operating system' a buzzword.
(I'll grant you that 'ontology' is bit overused at times, but we don't use it
for buzzword compliance, we use it because it actually describes the data
models that we build for each instance. But you know, even 'synergy' means
something and can be used in a way that's correct?)

The picture of the internet is in the section referencing how the computer
operating system has brought us into the information age -- hardly a a non-
sequitur. The rise of computing power and the labor savings that good
abstractions like the operating system gives us have actually shaped our
modern world.

I'm sorry you think the piece is fluffy. I'm a software engineer who helped
build the system. I interact with operating systems every day. I ran the piece
by a whole slew of engineers in the company to make sure that I wasn't off my
rocker in what I was describing, from the analogy about operating systems to
why our system has incredible potential for the future. It's not gratuitous
wanking, I promise you.

(Maybe the title just needs to be softened? Make the analogy more clear rather
than making it seem like we're declaring our system to be a 'operating system'
in the classic sense of the phrase?)

~~~
elblanco
I believe you!

I think the problem I had was the piece read to me like it was arguing an
equivalence not analogy.

 _> We build actual ontologies. Our system is an actual abstraction on top the
data._

Pretty much all software does this. Perhaps implicitly, but it all does on
some level. For example, edlin uses an ontology composed of concepts like a
byte, a file, a line, a file handle, standard i/o, character codes to
characters, etc. It abstracts the data from a bit stream into characters and
characters into a line and lines into files, etc. Is edlin an "operating
system" for editing files? I don't think anybody would argue that was case.

The same with Palantir, it's a software application (or set of applications)
that succeeds in using an ontology for object and relationship semantics, it
abstracts away the data handling drudgery, and automates lots of the search
and correlation work. But make no mistake, it's an _application_.

The analogy I'd draw would be to claims of Google building a web OS
(discounting Chrome OS which is still vaporware).
<http://googlesystem.blogspot.com/> Google is building a lot of interesting
and useful apps, but the apps taken together != an Operating system. In this
case "Operating System" has become an unfortunate buzzword.

Palantir is a particularly interesting application, full of interesting and
compelling capabilities, but Palantir's code does not sit in BIOS at the CPUs
entry point memory address, booting up upon power-on. It's not handling and
abstracting raw input data from hardware, carefully monitoring various
interrupts into mouse movement and other I/O interaction, it's not handling
task scheduling or memory management, or has a kernel.

Like any app, it's relying on those services as provided by its OS.

Abstraction likewise is abused as a buzzword and to imply "simplicity". "Oh,
you don't have to know assembly, you can use Java, it uses _abstraction_!" or
"you don't have to know how to use a command-line, just use Windows, it's an
abstraction!" But Palantir's abstraction on its data, while notable and
interesting, does not make it an OS. And user pattern don't make something an
OS. Even if you can write different components for Palantir using a common API
(like your mapping component), that doesn't make Palantir an OS, it just makes
it an application with different "stuff" in it. Is MS-Excel an OS because I
can use it to balance my checkbook, as a lightweight database, parse data and
play games? <http://www.excelgames.org/>

One fortunate thing is that Palantir really does use a proper ontology, so
claims to such a thing make more sense than clams that edlin does (or heck,
I've noticed Analyst's Notebook's marketing copy has stated to abuse this word
also).

I mean this constructively, you guys make cool stuff. But it sometimes becomes
possible to become too enamored in the revelry of your own creation, to think
of it as a mini-alternate universe with a different set of natural laws you
can bend to your will ;) I know I've done it. I'm happy for the willingness to
engage.

------
canoebuilder
Company co-founder - <http://www.charlierose.com/guest/view/6717>

To see the interview, it seems you'll have to go to the front page,
charlierose.com, click the 'archive' tab on the flash interface and scroll
down to 08/11/09.

\-- To charlierose.com site devs, linkable content - good.

