Hacker News new | past | comments | ask | show | jobs | submit login
Palantir is using War on Terror tools to track American citizens (bloomberg.com)
523 points by pdog on Apr 19, 2018 | hide | past | web | favorite | 260 comments

I'm much less concerned with Palantir being used internally by corporations to detect information leaks (although in practice this also means it will be used to quash whistleblowers), since that's at least a viable use for their technology. And it's good that Palantir is pivoting towards less invasive applications of their technology such as supply chain tracking.

But the article touches on the biggest issue with Palantir, which is that they seem to be completely fine with allowing law enforcement agencies to use their software for parallel construction, which is in my opinion a pretty egregious violation of the 4th amendment. Although it may be constitutional for police departments to purchase data from e.g. Facebook or some data broker, even local police departments are known to employ dragnet-style surveillance through things like Stingray devices. Of course we have a number of three-letter agencies, also Palantir customers, who don't even need to pretend that they're following the 4th amendment.

Since Palantir is essentially enabling the construction of all the tools required for a complete surveillance state, and actively using their tools for that purpose for financial gain, I think that makes them one of the least ethical companies in tech. I certainly wouldn't be able to live with myself if I knew I was writing software with the explicit purpose of being able to track down rebels in Yemen, catch illegal immigrants, and violate the constitutional rights of millions of Americans.

Thiel is an exercise in contradiction. "In his view, the best way to avoid such scenarios in the future would be to provide the government the most cutting-edge technology possible and build in policing systems to make sure investigators use it lawfully."

Prevent the police state by building the police state.

Is it really? I think what he's trying to say is that we should have tools powerful enough to let the Government find/track people efficiently and correctly as long as there is a lawful reason for doing so. Its similar to the weak crypto arguments where Govt. agencies argue they should be able to break crypto protocols via a backdoor when there is a lawful reason to do so.

That being said: the PATRIOT act and NSA's PRISM program have shown just how blatantly the US Govt. can abuse individual privacy and rights so I don't think its a good idea. Weak crypto using backdoors is just a terrible idea period.

I guess what I'm trying to say is that the logic make sense in principle even though in practice it wouldn't work that way.

Perhaps what he's trying to say is, "I like money and power."

"The logic" in this case is a proposition about how beings behave. That is, "We can build the largest prison system in history, give law enforcement unlimited surveillance powers and not expect massive injustices." That is simply wrong. Human beings are not like that.

For someone as deeply entrenched in Tech as Thiel you would think the "in practice" part would be quite clear to him.

Honestly I'd really like Thiel or someone else to explain this. He strikes me as too smart to be that dumb.

And then hire Palantir to monitor the police as well.

Contradiction? You’re too kind, he’s a straight up hypocrite. He used “privacy” to kill Gawker, in part because he didn’t like their coverage of his ventures.

> I certainly wouldn't be able to live with myself if I knew I was writing software with the explicit purpose of being able to track down rebels in Yemen, catch illegal immigrants, and violate the constitutional rights of millions of Americans.

Perhaps open-source licenses should explicitly exclude use in certain applications. Even if not legally enforceable, it would serve as a constant reminder to developers who DO work in these fields of what they are doing.

Something along the lines of "The Software shall be used for Good, not Evil"? ;) https://tanguy.ortolo.eu/blog/article46/json-license

When I was at IBM the story about IBM legal asking for permission to use JSLint for evil was a classic. The original post appears to be dead, but https://news.ycombinator.com/item?id=5138866 discusses it.

The Internet Archive has an archived copy of the blog post, which links to the original source at


Wow 2002 was so long ago. We had no idea.

What am I saying? Most of us still have no idea.

s/shall/should rather/

This raises some good points, but that was to be expected as licenses are a tricky business.

I do think our profession needs more awareness of ethics, and a usage suggestion (rather than an order) in the license of software may be a good start.

Reminds me of how Google bans its employees from using code released under certain licenses: https://opensource.google.com/docs/thirdparty/licenses/#bann...

I wonder if Palantir has any similar policies?

I find it interesting that Google seems to allow GPL (though they ban AGPL). I know at many companies the GPL is an anathema that would be the first entry on a list like this one.

IANAL, but my understanding is that the GPL only requires you to release source code if you "distribute" the resulting binary. As a web company, most of Google's code would not be considered distributed, only the output of that code. The AGPL was created to close that gag for web companies, probably the reason Google bans it.

They certainly are legally enforceable and some software does this already. I can't recall exactly which software it was, but I know I have encountered multiple times software licenses which forbid the softwares use by military organizations or for military purposes, or which forbid government use. Licenses can contain practically anything you like. And thus far, courts haven't been restrictive at all in terms of what can be included.

Can you expand on exactly how Palantir is used for parellel construction?

Law enforcement uses dragnet warrantless surveillance to gather data (such as who is calling whom, who is in physical proximity with whom). That data is loaded into Palantir's software.

Law enforcement uses that data to acquire admissible evidence of the perpetrators' crimes. This is the crucial step, because although the original evidence couldn't be admissible in court due to its collection without a warrant, it can be used to acquire more evidence. If the warrantless surveillance includes information such as the time and place of where you plan to commit a crime, law enforcement can stop you for a random check on the way there, giving them actual evidence for prosecution. Now that evidence can be used in court, even though the only way law enforcement were able to acquire it was through unconstitutional practices.

The thing about Palantir is that it needs big data to work. Police departments don't just have random chunks of data lying around containing large numbers of connections between large numbers of people - they might have something like that for something like suspected criminal gangs, but that wouldn't be enough data to make Palantir's software worthwhile. So acquiring the data necessary to use Palantir's software requires mass data collection. Some of their mass data collection techniques may not require a warrant, such as using license plate captures, but some does, such as who you're calling.

So does Palantir perform warrantless surveillance themselves? To my knowledge, no. But when they sell their software to law enforcement agencies, they must know that the only way their customer could get bang for their buck is through mass surveillance. In my opinion, the person willingly and knowingly selling tools for oppressive purposes holds a lot of blame for the actual oppression that occurs as a result.

You raise several good points, but a few clarifications here:

- In our software that we deploy to police departments, generally the origin of any piece of information is tracked, as well as when it was entered and by who. The intention with this is to prevent this (and other) kinds of abuse. Malicious users can still abuse the system and do a parallel construction e.g. outside of our system or maybe in a way that the two actions (finding non-admissible data and finding admissible data) seem unrelated, but at that point they would be spending quite a bit of time and criminal energy on this. I don't think this would happen commonly for a variety of practical reasons (but ultimately there's no way to completely prevent it).

Ultimately trust in the government and that law enforcement (in the various shapes and forms it comes in) is a force for good and prevents many bad things from happening every single day, is at the heart of the palantir philosophy. If you fundamentally think the government is evil, palantir would probably not be the workplace for you. But once you see all of the bad things LE can prevent thanks to our software, it becomes pretty easy to believe in, even if individual bad actors exist (and always will), like racist cops, cops that abuse their power, etc.

- If the organization is acquiring warrantless surveillance, then this generally means that there is the legislative base for them to acquire the data in the first place. What this looks like varies from country to country, some countries are much stricter with the regulations than others. So for instance in some countries it is not allowed to get information from a suspects public facebook profile, whereas in other countries this is considered A-OK. Especially in europe these things have recently become much more strict with GDPR (and similar rules that apply to government entities). You'd think that be a downside to palantir (less data = less value?) but actually it's a big business opportunity, because our software is the only thing available that is even remotely close in this space to having enough access control 'finesse' that it can enable organizations to be compliant to this law. So this is an area (and competitive advantage) we are investing a lot into -- and typically organizations go from 'completely non-compliant' when we arrive there (e.g. never even deleting data like license plates that are supposed to be deleted after 3 months etc, audit-logging any searches investigators do, limiting search scope, ...) to being fairly or even fully compliant.

Now my personal opinion is that everyone should be able to enjoy great privacy and control over their PII just like citizens of the EU do, so if you live e.g. in the US other another country where the law is lax, you should consider taking political action to change the situation.

Of course we never endorse or support any workflows that are in any way unconstitutional, and we have terminated relationships before with very big government agencies when we had doubt about whether our tools would be used for unlawful purposes.

- The "big data" thing is a bit of a misunderstanding I think -- we have "big data" tools too, but usually these are not of interest to local law enforcement. If you check out our youtube channel you will find some (atrociously old!) videos of the tool that's popular in LLE (https://www.youtube.com/watch?v=yMv3TBxulu4 for instance). This tool is in fact often used with just hand-entered information -- analysts create objects (e.g. persons, links between them) in their investigation. So the data really isn't that big and mostly hand-curated, and the tool works fine without mass data collection (better than with, perhaps). Most of what local law enforcement does with 'big data' tools is to generate reports like crime statistics.

"Police departments don't just have random chunks of data lying around containing large numbers of connections between large numbers of people" -- actually, they really do! Any police department of any size that has existed for a while will have a database with millions of convictions, suspects, court-cases etc in them. Usually on some crufty mainframe or in some crufty old SQL database that contains a lot of terribly inconsistent data (dead links, data duplication, etc.) Also sometimes some of this data is shared between countries, states etc. You can find some videos on youtube of our CEO talking about the challenges of data integration and such.

Ultimately of course there is a general statistical trend (belief?) that 'more data is better', because if there is more and more complete data, you have a bigger chance of finding that connection between e.g. a terrorist and some billionaire who might be funding said terrorists, etc. At least up until a point (at which the data probably becomes just too noisy/hard to deal with, because every individual piece of information carries very little meaning. At least the NSA seems to subscribe to this belief according to their public statements, and I'd think if anyone knows about this stuff, it's them) and assuming you have the necessary CPU power and talent (data scientists) to actually do something useful with this data. Local law enforcement orgs like police generally lack the latter. In police departments, most users are only sophisticated enough to run searches for things like names, SSN, number plates etc, really.

- We generally know a lot about how our software is used at most of our installations, and almost always actively take steps to prevent abuse (auditing etc). Most government organizations will not let us see things like what their analysts exactly searched for (because that is obviously sensitive information) but we do a lot of work to help organizations prevent abuse and insider threats at a higher level. As mentioned above, we have taken action in the past in situations where we suspected abuse. It's on a case-by-case basis, of course -- imagine if there is e.g. one analyst abusing the system, then that analyst getting fired/reprimanded would probably be sufficient action (obviously this is not in our responsibility), but if we suspect systematic abuse or we see signs of repeated abuse without repercussions, then we might pull out completely.

Thank you for participating here.

"Ultimately trust in the government and that law enforcement ... is a force for good and prevents many bad things from happening every single day, is at the heart of the palantir philosophy."

Trust is built on transparency, accountability, responsibility.

Can I make FOIA requests to Palantir?

FOIAs are not made to companies, they are made to government entities. But yes, of course you can make FOIA requests to government entities we work with (the law guarantees you this privilege), and about the work we do with them -- reporters do this all the time.

In some of the more liberal countries the things we do with the government are even part of public record and you can find contracts and services we provide them etc documented online without having to do any kind of request.

Yep, and they get squashed for "national security" reasons because learning how gangbangers are tracked will also let the $current_enemy_in_the_war_on_terror know how they are being selected for drone strikes.

People FOIA palantir related contract successfully all the time, esp. in local law enforcement. If you try to FOIA certain other government institutions, then yes, you might get that answer.

Of course palantir has no input on that in any way. Certain government agencies and activities are excepted from FOIA (and for very good reasons), but if you think it should not be that way (or that the rules should be different) then you should take political action.

How do I determine everything Palantir knows about me, and all the ways they've used and shared that data?

That is like asking everything Oracle or Postgresql knows about you. It doesn't make sense.

Palantir is software deployed for specific customers. The customers then use Palantir against their data and publicly available data.

Thank you for the clarification. I had always assumed Palantir, like their competitors (Seisent, ChoicePoint, etc), did both services and software.

i too would like to know the answer to this question? is it being selectively ignored?

When I send a FOIA request to [insert law enforcement agency here] and ask them to provide me with the source code and data used to classify me as a criminal so that I can determine whether the chain of reasoning holds up, and the law enforcement agency then needs to turn to Palantir to request that source code and data, what response will they get?

I think there are a couple of misconceptions about FOIAs, our software and the law in general here...

- FOIAs cannot be used to obtain specific information about persons, this kind of information is protected. In the EU under the GDPR however, you can get some of this information (about yourself only) -- but I think law enforcement agencies are probably exempt from this (not sure, but would make sense.)

- Our software does not make decisions on who is or is not a criminal in an automated fashion (or indeed any fashion). In fact, law enforcement can not make such a decision at all, ever -- only a judge in a court can deem you a criminal. Law enforcement can merely decide that you are a suspect. (And again, this is also not something our software (or any software anywhere) decides in an automated fashion.) So there isn't really a 'chain of reasoning' here

- When you actually ARE accused of a crime (this would happen in a court, and only if there is substantial evidence against you), you will be told what the crime is and why you are suspected of the crime. Thus the 'chain of reasoning' will be presented to you, and be available for verification by anyone participating in the court case (including, depending on country, e.g. a jury)

There have been court cases already wherein, for example, a person was stopped for a DUI and subjected to a breathalyzer test. The defense lawyer in that case requested access to the source code of the firmware running on the breathalyzer device in order to determine if it was even a reliable source of accusatory evidence-gathering. It is under that sort of situation I imagine a law enforcement body being led to turn to Palantir to request full source code of the Palantir system, description of its algorithms, data it uses, etc. In the breathalyzer case, the source code was actually turned over and they discovered it to be of deplorable quality, with many features such as in the case of any internal failure, returning an illegally-drunk answer without otherwise indicating any fault.

In another similar case, someone issued a citation for speeding by an automated camera system requested the source code for the firmware running on that system. The court greed this was a reasonable request, and they issued a court order demanding the police department turn it over. The police went to the company they got the device from and found that they had no right to request the source. They had no right to know how the system they'd purchased and were using to charge people with infractions even operated. I expect this would be what would happen with Palantir.

Here in Belgium, every speeding fine you get mentions the number of the calibration report of the camera and supposedly you can request this report and if it's absent, incorrect or outdated you don't pay the fine.

> But once you see all of the bad things LE can prevent thanks to our software, it becomes pretty easy to believe in, even if individual bad actors exist (and always will), like racist cops, cops that abuse their power, etc

NO! The ends don't justify the means. If we give up our freedoms, LE can prevent a lot of bad things. But it isn't worth it. Abuse of power is one reason we have checks and balances. If I understand correctly, checks and balances and due process are thrown out the window. And the justification is simply that LE prevented some bad things... But at what cost?

Thank you for this honest and straight-forward response, it was eye-opening to hear the perspective from the "inside". It helped me to have a more balanced view of the issues under discussion.

> Ultimately trust in the government and that law enforcement (in the various shapes and forms it comes in) is a force for good and prevents many bad things from happening every single day, is at the heart of the palantir philosophy.

This is disingenuous at best this is a tool for surveillance and its misuse in public life is more obvious than the advantages it might bring. People will naturally distrust a government that spies on them constantly and that has nothing to do with good or bad government.

There is an episode of black mirror on the mother who surveils her daughter constantly if the mother is good why do we feel disgusted by her attitude?

I applauded apple for publicly denying the government a backdoor in their phones.

It is unclear to me what you are saying here. It is well-known fact (amongst the intelligence community) that our software has been used to save tens of thousands of lifes, prevent hundreds (thousands?) of terrorist attacks, saved billions of dollar in fraud etc. In terms of most of the kind of projects governments do in an attempt to prevent {crime,fraud,terrorism,...}, on a scale for how much bang you get for your buck (where 'buck' might be how much data collection you have to do), I think you won't find many other alternatives that compare favourably. (But then, I might be biased, of course.) In fact, as I described in many agencies the situation actually drastically improves once they start using palantir software, since they already had all the data all along, but weren't properly using it as the law intends them to.

Everyone knows the various forms of law enforcement perform some measure of (and have to, to do their job) surveillance. That ranges from e.g. ALPR cameras to patrolling the streets of some city. Most people agree that this is a perfectly acceptable thing. Do you think instead the police should be abolished? Or stripped of their investigative or enforcing power? I don't think most people would actually enjoy living in whatever society would result from this.

Of course the exact boundaries of what is acceptable and what is not should be fiercely debated.

> Of course the exact boundaries of what is acceptable and what is not should be fiercely debated.

What makes most people uneasy, and especially those of us who work in tech, is that many police departments and 3 letter agencies have routinely abused the trust that people put in them. If you want the citizenry to have trust and faith in law enforcement, they have to demonstrate a desire to not break constitutional rights when they deem fit to do so.

I genuinely don't understand why these agencies wouldn't try to build a good relationship with the citizenry they are supposed to be protecting. It often seems like we live in a very dangerous society and everything we say or do is being monitored by someone and may be used maliciously. That itself is a very terrible thing to live with.

I think it's important to distinguish between individual abuse and systemic abuse.

Every one of those agencies has had, in their history, at least some cases of individual abuse (e.g. one particular individual abusing their power). This is unavoidable (because these agencies employ humans, and humans are assholes sometimes) but as long as the situation is appropriately handled at the organizational level (the responsible party is reprimanded in an appropriate way, e.g. let go, tried, ... and practices and protocols are put into place to deal with and prevent these kind of situations) this is OK, because the agency operating is still a net win for humanity.

Most people who are paranoid about these kind of things are so because the media loves to pick up these stories and hype them up and spread fear -- but I don't think it's that much of a problem.

The thing that's really a worry (and that media usually does not report on) is when abuse happens in a systemic fashion, e.g. agencies bending the interpretation of the law to extend their reach. Incidentally, this is exactly what palantirs software is designed to help with, because it can actually enforce these rules.

That is an excellent point. Its much more sensational when you have a few bad apples but exposing a rot within an organization isn't as "newsworthy". Besides, if the corruption spreads far enough, it might be able to actively silence its critics (e.g. Nixon's shenanigans).

Perhaps a systemic fear of Government overreach is embedded in the DNA of the American people. I would argue that a healthy criticism is very important; but paranoia can be dangerous and unproductive.

> our software has been used to save tens of thousands of lifes, prevent hundreds of terrorist attacks... This is exactly what I mean by advantages vs disadvantages these look like stapled platitudes from the marketing team. It looks like you are conflating war zones scenarios with civil society a bit like promoting an M1 Abram for New York police, due to the effectiveness in Iraq. If you aren't then I'm baffled by how many terrorist attacks we've thwarted, and incredulous. There haven't been thousands of terrorist attacks in civil society before the software was created.

> Or stripped of their investigative or enforcing power? So it's either all the power or none of the power? This is a logical fallacy you are creating a duality for something that does not exist. The police is not the same as your software like you implied. I'm against the government spying on their citizens even with good intentions. See the black mirror episode since it demonstrates where problem lies, it's not in good intentions, it's in the control.

Does this software help the government to spy on their citizens? Yes. Should the citizens have a say in if they want this software to be used for their data? Yes, why don't they have say? Can we live without this software? Yes, like we've done so far. Is this software helpful in a police state? I would say it's the most valuable software for a police state.

No, most aware people do not agree with your anti-privacy view.

> we have terminated relationships before with very big government agencies when we had doubt about whether our tools would be used for unlawful purposes.

Let that sink in for a moment, folks. Then go back and read this other part.

> trust in the government ... is at the heart of the palantir philosophy

So, at Palantir they trust the government that they know is trying to use their product unlawfully. With all due respect, I think many of us are a little more sparing with our trust.

Not every government in every country is of the same quality in terms of e.g. corruptibility and willingness to improve. That's just how it is. It's not too different from other companies that are not expanding into certain regions because they don't consider the region politically stable or developed enough, for instance.

That doesn't contradict the belief that law enforcement is necessary and good.

> If the warrantless surveillance includes information such as the time and place of where you plan to commit a crime, law enforcement can stop you for a random check on the way there, giving them actual evidence for prosecution.

why is that a bad thing?

If the crime is victimless (such as any drug usage "crime"), then the issue isn't with the police, but with the state defining what "crime" is, and that definition doesn't align with the people's definition.

Selective enforcement. Practice surveillance of the population. Select political opponent, as desired. Consult surveillance data. By random chance, sometimes you will find something. Parallel construct a case. Proceed to blackmail/extort/convict. Maintain political power. Repeat.

There was no probable cause for you to be under surveillance prior to the "random" stop.

Some people just get more randomly selected than others, a lot more.

Any interaction with the police is a potentially violent interaction with the State. Even if that were not the case, limiting infringement on your rights as much as possible is a good thing, and the government has limitations placed on it for this reason. Stopping people for 'random' checks may violate those limitations.

I wouldn't blame the tools here. When law enforcement needs something done, they'll get it done, justice be damned. Before Palantir, law enforcement would have just spot checked every single car passing through an area, under the guise of a Ride Program.

It's like billboards vs. targeted ads. Both have the same effect (pushing you to buy stuff), but the latter cuts out all the noise.

I see other responses explaining how surveillance data, itself inadmissible in court, can lead people to evidence that is admissible. That's great, but it still makes it seem like only criminals are at risk. Far from it. It's important to remember that this same parallel construction can be used to strengthen false allegations or convictions, allegations or convictions under an unjust or unreasonable law (e.g. marijuana prohibitions), or discriminatory application of a proper law. Got a political opponent? Use Palantir to discover some little pecadillo, then engage the police/court machinery either to get an actual conviction or - almost as good - a whole bunch of negative PR. It's not just criminals. This kind of thing endangers everyone.

Let's say I'm law enforcement and I have some information obtained in a way which wouldn't stand up even in the most farcical of courtrooms.

What I can do is use surveillance tools to find a way to effectively entrap you, to gather evidence. Think of it like a tip-off that you're not allowed to have.

Palantir specialises in providing tools to analyze and aggregate big data. If law enforcement has an interest in it, it's probably for dragnet-style surveillance. The problem is that the 4th amendment means there must be reasonable suspicion before the investigation can begin.

Most dragnet-style surveillance is incompatible with the 4th amendment. So law enforcement has to construct a reason for investigating the case that they should never have been investigating.

If the data was sourced from public information that the target willingly posted online, does law enforcement even need to use parallel construction? Couldn’t they use the palantir data to obtain a warrant for more thorough searches?

I too have a problem with business models that invade our privacy. But I think it’s a bit disingenuous to conflate “private information” with information that you voluntarily posted on the internet.

Whatever happened to educating people that what they post on the internet is permanent? Dragnets over public data are a symptom of the real problem, which is lack of user education and understanding of what data they generate and share.

I wasn't even thinking about public data. There's a lot of data that law enforcement already collects. What about license plate data, data from IMSI catchers, and leaked data. Add in the publicly available online data and you can supposedly build up quite a profile about them.

What if somebody is arrested and happens to have a very convenient data set on a lot of people, such as data used for marketing purposes. Is it okay if those people are stopped and searched on the basis that somebody else held data about them and the police have supposed that 10% of the clients are suspected of criminality?

Normally there is supposed to be evidence that the crime took place before an investigation can begin.

If law enforcement starts with the evidence and then look for the crime that fits it, that's something different.

You bring up a good point. Palantir is not just mining public data, but also organizing data that law enforcement provides to them (like license plate readings). But if law enforcement already has that data, is the problem that Palantir organizes it, or that law enforcement collects it in the first place?

I think you may have causality reversed. Without Palantir, law enforcement would not only have no use for the data, it probably wouldn't be able to even collect it in a meaningful sense, in terms of technical and financial resources.

Palantir goes to the agency and says, "Give us data and we'll give you "actionable intelligence".

You know, back in the 90s and early 2000s, the term "data mining" was perjorative. Financial operators would look through their mass of data for correlations and sell them to people who would discover that they didn't actually work.

This is not in the least accurate...

1) yes, the LE do already have the data. 2) agencies don't 'give' data to palantir, law enforcement data almost always lives in air-gapped environments it cannot leave in any way (or only in highly restricted ways) 3) we don't 'give' actionable intelligence, we just provide tools for analysts to generate such data. Palantir is a company of software engineers, not of intelligence analysts.

you seem to be quite defensive of your choice to work for a company that builds spy software and sells it to anyone with the money to buy it. are you trying to convince us your employer is good, or yourself? its most certainly an ethical grey area, but you try hard to make it seem benignly white. these engineers know what the software is being used for, they can act innocent all they want, but what happens when this tool is used for bad? can you guarantee that it wont be? seems like your teams are turning a blind eye in exchange for large paychecks and doing some fine mental gymnastics to justify it all.

I would argue that law enforcement would have no use for collecting such vast amounts of data without a service like Palantir. So the blame rests on both Palantir and law enforcement

No, not really. We don’t expect all of our actions to be monitored and collected 24/7 throughout the course of our lives when going about our business offline and then used to entrap us. We should have a similar standard when it comes to our civil rights in the digital realm.

As an aside, education does not scale with the ever increasing number of ways that personal information can “get online”. Additionally, it assumes some degree of privilege as more and more of everyday life moves online. Not everyone can afford to care for their own privacy, and those that do must pay the time, money, and convenience costs associated with it.

New Orleans PD...there was an article about it here recently. That is PT/Palantir's vision of the future. It is going to be pretty damn hard to stop once it gets going hopefully Facebook being on the front page of MSN for a couple of days will we a reckoning because the things the average person does not realise would REALLY upset them.

On a positive note, New Orleans decided to not renew their contract with Palantir. I am not sure whether this is the result of that article or not.

The mayor is leaving office.

They dig up data telling the police where to look. The police then look in just the right place and "legitimately find" the evidence that they would never have stumbled on and couldn't have legally known was there.

Let met get this right you ae cool with "corporations" doing this with zero oversight but not with law enforcement and TLA's who have at least some legal oversight.

Are you are also cool with companies in the past hiring pinkertons to kill people also - because that's where your argument is leading.

The difference is that corporations closely monitor only their own employees while Palantir closely monitors EVERYBODY.

Corporations take an interest in tracking people only to stratify them into groups suited to specific ad strategies. The worst their errors will do is pepper your browser and email with bad ads.

Not so with Palantir when their tracking data can misguide police departments, credit agencies, and even your employer into making your life a living hell. Surely any company with that much power must be held MUCH more accountable than they are now.

Palantir doesn't really _do_ anything. They don't collect the data, just save time/man-hours spent processing it. If that's your stance, Oracle/IBM are just as evil because they also sell to police departments to enable analysis.

Facebook on the other hand is actively collecting the data. Without Facebook that database does not exist. In my mind that makes them far worse.

I think it's not a fringe opinion that Oracle/IBM are just as evil. I don't have a strong opinion on this issue at this point, but just saying.

>But the article touches on the biggest issue with Palantir, which is that they seem to be completely fine with allowing law enforcement agencies to use their software for parallel construction

I think that any company that works for the government to any degree is considered part of the government and is bound by all rules a government agency/employee would be.

Yes and no.

Government pushes lots of compliance down in contractors and state government, but tends to not push down transparency requirements.

Thiel's genuine commitment to freedom is questionable. Also, he's a misogynist and jerk.

I feel these issues are orthogonal. There are plenty of nice people who have no commitment to freedom or privacy, and there are plenty of assholes (and even assholes-to-women!) who are also pretty passionate about both.

If they are orthogonal, then the conjunction is maximally informative...

Excellent point, now that you mention it, let's bring Trump and deep learning and 90's grunge into the discussion to form a maximal eigenbasis of discussion

Except most of those have no bearing on the conduct of Palantir, obviously, while the character of Thiel very much does.

That's not obvious. In fact, it's quite likely that my reading Trump's tweets and news along with my internet history from the early 2000's indicating an interest in Nirvana may have a weak correlative effect in a deep learning system that indicates I am slightly more susceptible to committing certain crimes or personality affects.

Ergo, can we all refocus this discussion of universal privacy rights violation on my partisan social group interests?

I do question his commitment to freedom. But a misogynist?

I base my assertion on sentiments he has expressed, one if which was quoted in the Bloomberg article:

>The 1920s was the last time one could feel “genuinely optimistic” about American democracy, he said; since then, “the vast increase in welfare beneficiaries and the extension of the franchise to women—two constituencies that are notoriously tough for libertarians—have rendered the notion of ‘capitalist democracy’ into an oxymoron.”

some context from his response to the article. https://www.cato-unbound.org/2009/04/13/peter-thiel/educatio...

"I had hoped my essay on the limits of politics would provoke reactions, and I was not disappointed. But the most intense response has been aimed not at cyberspace, seasteading, or libertarian politics, but at a commonplace statistical observation about voting patterns that is often called the gender gap.

It would be absurd to suggest that women’s votes will be taken away or that this would solve the political problems that vex us. While I don’t think any class of people should be disenfranchised, I have little hope that voting will make things better.

Voting is not under siege in America, but many other rights are. In America, people are imprisoned for using even very mild drugs, tortured by our own government, and forced to bail out reckless financial companies.

I believe that politics is way too intense. That’s why I’m a libertarian. Politics gets people angry, destroys relationships, and polarizes peoples’ vision: the world is us versus them; good people versus the other. Politics is about interfering with other people’s lives without their consent. That’s probably why, in the past, libertarians have made little progress in the political sphere. Thus, I advocate focusing energy elsewhere, onto peaceful projects that some consider utopian."

That's definitely an eyebrow raising quote, but I'm pretty sure he meant it the opposite way it's being taken. He's fine with the franchise being extended, he wants those groups to have more libertarians. It's the same thing as Democrats trying to make it easier to vote, people who vote less or have a harder time voting tend to vote Democrat.

Yes, he's observing a trend and comparing its results to the ideal “capitalist democracy” that Thiel wish existed. Women, in general, aren't very libertarian especially on fiscal issues. He's not saying their rights should be taken away. I'll admit, it isn't the best worded statement in the world.

He isn't saying that they ought to never have been granted?

No, he isn't saying that.

Of course not. He's just invoking all of those elements in a provocative sounding statement for no particular reason.

He's saying that women tend not to be libertarians, which is a large enough block of people to where it's very difficult to implement libertarian policies to law. There's no implication that women shouldn't be allowed to vote or women's suffrage was a mistake, it's just a really awkwardly worded statement.

Describing your “ideal society” and then depicting one group, especially one who was politically and socially disempowered at the time, as an obstacle to it is called scapegoating.

Historically, this ideology has had some very bad consequences.

Why is this being down-voted? It's the truth.

Thank you for bringing your sexism agenda into a discussion on Palantir

You are most welcome.

Do you agree that a company's culture starts from the top?

I take it that your consent is agenda-free?

Palantir's repeated forays into politics are a bit troubling as well, particularly when combined with the issues you mention.

What are their repeated forays into politics?

> The U.S. Department of Health and Human Services uses Palantir to detect Medicare fraud.

Having worked in this field, I can say that this is hilarious overkill. The vast majority (in dollar terms) of Medicare/Medicaid fraud is drug companies misreporting their Average Market Price (AMP), the critical variable in how much they get paid. In 2012, Glaxo settled(!) a small bundle of cases, some of which I worked on, for three billion dollars [0].

You don't need police-state surveillance of individuals, you need visibility into company transaction records.

[0] https://www.justice.gov/opa/pr/glaxosmithkline-plead-guilty-...

Medicare fraud is estimated to be around $60B per year or 10% of the whole budget. So if Palantir can decrease that rate by 1 or 2% then it's tough to call that overkill


>Going even further back, in 1998, a Senate investigation into Medicare fraud found $6 million in payments to a “business” whose fake address would have been smack in the middle of the Miami Airport.

>The report released today shows no improvement, but investigators say it is not that hard to fix the problem.

>ABC News went to one of the locations listed in the report that was on Chicago’s Southside next to a porn shop, with no doctor’s office evident, where Medicare sent nearly $600,000 using an ineligible mailbox shop location as a billing address until 2013.

Seems like exactly the kind of problem Palantir can solve to the advantage of taxpayers.

These things could be fixed by simple address verification. Amazing they have to wait for something like Palantir.

This is like someone saying web search could be solved by a simple database lookup.

Medicare fraud goes well beyond price reporting by manufacturers. Physicians can bill for more expensive procedures, bill phantom patients, etc.

That's a huge part of the multi billion dollar fraud that Medicare has.

I assume being able to identify physicians will outlier billing patterns would be helpful in figuring out where to look closer.

I mean, if you want to get into it, you're both right and wrong.

The second largest cases are typically against hospital chains that encourage (read: basically force) their coding teams to use particular billing codes. It's easy enough to model this from claims data (and we did). When a chain buys a new hospital, suddenly patients that present with the same diagnosis codes are all getting extensive exams that ostensibly require 45+ minutes of a physician's time and cost 4-10x more.

Again - the large operators (hospital chains) absolutely dwarf individuals (doctors). DOJ barely has the resources to prosecute the largest offenders, and have faced repeated budget cuts. Wasting time on individual doctors or private practices (beyond maybe making examples of a few egregious actors) would be an inefficient allocation of their resources.

Do you know the details of Palantir's investigation? How do you know they're not doing exactly what you just described? Palantir is just a data analysis company, they're not the NSA.

> The LAPD uses Palantir’s Gotham product for Operation Laser, a program to identify and deter people likely to commit crimes. Information from rap sheets, parole reports, police interviews, and other sources is fed into the system to generate a list of people the department defines as chronic offenders [...] The list is distributed to patrolmen, with orders to monitor and stop the pre-crime suspects as often as possible, using excuses such as jaywalking or fix-it tickets.

This is pretty much everything I was afraid of when it comes to this field - big data "prediction" being used to predispose people into crime, and by extension, algorithmic racism.

Minority Report. Literally.

For all we know, keeping a closer eye on previous criminals deters them from repeating their crimes. This strikes me as a very benign way of preventing a crime from happening in the first place. Rather than letting it happen, creating a victim and perpetrator, and then having to punish the latter yet again.

At the very least, this should give them the impression that policing is much higher than it actually is. Sounds like a good deterrent to me.

There is a lot of problems with this strategy when introduced into the real world; the chaotic vortex that it is.

The data can be wrong. This could be easily fixed if law enforcement agencies had to actually verify that the data was correct, but in practice this seems to rarely occur.

When people are told by a group that they are a thing, repeatedly, they eventually accept that they are that thing. If the cops keep harassing you for being a gangbanger when you have never been one your whole, you just happen to be associated with a few individuals who are, you eventually just move towards that group. Why not? The powers that be already think you are one and they treat you like one, which causes other people to treat you like one, so why not just be one?

You also need to consider the idea of enforcing crime that hasn't happened yet. That defeats the whole point of law enforcement. It is meant as a reactionary force, not an offensive force. Policing someone who has the potential for crime assumes that they will do it again, which makes it so your essentially treating them as guilty for something they haven't done yet. From my perspective, that seems to defeat the purpose of our legal system.

Anyway, I think I will end my rant here...

This isn’t Palantir, this is the companies who buy Palantir software.

The data from JPMorganChase isn’t visibile to Palantir as an organization, but to the bank.

This headline is weird and as accurate as “Microsoft knows everything about you” because the bank uses SQL Server. Or “the Python developers” because python and pandas is used to link a bunch of data together.

Since this is Bloomberg, I would expect them to know the difference between software and data services. Google gets your data when you use them (except for enterprise), most other products don’t.

Cynically, this seems like oppo PR against Thiel. How could you prove this suspicion?

Is it really the case that data does not cross boundaries among Palantir clients? And that the algorithms used by any given client are only informed by the data that client provides?

When I worked there (mid 2009 to early 2014), almost all customers had dedicated installations (as in a rack of servers) with just their data on-site. Even the hosted instances had siloed data. There are features for peering, but AFAIK that has to be explicitly set up between installations.

I’ve used them on prem and hosted. For on prem, definitely. It’s like any other big install of software (sql server cluster, oracle financials). All on site, no data out. Brought in 3rd party data, shared no data out.

Cloud same, but they host and promise not to do anything. Like any other enterprise cloud product.

The only data they gave us was either open sets like weather or census or stuff anyone can buy like Quintiles. So there was no “magic data set” we could mix in or something.

If so, then Palantir is missing a big revenue source: integration with public data that each client would otherwise have to collect on its own.

Palantir is a consulting company, plain and simple. They don't do anything IBM couldn't, they just (plausibly) do it better than IBM.

Unless proven otherwise, then that is hearsay.

Good point, but this is the risk for every cloud hosted software. So every AWS and azure product has this suspicion that’s impossible or extremely difficult to remove.

If you’re really paranoid, then all non-OSS software.

So again, why is Palantir more worrisome than Salesforce?

This is covered further down in the article:

>The military success led to federal contracts on the civilian side. The U.S. Department of Health and Human Services uses Palantir to detect Medicare fraud. The FBI uses it in criminal probes. The Department of Homeland Security deploys it to screen air travelers and keep tabs on immigrants.

But that’s using the data that DHS, or FBI, or CMS has already. It’s not using some Facebook-sleazy data source. It’s not data reuse, but algorithm reuse. And algorithm is a stretch as it’s like teaching people how to use graph databases.

Whilst the article may not have any surprising information for the regular hacker news reader, it may come as a surprise that a non-insurance, non-government business holds more private data about an individual than Facebook does, especially since most individuals will not recall ever interacting with said company. I think it is beneficial if more people become aware of this.

> it may come as a surprise that a non-insurance, non-government business holds more private data about an individual than Facebook does

Where in the article is that stated?

I guess I should've prefaced my comment that it's more of a response to the comments that existed here before I posted - most people just stated that they don't care and they were aware of the fact that such businesses have all this information.

In the title?

It’s clickbait. It doesn’t state it or give a citation for it anywhere in the actual article.

Yeah, but you edited your post to specify "in the article" after I'd posted. At the point of my reply, it was correct.

I don't think it's directly stated in the article. But with a little common sense we can probably safely assume that a company that is designed specifically to analyze huge amounts of data for law enforcement agencies, governments and 3 letter agencies will probably have a lot more data than Facebook.

In fact, we can probably also assume that people's Facebook data has already been inputted in Palantir's system as just another data source.

Sure that's a lot of assumptions and probabilities, but on the other hand: Peter Thiel.

I used to be intelligence analyst that utilized Palantir on a daily basis. Ask me anything.

To what extent are searches/queries/data-accesses performed by analysts logged and audited? Were there actual and official repercussions for looking at the data of family members and romantic interests?

From what I recall all activity is logged. We also had numerous external databases/tools that were essentially API connections to reduce the number of separate logins we'd have to do, and each of those databases had their own audit mechanisms. Long before we ever had Palantir in my department, we did have an employee who had misused a database for personal reasons. They were initially terminated but that was undone upon appeal. They were reprimanded either way and lost access to various tools/databases going forward for a long time. I know of many stories of police, for example, getting fired for abusing the NCIC databases for personal gain. Sadly it often falls on the agencies to self-police these issues, and most don't have anyone dedicated to this effort. So it really falls on the first line supervisors, of which some are great about it and some couldn't be bothered. I am of the opinion that all security, intelligence, and LE agencies should have an internal team solely focused on audit & review actions for internal abuse/misuse/privacy concerns, but that is a minority opinion from my experience.

From what I recall all activity is logged.

My apologies in advance to offtopic haters. But this is too good to overlook. Did anybody else notice how this question relates to the origin of the name?

What kind of databases were used in your case for the analysis? Was it only the databases you provided them? Or are there also some generic sources Palantir provides by default? Or do they even cross-reference data from different clients?

We used a variety of 3rd party records databases (think Lexis Nexus, TLO, etc.), government/police databases (NIPR, SIPR, HSIN, NCIC, LEO.gov, etc), internal databases, and state public records systems (think DMV). We only had databases which we owned or had access to. Palantir did not provide any actual data for my department. They did have pre-built connectors for certain data sources, and pre-built attribute models for tagging data, which I assumed were developed for previous customers. But the initial install of Palantir was empty, and only populated with our data sources. To my knowledge Palantir does not provide any data. I actually recall walking into a room and finding 3-4 Palantir engineers doing nothing but migrating data from our old intelligence software into the new Palantir system, mostly by hand.

I can say with 100% certainty that they did not cross-reference our data with other customers, because the software not sending any data back to Palantir was a primary stipulation of our purchase. I was always under the impression that no data was shared across customers, just code, connectors, and best practices. Our installation of Palantir was 100% local, none of the infrastructure sat in a vendor cloud environment or on Palantirs home servers.

How effective a tool was it compared to other tools at your disposal?

This is probably my favorite question, and the reason why I like Palantir, but tell everyone not to bother buying it. The most common "low-tech" tools used by analyst's across the board are IBM's I2 Analyst's Notebook, ArcGIS (or something similar), Excel, various external databases, and Microsoft word. To super dumb-it-down: You use external data bases and excel to collect initial data, you use i2 to visualize it, you use ArcGIS to map it, then you use Word to write a report about it. Palantir does all these things on it's own, so in theory you'd only need to buy and utilize one product vs multiple. The problem is that it does each one fairly sub-par as compared to the more common "low-tech" options. You have to deeply invest in Palantir, and have them build in the customization necessary to really make it shine within your department. Unfortunately that ups the price significantly, so many agencies are buying essentially the off-the-shelf version and then complaining that it doesn't do what they need it to do.

What kind of search filtering options are available for fishing expeditions?

You can search on or around nearly anything. That's one of their selling points to customers. The search function is multi-faceted, in that 1) it can index the data of any files you upload to Palantir (making everything text searchable), 2) Analysts can indicate specific attributes of an entity just by highlighting text in a file and marking it as a DOB or SSN or whatever. 3) The # of attributes is nearly limitless.

One of the primary screens in Palantir is a simple search page designed to work (and look) like Google, to search upon your entire database however you'd like. You can enhance the search function via the various attributes built into your installation or using advanced search tools. So I guess a "fishing expedition" in the traditional sense is very easy, as it's limited by the data you have, not the system itself.

Are there options to restrict search space (i.e. limit data sources that could potentially contaminate an investigation)?

And if so, is that a common practice, or is it more typical to search through all avaliable data for anything potentially useful and weed out later?

I don't specifically remember an option to weed out searching, but I'm reasonably certain there were advanced options that included "does not equal" or "does not contain" type boolean searching. From my experience though, it wasn't common practice to weed out from the start, but rather to start with a general search and slowly narrow it down to the targeted data. A lot of data sources weren't automatically imported, they were more like extended queries. So if I wanted employment records, for example, I had to initiate that as a custom search to query that external system, and pull back the data for the specific individual I was investigating, it was not available in the database by default. But once it was imported and tagged, it was available in any future searches for the remainder of the data retention period.

Why did you quit?

I was frustrated with management in my department and ended up leaving for private sector.

I have heard it was pretty rough. I extricated myself after the technical interview after reading up on process and conditions seemed like interesting work but not remotely altruistic in its goals. I hope you found something much better.

Is Palantir not considered to be private sector?

It is. I did not work for palantir though, I worked as an intelligence anaylst in public sector who used Palantir in my job. Palantir does not provide analysts, generally, they just sell the software. The forward deployed engineers help install and instruct, and sometimes they have people come on site to walk through analytical use cases, but generally they don't employ many, if any, actual full-time intelligence analysts, from what I saw.

Palantir is private sector, but they sell to governments.


Not drinking too much. Regular exercise at the gym, three days a week.

This isn't surprising. In fact it was expected. This company was born out of the Total Information Awareness mandate by GWB shortly after 9/11. Total Information Awareness means monitoring of everyone and everything. They got contracts and funding and grew to the original vision set in motion 15 years ago.

> Aided by as many as 120 “forward-deployed engineers”

This term popping up again made me curious. To my surprise, that's not just some overloaded nickname (what's cooler than styling yourself with a military or martial arts term, right?), that's what they are actually called:


What's the difference between a normal software engineer and a "forward-deployed" software engineer, though? It sounds to me like that's just their military-flavored version of "rockstar developer" or "code ninja".

Well... simply put, a forward deployed engineer is on the front lines of totally crushing it

"normal" software engineer - works at Palantir facilities

"forward-deployed" software engineer - works at Palantir's client's facilities

my understanding is forward deployed engineers set up shop in Palantir's clients' offices and implement the solution using Palantir's tech. Basically doing rollout and front-of-house tech support.

They are professional services. But instead of regular consulting, they are programmers and analysts, etc.

They sound cool and, to me, signal that their work is more meaningful.

they fear the words "consultant" or "contractor" I think

Forward deployment had a much hipper, militaristic sound.

Yup. And FDE is just a consultant, an FDSE is a software engineer attached to a specific client. A regular SE works only on generic products.

I’ve always thought - I don’t care if the NSA is spying on US citizens because their job is national security. The problem is when those tools trickle down. All of a sudden you have a local cop with too much time on their hands spying on you. But if you’re not doing anything wrong, you shouldn’t have anything to hide, right? Isn’t that how the saying goes?

> But if you’re not doing anything wrong, you shouldn’t have anything to hide, right? Isn’t that how the saying goes?

The problem is that "wrong" can be anything arbitrary, and what's not "wrong" today, it can be "wrong" tomorrow, however arbitrary it is. Somewhere it is wrong (illegal) to call someone a "n*zi", even online.

Also anyone who tells me that they have nothing to hide, I ask for their password. For some reason they don't give me their password. Interesting, huh?

Nothing to hide from the law != nothing to hide from you.

Law? These are people who have access to your data, how am I, a stranger, any different from any other stranger who have access to your data? Would it be OK if I indirectly obtained your password and use it to access your data or something?

Mind you, I was speaking in general about people who claim they don't care about their privacy, or about these companies collecting their personal data, or having a history of their private messages, etc.

How can you not care about them saving your conversations in plain text, but then suddenly care about it when I ask you to show it to me? The only difference seems to be that I'm actively, personally, directly asking you for it, but ultimately it would be the same outcome.

> Mind you, I was speaking in general about people who claim they don't care about their privacy

We are talking in a thread about NSA having your data vs other companies having your data.

> The problem is that "wrong" can be anything arbitrary, and what's not "wrong" today, it can be "wrong" tomorrow, however arbitrary it is.

(Un)fortunately you can't be prosecuted for crimes which were not illegal at the time of the offence.

Of course this doesn't take into account activities retroactively deemed "wrong" by The Great Twitter Mob which can have some serious effect on your livelihood because you made a Richard Gere gerbil joke some 20 years ago.

Or worse, at some point that information is available to people who actively want to hurt or stalk you. Every bad divorce leaves at least one person with an axe to grind. It’s only a matter of time before collected information gets to adversary hands.

As someone who recently left a very abusive, toxic workplace, I see this as a reality.

I'm going to have to dredge it back up, but I was just reading a rather alarming tale from a medical assistant. She had a child come into her practice, saw obvious signs of abuse, and filed a supposedly anonymous report - as she's legally mandated to do.

The abuser of the child, a parent, apparently harassed the agency that received and forwarded the report until they gave up that "a medical practitioner had filed the report", which narrowed it down to a single practice.

At which point then the abuser was able to call the practice, get a record of who had seen the child, and even get their work schedule.

Even tiny leaks are a major problem in these cases.

And really only adversaries will be willing to go through that trouble.

> I don’t care if the NSA is spying on US citizens because their job is national security

Since when is the NSA supposed to spy on their own citizens by default? They should only target individuals that represent risk anyway. (if they were respecting the constitution)

> They should only target individuals that represent risk anyway. (if they were respecting the constitution)

If they were respecting the constitution, they should only target individuals for which they have a warrant (assuming we're talking about US citizens). Even then, it's my understanding that this is the job of the police, rather than the military.

Thank you for understanding this. I've been looking for someone to point this out.

I've always thought that the NSA should respect their statutory mandate and not spy on US citizens.

Why are you comfortable with casual lawlessness?

Must be a nice way to live, having full faith that your government will always do the right thing. Relaxing.

We literally have thousands of years of proof that governments always abuse their power when they think someone is their "enemy". So it's strange that people default to "trusting their government." I blame this on America's blue-team/red-team political thinking, which forces one to stick to a party no matter what.

The standard for "enemy" could also be as low as they choose it to be, which is also weird that people think the government wouldn't go after them because they are "innocent", as if these people get to decide who is innocent and who isn't.

No, it's the government that does that. Courts may have the final say in the end (unless you're indefinitely detained for multiple years first), but until then the government can do a lot of damage to your life if it comes after you. Even putting you through years of lawsuits, stress and money spent on lawyers would be bad enough. You may win in the end, but it will be a Pyrrhic victory.

And that's if you don't take their plea deal first when they scare you with multiple bogus charges that you think shouldn't even apply to you but you're too scared of the 50 years in prison sentence they're threatening you with.

It's more a matter of focusing the discussion. Once you bring government into it, it makes the conversation a lot more complicated. This is not a complicated issue.

But it is the Government that complicates the thing: USSR, China, Fascist regimes: they persecute OPINIONS not deeds.

Welcome to Denmark.

Their job is national security... within the confines of the constitution. I don't get how an entity egregiously violating the supreme law of the land can ever be dismissed with an "I don't care" by anyone who claims to care about the rule of law.

COINTELPRO. Learn your history. https://en.wikipedia.org/wiki/COINTELPRO

Wrong is too subjective for that to be a good enough reason to let these sort of entities spy on everyone. Let's say I disagree (I do) politically, legally, ethically, and morally with this invasion of privacy. Those collecting the data may see it as "wrong" that I feel this way and seek to punish me in some way for it, because this data point among others may give them power to oppress me. Within a democratic society I should have the freedom to peacefully disagree in this way, and that freedom is being eroded on a daily basis by these people and organizations.

Let’s suppose you have an actual solution, like a free open source platform that is used by millions of people and took you 7 years and half a million dollars to build, because you saw the problems coming back in 2011? A platform with 5 million users in 110 countries?

It seems the press of today won’t really care about solutions, but only highlighting a scandal. Am I wrong? What would one do to get them to write a story about actual solutions like solid.mit.edu or ours:


Press kit? News conference? Some kinda weird publicity stunt?

Serious question.

And they sell GPDR compliance solutions: https://www.palantir.com/solutions/gdpr/

Oh, the irony!

Is it though? It’s pretty hard for companies with lots of diverse data on their users to comply. Finding it all and deleting it is pretty hard. The hope I had was that companies wouldn’t track it at all. I fear that only companies that afford stuff like Palantir will collect it, because they can comply and show evidence of complying.

There is a lot of irony in capitalism

When a company is named after the crystal ball an evil wizard uses to spy on his enemies in The Lord of The Rings, and that company's founder was the first large investor in Facebook, and that same founder is able to fund company ending lawsuits against media outlets, headlines like this are akin to "water is wet"

After September 11, thre was a big push for "Visual Analytics," which is basically rebranded statistics plus visual data mining tools. There's a report called Illuminating the Path that lays out the research agenda. I read it in grad school when my advisor was trying to tag everything under the sun as Visual Analytics in an attempt to get access to the Visual Analytics money pot. (We didn't.) It's creepy to see it come back around, fully realized by Palantir.

Ignoring the enormous argument over whether what I am about to say is legal in the US/America's particularly constructed law...

Is it time to consider whether companies who's primary business is the collection, summation, synthesis, and re purposing of information should be defacto illegal?

Its not that having the government do this is meaningfully better, but I'm convinced that is a better overall path. The public seem more afraid and more willing to engage in a more serious discussion over ethics. At the very least, it would enable the public to have more oversight over how such data is collected, stored, embargoed (or not), and used. The level of abstraction and fuzzing that is allowed by allowing a non-governmental entity like Palantir or Facebook to have (whether as a collector or aggregator) the data and governments be only a user seems too difficult to regulate and even have a transparent discussion about because claims of national security and secrecy are nested within claims of trade secret and intellectual property in ways that create a moat around effective oversight.

So seriously...if we made this business model illegal, what do we lose? What exceptions would need to exist?

> Is it time to consider whether companies who's primary business is the collection, summation, synthesis, and re purposing of information should be defacto illegal?

You mean IBM/SAP/Oracle?

I don't think it is unreasonable within what I said to differentiate the industry that supports data collection and archiving as a broad process (e.g., those you describe) with those that seek and intent specifically to collect and operationalize data about people without fully informed consent.

Palantir does not do data collection in that sense. They're much closer to IBM than you realize.

Especially if you consider IBM's systems for tracking the Holocaust and Apartheid. Special purpose people-tracking systems.

> Cavicchia was in charge of forensic investigations at the bank. Through Palantir, he gained administrative access to a full range of corporate security databases that had previously required separate authorizations and a specific business justification to use. He had unprecedented access to everything, all at once, all the time, on one analytic platform. He was a one-man National Security Agency, surrounded by the Palantir engineers, each one costing the bank as much as $3,000 a day.

I've long thought that the "secret sauce" of Palantir is just that signing a Palantir contract gives executives a reason and excuse to pull together data that had never before been indexed against each other.

I mean, if you take any old dumb BI platform and give it access to data that people have not previously connected, you're going to get new insights.

Someone once described Palantir as a data mapping consultancy that marketed itself as a software platform.

Why isn't there a privacy watchdog that constantly monitors these kind of companies?

I also expect this watchdog to test for data leakage from one site/company to another (e.g. by taking on the role of customer).

They should do this on a frequent basis.

Without this, companies seem to be free to do whatever they want.


This is how the fucking government gets around the Bill of Rights, primarily the 4th. When it's outsourced, you have no rights. Parallel construction needs to be the next addition to the 4th or it's own amendment.

Sort of off-topic, but I have a need for a graph/relationship oriented database like this (on a relatively tiny scale) and am having a hard time choosing between the many alternatives. Open source is a must, but otherwise I'm primarily interested in ease of use and visualization compatibility - I want to focus on my problem domain, rather than writing a lot of code or hacking for its own sake. Right now looking at Neo4j and GraphDB, would welcome other suggestions.

Thanks in advance!

I think I heard the maxim once "If it can be abused, it will be abused."

We need to start getting ahead of worst-case scenarios and to hell with people that think that approach is paranoid.

Wouldn't be interesting if all the Facebook data being collected is really to feed info to Palantir...with a layer of deniability and they get data on billions.

This was a huge debate when the Patriot act was passed, that US citizens would get caught up in surveillance.

It seems common in history that during war time, a country aggressively pursues it's own population for spies or sympathizers for the enemy. It send also common that the country goes overboard so as not to let anyone through the cracks and a lot of innocent people get swept up in the process.

Why does the media seem to be publishing so many "well duh" tier articles about various companies at the moment? "Big data company collects big data", "company known for monetising private data collects private data", "political consulting company profiles voters and tries to affect vote".

No shit?

> Why does the media…

Here’s a handy formula next time you ask yourself a similar question:

Why does the media X?

Because people watch / listen to / read X.

Why do people read about companies collecting data? Because it sounds scary, and it blames someone who is not them. Fear + scapegoat + victimisation of people = read read read read.

Don’t think of it any more complex than this or you will drown in irrelevant details. They don’t matter, and they never will. Read some 19th century news papers to see this effect take hold with comical clarity.

yeah, good point. It makes sense how such clearly irrational articles could come about through those incentives.

There are obviously people out there for whom this is new information.

media will do this about every subject, think of it as the original analytics, if the story drew a lot of attention then another story similar to the first story will also draw attention because people will associate it with the original really interesting story that drew all the attention. Sooner or later the quality and interest of the stories on subject X will decline and people will not react so much to them so the writing of these articles drops off.

Currently, because of facebook, there is a spike of Company Y knows everything about you type stories, as well as a spike of Facebook is evil stories. These things permeate out and loose interest over time.

Here’s the problem with the journalism industry:

They love to write about scandals and problems. But if you have an actual solution to the problem, they aren’t going to publish it.

You can also see articles on court cases about some scandalous things, but can hardly find out how the case eventually turned out. And so on.

Journalism highlights problems not solutions because that’s what gets the most “outrage reshares”. The exception is some “cool new technology” that can have a story about it, but not as a SOLUTION associated to a lot of PROBLEM stories.

> They love to write about scandals and problems.

They particularly love to write about scandals and problems among their rivals. That's the real change here: journalists and bloggers have gone from thinking of Facebook (and other social media) as potential allies to thinking of them as rivals. Negativity sells, as you say, but negativity that reinforces one's view of rivals as evil is especially appealing. People whose own Google Ads revenue is declining because the eyeballs are elsewhere are very highly motivated, both consciously and unconsciously, to write about why those eyeballs are elsewhere when they clearly shouldn't be.

I think the problem is market economy. Media organizations are businesses which are organisms that focus on the bottom line. I.e. making as much money as possible. Usually the only way to fix these market failures ( I consider this dissonance between monetray value and actual long term value to be a market failure) is with formal regulations or non formal moral regulation. The former is tricky and the latter is requires cohesive cultural values and structures. Sadly (imo) the most effective coltural structures are religious.

Because what is clear to you, isn't clear to Joe public. Oh - and "company known for monetising private data collects private data" isn't the story. People kind of understand that. What they didn't understand is that "company known for monetising private data" actually sold the raw data, rather than using it for its own targeted advertising.

has there actually been any suggestion/evidence that Facebook sells the data? All I'd heard is that companies like Cambridge Analytica were using surveys and Facebook's at-the-time lax data security to scoop up data about the friends of people who answered their Facebook quizzes.

"A more productive answer to someone saying something you agree with is “I agree”, not mistakenly berating them for not agreeing sooner."


People have been aware of the basic facts for a while, but not its prevalence or significance. Also, nobody is as zealous as a new convert. Everyone wants to raise awareness of what they've just noticed themselves, even if others had noticed it long before. Thus thousands of articles have been written by journalists and bloggers about the same few things, while hundreds of HNers happily submit and upvote every one here even when they're redundant or poorly written.

>Why does the media seem to be publishing so many "well duh" tier articles about various companies at the moment?

Did you miss the whole Facebook and election thing?

no I didn't, that's partly what I'm referencing. What specifically are you pointing to?

Most non-technical people (and even many technical people) don't know this. I am myself surprised that the Facebook thing caused such an out cry now, but that's what happens when you bring these things to people's attention.

That's why they publish these kinds of stories. It's related to current events.

I realise this, but it stinks of artificial controversy. I get that a lot of people are unhappy that Trump was elected, but there are so many articles trying to spin Cambridge Analytica as some kind of shady political Illuminati instead of a me-too political digital consultancy, suggesting that Facebook collecting private data is a new shocking revelation, or now that we should be shocked that a big data processing government/enterprise contractor is processing big data for the government and enterprises. Alongside all the "democracy is broken" articles we're seeing, it seems to be all focused on harming the legitimacy of the election and implicating shady power players in the outcome, a play that would seem more at home on Alex Jones' show than splayed across every mainstream media outlet.

Not a bad idea to yell at someone asleep at the wheel.

I know this might be a surprise to you, but not everyone knows everything that you do.

I'm not exactly tapping my own research here. Palantir is a big data/spying company, what do you expect from them? Facebook asks you for lots of private data and it's clearly obvious that when you post data on Facebook, Facebook gets that data. So what's the shock that they hold this private data? The only surprising aspect of the current Facebook drama is that they apparently mine your phone for additional data but that's not the main focus of the outcry. Why are people surprised that Cambridge Analytica, a political consultancy, did political research and affected a voter outreach strategy? None of this requires any real background knowledge, and yet the outcry seems extremely disproportionate to the revelations at hand. It seems like the Snowden reports all over again, where the world was up in arms that an intelligence agency was collecting intelligence.

Most of the people knows nothing about all these things, until the media picks up the story. They at most believe Palantir is a Skype stone for elves and that Facebook makes money just running ads just like printed newspaper TV stations and radios run ads on very basic criteria (i.e. different ads during a soap opera than during an action movie), but they're not aware of the depth of analysis FB et al. perform.

Example from other field: aircraft passengers have breathed compressor bleed air since jet engines were introduced in the 60s, everyone in the industry knows this is how it works, and yet it wasn't until last year, when major publishers such as Fortune [1], The Guardian [2], Bloomberg [3], The Telegraph [4] etc. happened to simultaneously run this issue, that lot of the general public learned about it for the first time, most of them having assumed that cabin air was taken directly from the atmosphere.


[1] http://fortune.com/2017/08/09/dangerous-cabin-fumes-planes/

[2] https://www.theguardian.com/science/2017/aug/19/sick-crew-to...

[3] https://www.bloomberg.com/news/articles/2017-08-08/airline-w...

[4] https://www.telegraph.co.uk/travel/news/world-health-organis...

the thing is, articles about how facebook collects private data and analyses it have been running pretty much non-stop for the last few years. I would totally understand the reaction if that hadn't been the case.

> Palantir is a big data/spying company, what do you expect from them?

How can expect anything of them when I don't even know that company exists, let alone what it does?

> Why are people surprised that Cambridge Analytica, a political consultancy, did political research and affected a voter outreach strategy? None of this requires any real background knowledge, and yet the outcry seems extremely disproportionate to the revelations at hand. It seems like the Snowden reports all over again, where the world was up in arms that an intelligence agency was collecting intelligence.

This is simplifying the discussion a lot. The general assumption about both intelligence and research was probably that they stick to rules and are concerned with their own fields.

I'd guess the average joe would (before snowden at least) think that an intelligence agency is concerned with foreign diplomats and military strategists and a political consultancy does opinion polls.

It's something different if said agencies collect intimate data of random citizens without their consent. At least in public mind, that was something the stasi or similar organisations did - but certainly not their own side.

you raise a good point - with the intelligence agencies, I can believe that plenty of people think (or used to think, before Snowden) they are focused on specific valuable targets such as diplomats given that's how they supposedly used to operate before asymmetric warfare, terrorism and sole attackers became the median threat.

That is a life lesson that keeps on learning

Pop Quiz: Which two companies had their first office at 101 University Avenue in Palo Alto?

The answer says quite a lot, I think.

I've found Piazza's Fine Foods Inc and PhysioTouch Inc. Not sure what I'm supposed to take away from this answer though.


this is kind of scary and sad when powerful technology is being misused ...probably it is better to live in China ??? ..the government openly monitoring its citizen instead of spying on them

Definitely not, although that specific aspect - you _know_ you're being watched - makes life different, in some way, simpler. The rules are obvious, however, if they change, you might find yourself in a much worse situation, than in the West - for now.

> you _know_ you're being watched - makes life different, in some way, simpler.

Arguably, what happens when people are aware of surveillance is one of the worst parts of it. Following the Snowden revelations, there were noticeable chilling effects:


Reminds me of a friend who said he never wears a condom, because condoms can break.

So it's a double play on the name now. I.e. it's in the hands of Sauron?

Given Thiel’s purported views on privacy, the hypocrisy is overwhelming.

My main takeaway is that the corporations don't have the funds to waste on this and don't see the ROI (given how many large companies listed in the article stopped using Palantir) whereas the government does. Peter Thiel being a libertarian is clearly aware of the scale of government wastage and decides to go in for the win.

How can we build a case against them, and sue them? EFF?

> As shown in the privacy breaches at Facebook and Cambridge Analytica, the pressure to monetize data at tech companies is ceaseless

Says Bloomberg, and includes Facebook code on their pages

"the vast increase in welfare beneficiaries and the extension of the franchise to women [...] have rendered the notion of ‘capitalist democracy’ into an oxymoron."

That guy is a nice piece of work.

Does anyone else remember a time when paranoia was considered a mental illness rather than a virtue? Or am I imagining that?

i find the peter thiel conspiracy-theoretic spiderweb in the article ridiculous (i believe you could make a similar one for any well known vc). I wonder however why he invests in these companies. i mean centralizing intelligence seems a very anti-libertarian thing to do.

I can't wait that EU citizen will hit them hard after May 25th.

As the yearned-for fake privacy walls go up, the only entities that will still have the ability to abuse privacy, will be politically connected corporations and governments.

Germany supposedly has some of the best privacy laws in the world. Their government freely abuses their people with domestic espionage. They'll continue to work with technology businesses to that end as it fits their purposes.

There's a lot of outrage about Facebook. Meanwhile, absolutely nothing has changed, or is going to change, about the NSA spying. Nothing got fixed. More or less the Snowden revelations were entirely meaningless in terms of changing the status quo. The NSA is spying on EU and US citizens just as they were five years ago, just as they will be five years from now. Where's the outrage today about the NSA? Almost everyone gave up, that's why Facebook is the new target, the people couldn't do anything about the government spying.

GDPR creates the haves and have nots when it comes to access to abuse privacy. That's all it will change. If you have the political access (which Palantir does and will), you're going to get the security keys to abuse privacy (the alternative is for EU governments to entirely shut down all domestic espionage, that will never happen).

I think palantir is CIA related... They probably are not going to worry about pesky international regulations

Sanctions? I am not sure the EU has enough clout to pull that off against something the likes of Palantir, I am sure that they would have to know it is there before they could ask for removal and FIOA or other legal devices are not going to be enough pry that open under this administration.

That's what sanctions are for. But I highly doubt the EU will ever sanction the US as they are so dependant on them.

EU has more people and an equivalent sized economy.

The EU could stand up to the US if they felt it mattered.

Other than defence partnerships the EU is no more dependent on the US than the US is on the EU.

It's a two way street.

Unless you think the US would be happy to walk a way from the largest single market in the world (by GDP).

The USD is currently 62.70%[1] of all currency reserves. If you lose the ability to trade in dollars you are screwed and the US knows this.

[1] https://en.wikipedia.org/wiki/Reserve_currency

Except the EU isn’t a country. Poland doesn’t have the same incentives that Greece might, as an example.

Except trade deals and regulations are negotiated/implemented top down.

So the EU has teeth if the US pushed it too far.

It's hard to get the EU to agree on a lot of things but something sufficient to make the EU consider punishing the US would likely be important enough they would agree.

My biggest issue with former military people is that their skills don't easily translate into any other fields and that their mindset is usually heavily canted towards statism, which is a big problem when you combine the kinds of surveillance technology we have with needing to turn a profit. It makes anything East Germany did look like they were underachievers.

Culture flows from the top. Peter Thiel apparently views women's suffrage and poor people as "tough for libertarians [like him]". Can we talk about, well, how problematic it is that a company that has information on just about everyone (and all the power this yields) is lead by someone like this?

> As Thiel’s wealth has grown, he’s gotten more strident. In a 2009 essay for the Cato Institute, he railed against taxes, ­government, women, poor people, and society’s acquiescence to the inevitability of death. (Thiel doesn’t accept death as inexorable.) He wrote that he’d reached some radical conclusions: “Most importantly, I no longer believe that freedom and democracy are compatible.” The 1920s was the last time one could feel “genuinely optimistic” about American democracy, he said; since then, “the vast increase in welfare beneficiaries and the extension of the franchise to women—two constituencies that are notoriously tough for libertarians—have rendered the notion of ‘capitalist democracy’ into an oxymoron.”


Freedom and democracy are always in a conflict, that's why most democracies have constitutions which put the individuals and their rights first. A total democracy would mean that a majority can override personal freedoms and basic rights of individuals (like, vote about executing your neighbor). Because of that the combination of democracy with a bill of rights has become so commonplace that you don't even notice anymore, and think about someone who mentions that simple fact as a fascist.

But what kind of libertarianism is non-democratic?

Here's an interesting answer I found when I became curious about this question a few years ago:


Values will always be secondary to money in this world.

1. That's a gloomy self-fulfilling prophecy. Try thinking more constructively.

2. Money is a value, so your assertion is a paradox.

if libertarianism is equated with fascism then i believe we have at last reached peak fascism

edit: liberatianism is by its nature less demo-cratic because it reduces the "kratos" part of democracy but it is not fascist because it is not authoritarian. anarchocapitalism is its conclusive endgame.

Anarchocapitalism is authoritarian rule by capitalists, where property is valued above all other rights.

there is no rule or authority in anarchism. ancaps advocate a voluntary society.

Anarchocapitalism does not allow someone to opt out of the concept of property. That is not a voluntary society. Capital owners are the authorities in anarchocapitalism. A truly anarchistic society would have no private property.

of course it does. you can choose to create a commune, nobody is going to stop you unless you try to create it on the property they are homesteading in.

I didn't voluntarily agree to respect other people's purported ownership of property. They have authority to keep me off of their property that I didn't agree to. That is a non-voluntary society. Property ownership is inherently non-voluntary, otherwise it isn't ownership.

We are off-topic here, and it seems you ve made up your mind already so i ll just leave this: https://en.wikipedia.org/wiki/Anarcho-capitalism

the lack of democracy is not fascism. Anarchy is non-democratic but is pretty much the exact opposite of fascism. Thiel is a libertarian which is a political system that is neither inherently authoritarian nor democratic.

[edit] clarify "inherently". Libertarianism can be a lot of things given its primary focus on economics and contract law.

Since Thiel is a capitalist and not in support of democracy, that means he is either a fascist or an extreme libertarian. Correct me if I'm wrong, but mainstream libertarianism is still democratic, they tend to be supporters of government in the style of the original US constitution. That would mean that if Thiel is non-authoritarian he would be an anarcho-capitalist, which I have a hard time believing he is.

I see no indication that he's a fascist unless we have vastly different understandings of the term. His comment about freedom and democracy being incompatible reads to me like the classic issue of the inherent conflict between the interests of the individual (freedom) and the interests of the collective (democracy). I can only speculate what his ideal political system would be, but calling an individualist a fascist just doesn't fundamentally make sense.

As an example of a non-democratic libertarian governance system, consider minarchism - https://en.wikipedia.org/wiki/Minarchism.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact