
The TSA Randomizer App Cost $336k - andrewguenther
https://kev.inburke.com/kevin/tsa-randomizer-app-cost-336000/?lobsters
======
swanson
Let's assume a hefty markup for the long sales/BD time for a government
contract like this: $336k * 0.60 = $200k

Since this deals with national security, could probably justify higher rate
resources staffing (security clearance required?). Ballpark $250/hr @ 8 hr/day
so $200k buys you 100 days of capacity.

Project breakdown:

* UI/UX Design: 10 days (lots of stakeholders+approvals needed, maybe field testing with TSA agents, accessibility audit)

* Development: 15 days (this is very conservative but might be some business rules that aren't being shared -- remote monitoring, analytics, auditing, static analysis, etc)

* Project Management: 25 days

* QA: 15 days (multiple devices, high perceived security exposure)

* Third-party security audit/pen testing: 5 days

* Technical writer for documentation and training materials: 5 days

Total: 75 days of capacity spent.

Math doesn't work out exactly and some of these numbers are pulled from thin
air, but it seems like it's at least in the right ballpark.

Do you folks really know people that are doing government contract work (for
the TSA no less) for less than $100k?

~~~
cha5m
Yeah clearly this ui [http://elliott.org/wp-
content/uploads/IMG_6317.jpg](http://elliott.org/wp-
content/uploads/IMG_6317.jpg) required 10 days to design.

Why would this require more than a few days? Simple UI, then just generate a
secure random number and draw an arrow. Bam. Give me 200k.

I mean the largest expense would almost certainly be third-party pen testing
and QA. And why would it take a technical writer 5 days to document this? It
has 1 function.

~~~
swanson
That's one screenshot, are you prepared to submit a legally binding quote for
the project with that little information? Are there more screens? I don't
know. Did the client require multiple designs and iterations -- each stamped
with the approval of several stake-holders? I don't know.

Judging the final output without any consideration for the process is akin to
asking why it took a week to fix that hairy bug. It was only one line of code
that changed, how could it possible take you all week to fix it?

~~~
SilasX
Sure, since, in government contracts, "legally binding quote" doesn't mean you
actually start losing money or have a judgment issued against you that you're
personally liable for if there are time or budget overruns.

------
seibelj
I worked at a company that had to resell its software through a disabled,
veteran-owned sole proprietorship so that the TSA could find budget to
purchase it. The way it was described to me was that a portion of the budget
is earmarked for these businesses, so if the primary technology purchasing
budget runs out, you can still sell to the TSA if you resell through one of
these businesses. Essentially, the reseller took a cut for doing nothing, and
fleeced the American taxpayer for a few hundred thousand dollars. I'm sure
this happens all the time.

~~~
rhizome
To be fair, if the taxpayers are going to be victims of bureaucratic
inefficiencies, I'd prefer that the beneficiaries be disabled veterans vs.
that money going to Boeing or Raytheon or Palantir or whoever.

~~~
gaur
I still can't believe someone named a company Palantir. In Lord of the Rings,
the palantíri fell into the wrong hands and were used to do literal, honest-
to-god evil. It's like naming a company Death Star.

~~~
jlebar
_It 's like naming a company Death Star._

Or like making a mission patch that's an octopus with its arms around the
earth, with the tagline "nothing is beyond our reach"?

It's all about audience, right? Palantir used to have t-shirts that said "save
the shire" on the back -- dunno if they still make those.

------
notmyemployer
I've done some federal contracting and the issues seem to be cultural. The
government hands out what they call "prime" contracts which are then
subcontracted out to multiple other firms. The primes tend to be stodgy old
companies filled with lawyers and MBAs that can win the contracts, who then
view the engineers as replaceable cogs. You then interact with multiple other
contractors that own particular parts of the stack, for example an independent
testing contractor and another for infrastructure. It wasn't uncommon to have
5 project managers for each engineer on the project.

Coming from tech startups this was completely shocking, I was so used to an
engineer driven culture. That developers couldn't write their own tests, or
manage their own infra, or deploy multiple times a day was super frustrating.

There sounds like there are some great initiatives to change this old
approach, but until then I can't imagine many talented devs would put up with
the bureaucratic bullshit.

~~~
thaumasiotes
> That developers couldn't write their own tests, [...] was super frustrating.

I can see advantages to having someone else write tests for the developers.

And I don't see how they stop you from writing tests for personal use, if you
want to.

~~~
rplst8
In the aerospace industry (commercial and defense are similar here because the
software for commercial has to be written according to government rules, see
FAA DO-178B) it's actually verboten. Testing independence is part of the
flight certification of the software if I'm not mistaken.

~~~
thaumasiotes
This has all the enforceability of prohibiting your workers from thinking
about what they're working on, or practicing in their spare time.

Tests you write for your own benefit are not a deliverable. They're part of
your work process.

As I said before, it's not ridiculous at all to require that _deliverable_
tests be written by some other party than the developer whose code needs to
pass them.

~~~
EdHominem
Actually I think this is one of those things that sounds so obvious, and yet
isn't right in practice.

I'l roughly segment projects into those that can fail a bit and those that
can't.

Now obviously, for things that can't fail even a little you can't just take
someone's word. But why can you take the word of two people, or ten? The
failures (in programming and code review) are absolutely correlated. If one
person fails at something another likely will fail in the same place or when
reviewing it. When people die if there's a bug, or you're even losing a lot of
money, you should not be trusting best-effort human _anything_. This is where
imperative programming against a test suite breaks down and just can't cope.
You _need_ to switch out your internals for something provable in its domain.
(eg, the math to land a space shuttle in a fixed-time infrastructure (ie
before it lands itself)).

And for everything else, it's a $/$ calculation. How much do you want to spend
to have some unknowably smaller amount of risk? And usually the best value for
the dollar is having the original engineering team work on the tests, with
outside oversight and good metrics. If an integrated team is having problems
getting full branch coverage (the only worthwhile coverage metric...) in a
given method, they rewrite the method. External teams have to test what's
given (or waste a ton of time in communication delays) and that usually ends
up with suboptimal tests _and_ suboptimal coverage. What you can't do though
is simply ask a developer if their work is properly tested and trust their
answer. You need real metrics and to know what they mean for you. (Like
benchmarks.)

~~~
RyanZAG
The tests are written independently of the original code. If someone has made
a logic error or misunderstood some part of the spec, there is a good chance
that the independent test implementation is not going to have the identical
error. It's definitely a higher chance of finding bugs than having 1 developer
write both, as long as the testing is written independently based on an API.

~~~
EdHominem
There's a difference between code tests though and spec verification.

Spec verification is (should be...) black-box integration testing. And yes,
this is a good chance to get a second set of eyes on the spec to make sure
nothing was missed in the implementation. (unit-tests can't catch missing
code!)

Code tests should be more white-box and should be measured against line/code
coverage instead of spec coverage. These are what I think the original devs
should write, and should be 95%+ of the total test volume.

The problem with expecting testing to pick up spec errors is that it takes
comes at the end of the design/build phase instead of the beginning when you
can make changes easily.

~~~
RyanZAG
Well you're not going to pick up spec errors with same-developer unit tests,
so either you have someone else write those spec tests or you wait until the
customer picks them up on initial delivery / QA.

------
noja
"..that a beginner could build in a day.."

Yeah... the clue is that an entire bureaucracy is required before even the
first line of code was written. This thing probably took months and months of
meetings with tons of people before the prototype was written.

~~~
drdeadringer
What would happen if, some fraction of the way through the long stretch of
bureaucratic meetings, someone just coded a prototype?

If they sit on it and whip it out the moment the green light is given, then
what?

If they whip it out at the next red-tape meeting, then what?

~~~
grecy
I can tell you what happens. I worked at a very large telco.

We went through months of daily meeting with 10+ very expensive execs, PMs and
Business Primes from all over the business.

One day the only other coder in the room leaned over to me and said "Are we
still talking about <x,y,z> that you or I could code in 10 minutes?"

Yes, we were.

I coded it and showed my managers and so forth. I was completely ignored, and
the meetings rolled on. In a place like that, there are so many people that
have job titles that have nothing to do with "getting it done", which in a
funny way makes them less interested in getting it done. Their job is to
analyze, plan, document, process map, etc. etc. so that's what they're going
to do, no matter how trivial the thing is.

~~~
AndyNemmity
Exactly! I would argue it creates MORE red tape and work to do that, than
less. I've done it, I've seen it, it actually makes your life worse, not
better.

~~~
grecy
When I stopped trying to make that place better (I'm banging my head against
the wall) and started just going with the flow, I knew it was time to leave.

------
slagfart
Honestly, for a custom app from one of the largest corporations (IBM), for the
largest of organisations (US Govt), $336k is pretty reasonable!

~~~
AndyNemmity
Seriously. 336k is absolutely nothing. Think about how fast that gets used up
adding up the numbers of the per hour of those involved, and how many people
it is (not just the developer who does it)

You can argue that you don't need that many people, and certainly for a start
up that's often true.

But in enterprise software, you actually do need a ton of people because of
the expectation.

I'm in the Business and came from start ups. I'd VASTLY prefer to do things in
small groups to just get things done.

But even though I want that, no one else does, including the customer. They
want all the bells and whistles of a ton of people. It makes no sense at all.

Hell, one of my current projects has about 20 people on a phone call, and 4
that actually understand anything going on. Think about all the money wasted
in all of that, and yet... It's the way business "works".

------
pratheekrebala
The $336K appears to be just the cost of one installment. The total project
costed $1,444,315 Here are the rest of the transactions under the same
contract:
[https://www.usaspending.gov/Pages/AdvancedSearch.aspx?k=HSTS...](https://www.usaspending.gov/Pages/AdvancedSearch.aspx?k=HSTS0313JCIO494)

TSA spends most of it's budget through a provision called "Other Transaction
Authority" which is essentially a vehicle to make purchases with barely any
oversight from congress. [http://time.com/4134368/tsa-price-of-
security/](http://time.com/4134368/tsa-price-of-security/) (Paywall)

~~~
jdmichal
I'm glad you brought this up, because this comment represents a
misunderstanding in how government contracting is typically done. My previous
job was working for a government contractor, and we did multiple discrete
contracts for several branches of the government. However, _every single one
of them_ would have shown up exactly like this, all bundled under one
contract.

Why? Because writing and signing a contract is _expensive_ , for both the
contractor and the government. So contracts are typically written in a way to
make them very easy to extend, and existing contracts are often used as
vehicles to tack on additional funding for new contracts. The original
contracting agency would also usually charge a fee to the other agencies for
use of their contract in this way.

So, this kind of glance at individual awards under a single contract is really
too simple a view. A lot of these funds could be (and from my guess, probably
are) going to completely independent projects from the randomizer.

TL;DR: Having multiple awards under a single contract is typically a sign of
the government working _around_ bureaucracy and attempting to save costs.

~~~
pratheekrebala
Yeah -- you are correct.

TSA later reported that the actual cost of the randomizer was around $47k. The
total figure I had mentioned earlier was part of a larger contract with IBM. I
saw the generic "IGF::CT::IGF MOBILE APPLICATION DEVELOPMENT" note on the
award and assumed it was all for the same project.

It's also weird that TSA's response to OP's FOIA request cited a different and
higher figure (~$340k) for the randomizer app -- I'm guessing they were
grouping in other projects under the same award here too?

Besides, $47k seems like a reasonable amount if that figure includes training
and deployment costs.

------
valine
The government would be a lot more efficient if everyone who worked for them
was a little more tech literate. I don't think the lack of education is
entirely the US government's fault. Tech moves so insanely fast it can be a
full time job just keeping up with the changes. My point being, this is not
the price of building a tiny app. 336k for a randomized arrow is absolutely
ridiculous. This is the price for making a change in a government run security
system. A change built on technology that government officials don't fully
understand.

~~~
brown9-2
This assumes all inefficiency is accidental, or done out of a lack of
knowledge.

~~~
CyberDildonics
Don't attribute to malice what can be attributed to ignorance

------
mikeryan
A few things.

1\. Kevin you may want to block out your address in the attached letter.

2\. I'm missing the part where this amount is tied directly to the creation of
this application and this application only. From my read this is a T&M
contract for mobile development there's nothing specific to this application.
It could span multiple engagements and apps. Considering this contract went
into place over 2 years ago this could likely be the case.

3\. This is a cap not an invoice. They could have billed 60k.

~~~
sailfast
Some details of past IBM wins here via UsASpending:
[https://www.usaspending.gov/Transparency/Pages/AwardSummary....](https://www.usaspending.gov/Transparency/Pages/AwardSummary.aspx?AwardID=17381939)

This may be the preceding contract but very similar scope and line items if
you want more details, no FOIA required.

------
cantrevealname
I suspect that one secret feature of this app is a discreet way to override
the randomizer and send a passenger to the "intrusive search" lane. That is,
if the screener suspects someone, he or she can swipe or click in a
clandestine way to guarantee that the passenger gets the annoying search.

The passenger can't complain about being targeted or profile or singled out,
because, hey, it's random.

(The actual rationale of the randomizer might have been to avoid accusations
of profiling, _while still being able to do profiling_. Leaving the bad guys
off balance is simply a benefit.)

------
caseysoftware
Notice this is also T&M (time & materials) which is not just a fixed priced
bid.. so that means someone "used" the actual working hours and materials for
the $336k bill.

(Contract negotiation, etc is not explicitly built into these, they're
reflected in a higher billing rate.)

~~~
fisherjeff
No, this is just the contract, which would be based on IBM's cost estimate.
Since it's T&M, the actual total cost and hours will only be reflected on
IBM's invoices.

------
kra34
Yo app got a $1,000,000 from Andreessen Horowitz, so they kind of got a deal
if you think about it that way

------
gojomo
The PDF contract the author shares doesn't make it clear what deliverables it
covers. Perhaps we can assume that if the FOIA process worked it is in fact
the contract that covered that app... but it's not clear from the document
itself, nor is it clear that the 'TSA Randomizer App` was the _only_
deliverable of the contract.

~~~
kevinburke
Yes, it was unfortunate I didn't get more data/information back from them :(

------
woodman
Before we even go into line item mode we really need to ask ourselves if this
is a $336k problem in the first place. They could have just asked the military
how they handle this sort of problem, and the solution they would have been
given is this: a $2 hand tally counter. Establish what the desired flow rate
is, divert every nth person, reset the clicker, anybody in line observed
shifting position gets diverted. Unless the TSA can't even trust their people
to count, or their real objective is adding some weirdly alienating layer of
technology between the employee and the traveler.

~~~
jcrawfordor
"every nth person" selection is just too predictable. You could try to prevent
people shifting in line, but in a crowded airport you will never be able to
control that effectively at current staffing levels. The cost of effectively
enforcing that rule would rapidly add up to more than $300k and cause serious
inconvenience to travelers.

~~~
mikeash
Yes, preventing shifting in line doesn't help if your attacker can just count
the length of the line before they get in it. You really want a random
element. Anyone familiar with Dungeons and Dragons should be able to name the
obvious solution there....

~~~
mfukar
But Mike, what good would Polymorph do in this scenario?

~~~
mikeash
If you can't figure that out on your own, then you're _obviously_ unqualified
to be a TSA officer.

~~~
mfukar
Haha, bummer.

------
tootie
Once again, contracts are rarely for just development. It likely included
design, development, deployment, testing, training, accessibility testing,
analytics.

------
factorfractal
They could have used a magic 8 ball with similar outcomes

~~~
venomsnake
Dice app that could support even 6 lanes - free.

[https://play.google.com/store/apps/details?id=tobi.wuerfel&h...](https://play.google.com/store/apps/details?id=tobi.wuerfel&hl=en)
[1]

[1] Not affiliated or tested. It was the first I found without inapp purchases
or ads.

~~~
AndyNemmity
But as a service it isn't controlled. What if someone took control of it, and
then used it to game the system. Security is shot.

Not that I really think that matters at all, but that would be shot down in
the first meeting it was discussed immediately, and you'd be considering
insane if you kept pushing it.

------
LoSboccacc
Couldn't find in the contract if the hardware was provisioned or not.

------
lowboy
Why is there a ?lobsters qs on the submitted url? Seems to load fine without:

[https://kev.inburke.com/kevin/tsa-randomizer-app-
cost-336000...](https://kev.inburke.com/kevin/tsa-randomizer-app-cost-336000/)

~~~
kevinburke
I submitted it to Lobste.rs with that on the end of the URL, and someone must
have submitted the URL without stripping that :)

------
jemfinch
If this is truly the price to develop such simple software, it's going to be
very bad for the government's reputation to be releasing code like this under
open source licenses.

~~~
Pxtl
Theyve already said they won't if there's a security or privacy risk, and you
can find one of those in _any_ code.

------
spilk
No invoices against the contract? This is only part of the picture.

It's a time and materials contract which means the contract award is the
"ceiling", but you have to perform work billed against it to get paid any of
that.

Plus, the second page details several extension options that could increase
the ceiling to $1,176,280. Were those options exercised?

~~~
kevinburke
We don't know; I've amended the post to reflect this. Thanks.

------
makecheck
One-sided negotiation is a cost riser. The last thing a client should be able
to tell is that the payer has very deep pockets; if I know you can _afford_
billions, I might ask a ridiculous price and expect you to not know any better
or not care about paying it.

An excellent use of secrecy technologies would be government contracts:
partition problems into pieces that don’t necessarily reveal the final purpose
or customer (e.g. “choose uniformly between A and B” is too vague to be
guessed as a requirement from only a government agency). Then, anonymously ask
for bids. That way, you might actually get the developer who offers a measly
price for a trivial task, and only finds out later that his check is from the
government.

------
intrasight
What is WAY worse is that they probably purchased 10,000 ipads to run this
stupid app.

------
kumarski
Less worried about that and more worried about the fact that baggage scanners
only scan from one angle combined with reality that they don't do material
differentiation.

Z-effective scores...findable.

------
bbcbasic
But no one got fired for choosing IBM

~~~
FriedPickles
I came here to say this.

~~~
bbcbasic
Pfft! Yet another cliché slinger

------
jsprogrammer
The page (2 of 8) that had the unit prices censored (exception (b)(4))[0],
claims the total award amount was $1,176,280.72.

Really, what we need is IBM's invoices and TSA's pay statements.

Also, you shouldn't need to file an FOIA request to get this information.
Shouldn't all bids and contracts be public?

[0] Exemption (b)(4): Records that contain trade secrets and commercial or
financial information obtained from a person that is privileged or
confidential.

~~~
AndyNemmity
Unit prices are trade secrets. If they published that, then someone else
getting a higher price could point to it, and say they only charged X for
them.

------
jonhohle
A coworker and I were laughing about this at the airport one day. I literally
wrote the app in the flight home and almost put it up on the App Store.

------
jqm
Ya well, that's 10K development, 30K profit and 290K meeting time and paying
people to fill out forms. Sounds about right.

------
tn13
Why does this surprise anyone? Forget an app. Just a normal computer that
government buys ends up paying 1.5 of what you will get on Amazon.com. This is
because government does not pay on time, there is a lengthy and resource
consuming process in selling something to the government.

The app was probably $10k, rest was the compliance cost.

------
venomsnake
It was 300000 for lobbying grease to land the contract, 30000 profit, 6k for
hardware and 500$ for one intern to write it.

~~~
AndyNemmity
Funny but ultimately wrong.

The real distribution of where the money goes on projects is certainly funny
from a sane persons perspective, but in a different way than this :)

------
blakecaldwell
Why not just have TSA agents flip a coin, or buy a bunch of Trouble board
games, and repurpose the dice rolling bubble?

~~~
EdHominem
The trouble bubble is not only a perfect solution for this, but an apt
metaphor for the whole process.

------
brokentone
We're talking about the app with 1 arrow on the screen that responds to a
press anywhere on the same screen? 1 icon, 1 tap target, hooked up to 1 secure
random algorithm.

I don't know how anyone can look at this situation and not see the government
is wasting a lot of money in their pursuit of not understand technology.

------
transfire
Hmmm... at the current rate of bureaucratic growth, I estimate in a 100 years
or so, this app (or it's modern AR equivalent) will cost 20% of the entire
federal budget. No worries though, 10% of all U.S. workers will be involved in
its digital-paper-pushing development.

------
ThinkBeat
With a reference to another recent hackernews story:
[https://news.ycombinator.com/item?id=11415747](https://news.ycombinator.com/item?id=11415747)

Can we expect to see the source code for this application pretty soon?

------
kevin_thibedeau
It seems a bingo spinner with two colors of balls would do just as good of a
job.

[https://www.google.com/search?q=bingo+spinner&tbm=isch](https://www.google.com/search?q=bingo+spinner&tbm=isch)

------
GreaterFool
A little bit tangential: when you go to the slower lane and they swab you with
those silly machines, can they _actually_ detect anything?

~~~
EdHominem
fyi, this is not a time to apply science.

------
wkoszek
It'd be interesting to hear what do "materials" can entail. It might be that
they actually delivered iPads to the airport.

------
abritishguy
If this is what it seems then that is ludicrous.

------
joeld42
What I want to know is does it still work if they turn on the rotation lock
and hold the iPad upside down?

------
brandonmenc
If you're a consultant or freelancer, and you're not charging that much to
build an equivalent app, you are undercharging.

This is a completely reasonable price. Honestly, it seems cheap.

------
satyajeet23
random rand = new random();

int tsaNumber = rand.nextInt();

