
Why Does DARPA Work? - MKais
https://benjaminreinhardt.com/wddw
======
maxander
A big part of the "DARPA formula" that doesn't seem to get much treatment here
is _consistency_. DARPA is going to spend a few billion this year, and it
spent a few billion last year, and it spent a few billion twenty years ago,
and you can feel confident they'll spend a few billion next year. That means
that if an academic researcher or similar can position their lab in an area
that DARPA likes, they have a good shot at reliable funding for the length of
a career. With all the economic uncertainties around research, this is
_tremendously appealing_ and will cause a lot of smart people to re-orient
their entire research programs around DARPA-friendly subjects [0]. A funding
organization that might disappear in a few years, by contrast, will get plenty
of grant applications, but won't attract nearly the same level of researcher
devotion.

(As a side note, I want to pull out a quote I thought was really nice, hidden
a ways in:

"DARPA funds wacky things that go nowhere. DARPA programs have a 5—10% success
rate and have included things like jetpacks, earthworm robots, creating fusion
with sound waves, spider-man wall climbing, and bomb detecting bees. _You
can’t cut off just one tail of a distribution_.")

[0] Which may, incidentally, go a ways towards explaining why DARPA heads
describe their projects as "idea-limited"

~~~
Aperocky
creating fusion with sound waves (or shock waves, which is essentially what
sound becomes at high enough energy) is one of the crazier ideas I've heard..

~~~
eximius
It's literally one of the ways they make nuclear bombs so it isn't that crazy.

~~~
Aperocky
Well yeah, if your shock waves for fusion are created by nuclear fission chain
reaction that technology has been quite ready and mature since the 1960s.

------
riazrizvi
IMO this analysis gets caught up looking for answers in process when the real
problem that DARPA solves is a political-economic one.

ATT didn't invent the internet not for lack of process, they were extremely
innovative. They held back on innovations because they were making huge sums
of money overcharging long distance mainframe-mainframe data link rates in the
50's-70's. DARPA succeeded as a trust buster.

They have the resources and legality to make whatever they want, regardless of
patents (because National Security/patent law). They can make technology a
reality, then develop political support with working prototypes. They build
things to show how the Government is getting screwed over by some giant
defense corp, because of a lack of competition in certain types of contracts.

I think they are becoming less relevant because the corporate-political
landscape is becoming more trust-based.

~~~
Animats
Back in the early days of the Internet, I met some of the Bell Labs people
working in that area. What they didn't like about an IP-based network was the
jitter. That was totally unacceptable for voice. They wanted something with
reliable clocking, and had come up with Datakit. That sends all packets for a
given call over the same path, in order. You don't open a virtual circuit
unless all the nodes have enough bandwidth for it. Telcos still use Datakit,
and Asynchronous Transfer Mode is a successor to it.

I never dreamed that people would accept the degradation of telephony to 1
second of delay jitter with random dropouts and echoes.

~~~
bane
I remember when cross-oceanic long distance calls required both parties to
shout as loud as they could into the phone -- and often still not be heard
well enough to make out what they were saying.

Modern IP telephony often has very high quality voice reproduction (my wi-fi
to wi-fi Fi calls sound fantastic), in exchange for some timing issues. Echoes
usually get solved in software (usually), and dropouts seem to be the main
complaint.

In exchange, my wife can call her family in South Korea for approximately
nothing, using the same data backbone as we use to watch movies and read web
pages.

~~~
justicezyx
You can have ip over time multiplexing data link layer, like the IP over ATM.
I believe they were still used in the core voice networking.

IP won because they are more flexible, and open. And the Internet cement the
win because of that.

That's way planning ahead too far works less and less successful for bigger
and bigger project. Too much dynamic is embed in the long wiring process. It
becomes impossible to have a plan work out correctly.

------
darpa_commentor
DARPA is effective, but this document paints a rosy picture that is far from
the truth. I've been a part of DARPA projects both commerical, university and
on the government side.

The real process is:

1\. PM gets picked due to knowing someone or being a former employee.

2\. The biggest test of a program isn't if it's doable or a good idea, but if
its able to be transitioned to another government agency with deeper pockets.

3\. Most contracts are lost before you begin writing, as people have insider
information about what the PM wants. This is done through just talking with
each other (remember that most of the PMs come from the same companies), and
not through any other formal process.

4\. DARPA has some really cool stuff, but fails to transition it well enough
(leading to 2.)

DARPA is not without it's problems, but has a better track record then NSF
(NIH has them beat). What is funny is that you quickly realize how much bunk
there is in scientific research and how many papers are not replicated.

~~~
dnautics
this aligns with my experience. NIH has a better track record than DARPA,
which has a better track record than NSF and the worst is the DOE. The DOE
knows this and is trying to cargo-cult DARPA through stuff like ARPA-E.
Predictable results have ensued.

~~~
raziel2701
What does a better track record mean? More papers? Products? Social impact?
Why is DOE the worst?

------
currymj
A bill, the Endless Frontiers Act, has been introduced in both houses of
Congress to make a section of the National Science Foundation that works more
like DARPA, while massively increasing its funding. The idea is to extend the
DARPA model to many more non-military areas, just as suggested in this
article.

[https://www.sciencemag.org/news/2020/05/us-lawmakers-
unveil-...](https://www.sciencemag.org/news/2020/05/us-lawmakers-unveil-
bold-100-billion-plan-remake-nsf)

Certainly a timely article. I don't know whether it's a good thing or not. The
DARPA model has certainly been productive, but it isn't suited for every
research topic or subdiscipline.

As discussed in the article, currently NSF is the most open to basic science
of funding agencies, and gives grantees the most latitude in what they work
on. That is a valuable thing to have in the ecosystem.

If Congress expects the majority of publicly-funded research even from the NSF
to be on short-term grants for specific visions and technologies, it will rule
out working on a lot of important things.

~~~
siege_engineer
Author here. I actually wrote a discussion of the Endless Frontier Act:
[https://benjaminreinhardt.com/the-endless-frontier-
act/](https://benjaminreinhardt.com/the-endless-frontier-act/)

I agree that the ARPA model is not suitable for all (or even the majority of
research) and the act doesn't inspire confidence that shifting the NSF towards
a DARPA-like model would do well.

~~~
lazyjeff
Just wanted to say, thanks for writing the article about DARPA, and this
analysis of the endless frontiers act. From reading both articles, obviously
you're very pro-DARPA and anti-NSF.

I have two thoughts that might erode your thesis a bit, that 1) I think you're
overestimating the amount of time that faculty spend on writing grants. We
like to complain about it, but I've tracked my time to the minute over the
past 7 years, and grant writing (both the proposals and the reports) is about
2% of my work time. 2) You might be misunderstanding indirect costs like many
people, where you say "Universities can take more than half of grant money as
administrative overhead". Indirect cost math is funny, but in order for
administrative overhead to be more than half of the grant, the indirect cost
rate would have to be over 100%. I can explain more if you're interested.

Obviously we're both biased by our job, but it's still useful to read your
perspective.

~~~
siege_engineer
Appreciate the correction about indirect cost math - I’ve applied for several
grants but never did lab accounting so I interpreted it incorrectly when
professors told me “50% overhead.” I’ll fix that when I’m at a computer.

I would characterize my position as less “pro DARPA and anti NSF” and more “I
think on the margin the world needs more DARPA-like activity more than it
needs more NSF-like activity”

------
thoraway1010
Small anecdote, but good experience with DARPA and SBIR both. One thing I
liked - they seemed pretty low overhead / low hassle and outcome / objective
focused.

That contrasts VERY strong with most govt contracting which is under allowable
cost or cost reimbursement setups where the absolute most important criteria
is to bill enough costs to draw contract in the right cost buckets (which can
be super annoying if local agencies require super complicated budget mod
processes).

In the cost based contracts, the focus really focuses on the accounting for
the costs and other compliance related items. Ie, did you send someone to a
conference with govt money, how can you prove you didn't use the govt money in
this way or that way etc etc. Bam, welcome to personal activity reports with
fund codes that no one understands (ie, major universities have an insane
number of codes), and all the nightmares that follow including a fair bit of
rule pending that even normally ethical folks find themselves being asked to
to get through the paperwork.

Seriously, you deliver the product at 50% of cost? You will get a nasty note
from the head of your agency saying make sure you draw full contract because
agency budget depends on the indirect portion of this award (ie, 30% to
overhead) and even the govt agency supervising (who also budgeted based on a
cut of full contract) AND other folks for whom leftover money makes life
difficult (harder to close contract etc) AND because a lot of govt funding is
on the repeat what we did last year model so drawing down everything avoids a
cut next year when you may really need it.

You determine it would be cheaper to do x vs y and that required a budget mod?
Wait 1-2 months for commission approval if you even bother trying to fight it
through (changes < 10% often ignored thankfully).

I don't know how the accounting for SBIRs etc work, but somehow those projects
always seemed more results oriented (so a LOT more fun to work on, focus is on
getting a solution going).

~~~
peterwoerner
I used to work for a company doing SBIR work. We finished a contract
underbudget returned the surplus money and got audited because of it.

I don't know all of the accounting details, but SBIR has some leeway (because
its supposed to be research) but it's not perfect. However, you are allowed to
use surplus money to fund development of things which are related to what you
actually put in the proposal/contract. So if you have a $100k contract, and
meeting the requirements only required on the $75k, you should and do spend
the rest of the $25k on extra features. But you have to spend all the money.

I think the real thing about SBIR money, is that three people get phase 1
money and only one or two will get phase 2 money so you really have to deliver
in order to get selected. Then if you get the phase 2, there basically isn't
phase 3 money, so you better get to a product that is ready to be sold.
Finally if can't show on paper a return on investment (e.g. nonSBIR funding or
revenue from sales/acquisition) they cut you off from the SBIR spigot. So the
incentives align with getting work done.

~~~
m-ee
I worked at company in SBIR phase 2 and one quirk I found strange was that the
money could be used for R&D not things deemed “production”. Endless 3D prints
were ok, random software features were ok, tooling for injection molding was
not. It wasn’t a particularly onerous requirement, but I think reinforced the
tendencies of our former academic founders to spend a lot of time on research
and none on development.

~~~
peterwoerner
Part of it is figuring out how to classify "production" as R&D. Well we need
to test a prototype and in order to that we need it made as the same was as
production due to structure property process relationships.

Or manufacturing defects is going to be a huge source of _bad thing_ so we
need to do some research into the manufacturing process.

------
dsukhin
The irony is palpable:

> _I would rather this be read by a few people motivated to take action than
> by a broad audience who will find it merely interesting. In that vein, if
> you find yourself wanting to share this on Twitter or Hacker News, consider
> instead sharing it with one or two friends who will take action on it. Thank
> you for indulging me!_

I'm glad of course it was shared here. As a distillation - I think the
author's theme lies with enabling more __researchers__ rather than business
people to take moonshots and do foundational knowledge building and discovery
that redefines a field and _then_ focus on commercialization from a birds eye
view by technically capable visionaries.

This is the SBIR [1] model (also a US Gov requirement to fund small business
research for any federal agency with >$100M in funding), the Bell Labs model
(which yeilded amazing foundational work like UNIX and the transistor and was
a direct result of AT&T's monopoly and excess resources), and perhaps even the
YC model (though that one is obviously focused on a shorter horizon and more
on commercializing existing tech and more rarely on foundational research).

I've personally thought about this problem a lot and done this at a small
scale and would love to expand upon it.
[https://augmentedlabs.org](https://augmentedlabs.org) Would love to hear
others experiences and thoughts.

[1]
[https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Re...](https://en.m.wikipedia.org/wiki/Small_Business_Innovation_Research)

------
dnautics
Does DARPA work(anymore)? The first example of how it works was it's "badass
program managers", but all of the program managers I've met have been, to put
it lightly, idiots. One of the worst was the PM in charge of the entire
subject research division. It turned out, they had a graduate thesis that was
entirely observing an instrument artifact (showing bad jugement). Not too long
after I interacted with them, there was a minor scandal in the community
because postdoc who struggled with reproducing the effect and complained about
the results was abused and railroaded by the PI in charge of the project.
Shortly thereafter the PM didn't seem to be a PM anymore.

~~~
throwawaygh
...so your best anecdote about why DARPA doesn't work is that they fire
incompetent PMs?

~~~
dnautics
Yes, after being promoted to the head of the division with several people
speculating that they were being groomed for even higher posts. And it took a
scandal to get rid of them. Person should never had been a PM in the first
place.

You never hear people say "it's proof that Congress works because they ejected
that pedophile (sending lewd messages to the pages) from their seat".

~~~
throwawaygh
Hate to break it to you, but "an incompetent person rising through the ranks
quickly before crashing and burning" is something that happens in pretty
literally every large org.

Any 60 year old organization with a large budget is going to have multiple
instances of this happening.

~~~
dnautics
Perhaps I have a flair for the dramatic, but consider my statement that I
haven't met a competent pm in the division I was in (n~5). The incentives are
not that well-aligned. Think about it this way. If you're really smart, why do
you become a DARPA PM, and not a PI?

Yes there are a few good reasons, but the population it applies to have
largely been selected out by the postdoc phase (where I would say you have had
sufficient experience with crushing scientific and engineering failure onself
and watching others to be effective) and the pool of candidates therefore is
vanishing.

~~~
throwawaygh
_> If you're really smart, why do you become a DARPA PM, and not a PI?_

Reasons are myriad.

100% of the PMs I've worked with had tenure and successful labs prior to
becoming PMs, and went back to their institution & restarted their labs after
leaving DARPA. The reason for leaving your lab to be PM is fairly obvious:
controlling the funding gives you a lot of leverage for shaping the priorities
of the field, in a way that merely running your own lab doesn't.

Maybe your division sucked. I've only ever worked with highly competent PMs,
and all of the programs I've worked on ended in commercialization.

Anyways, prosecuting individual cases doesn't seem like a particularly good
way of evaluating the effectiveness of an agency.

~~~
dnautics
Well, it's pretty obvious that DARPA has different standards in different
divisions (division I was in was relatively new), then. Which still makes me
wonder wtf they are anymore.

~~~
throwawaygh
_> division I was in was relatively new_

That makes sense.

------
killjoywashere
It's worth viewing DARPA from their customer's perspective. What DARPA brings
to the DoD in a crisis is a stable of highly competent engineers and
scientists who _want_ to push the envelope and are smart enough to not let a
crisis go to waste. I'm on the demand side right now and it's been awesome to
see all these crack folks swarm down on my requirements.

If you think the DoD is behind the power curve, um, well, I have seen things
that don't officially exist show up, obviously already engineered, so fast you
can calculate from the shipping label how long it took someone to get off the
phone and get it onto a plane.

So, yes, there are absolutely the long game 5-year efforts (which are often
really sprouts of 20 or 30 year efforts). But it's a bit like YC: the alumni
network is amazing.

~~~
davidhowlett
What does the power curve mean in this context?

------
fastball
I'd like to write a comment but I generally prefer to have read the entire
article before I do... see you in a week or so.

Jokes aside, I agree with the author (I think) that DARPA has been a
surprisingly effective org in an age of frequent failures of other orgs with
similarly lofty goals.

DARPA is effectively what things like the SoftBank Vision Fund should've been
(wanted to be?). It would certainly be interesting to see what it would look
like to have a privately run clone of DARPA if you injected as much cash as
the Vision Fund did. Per the article, about $400M/y is spent on actual R&D by
DARPA, where as Vision Fund has injected that amount into single companies
many times over.

~~~
jariel
Vision Fund is there to throw huge amounts of money at a business in order to
ramp it up quickly and capture large, global markets.

Think Uber: disrupting the Taxi business (or just regulations ...). Once
something like Uber starts to work, a 'Vision Fund' takes this fledgling thing
and backs it with billions to conquer the world.

It's not about moon-shots, it's about market power and speed on a global
scale.

Like hyper-supercharged 'Round C or D' \- instead of doing an initial public
offering for cash, you take on massive cash from Vision.

Wether or not it will work is something else, but there's logic there.

~~~
fastball
Oh yeah, absolutely. I just kinda wish we had that level of private
interest/investment with similarly lofty goals. Not just "eat the world".

------
scotty79
My more top level view is that DARPA and such are just side channels to
sneakily funnel public money into research.

In theory free market should take care of the economy. It's really good at
optimizing manufacturing processes and getting stuff as cheaply as possible
into hands of as many people as possible.

However there's one thing essential to the economy that free market sucks at.
It's research. And that's not because companies are bad at doing research.
It's because companies are bad at funding research. Research is inherently
risky and no sane capitalist will invest in anything beyond tinker level
research because he will loose. And he's not supposed to loose.

In theory you could fund research overtly with country budget, but no one is
going to support that. Why spend money on eggheads playing with useless stuff
if people are hungry and streets are dirty?

What nobody opposes is giving more money to the military. And what anyone
can't oppose is military spending money in whichever way they please. So they
can spend it on research. Most will be wasted, very few will have actual
military potential. The rest can be graciously dumped into the economy for
companies to tinker, optimize making of, manufacture, market and sell.

IMHO military is the core of USA success (or even success of global
capitalism), not through might, but through ability to get plenty of public
money and ability to allocate it into things that would never get funded in
any other way.

~~~
hutzlibu
Well, the public does support money going to universities.

And I believe there are quite some people, who opposes more money going to the
military.

------
empath75
When you say that ‘DARPA’ “works”, what precisely do you mean by that, and how
do you measure success? It seems to me that it primarily works by throwing
large amounts of money at academics and then claiming credit for anything they
invent, whether or not it contributes to whatever DARPA is trying to
accomplish.

------
mncharity
If it were possible, a closer reading of history might provide additional
lessons. For instance, ARPA took a hit in the early '90s under Bush. Some
silver linings, but that's a transition which might illuminate process
tradeoffs.

There was (at least back then) an ongoing meta discussion about how to do
better. So it might be useful to explore not just the organizational designs
which were realized, but also the space of things considered. For instance,
before Bush hit, there was discussion of tiny "fix that!" grants. Like there's
one person who is an outlier in understanding how to do X, and their book just
isn't getting finished. So rather than society waiting years on diffusion and
reinvention (which is what ended up happening), it might be worth paying
someone to sit outside their office, and stand on their desk, and be a forcing
factor on making the book happen.

At least back then, with a failed attempt at a commerce ARPA clone, it was
thought important to have a clear metric to prioritize projects. "What's
better for DoD?", rather than the far less tractable "what's better for the
commercial economy?".

Perhaps I missed it skimming, but a major issue has been the death valley
between research and commercial impact. And attempts to address that making
things even worse (ie, researchers encouraged to think commercialization,
sacrifice impact by holding things close, and then commercialization generally
fails, so there's no offsetting benefit). And there's the unfortunate pipeline
from research to patent to unsuccessful startup to dominant company having yet
more anticompetitive ammo. I wonder if it might be fruitful to broaden focus
to the research pipeline rooted in ARPA? Because a successful clone would
presumably again face this difficulty. And there might be some other design
point that is less ARPA-like, but does impact better.

How well ARPA works comes and goes. It's not a stable equilibrium. So instead
of asking how to create a successful clone, perhaps one might ask how to
create something viable in the vicinity of success, and separately, how to
increase time spent less distant from success?

The difference between an old-school autonomous PM, rolodex and checkbook in
hand, showing up on your doorstep and saying "I've heard you interested in
doing X - what would you need?", and say NSF exploratory grants of "groups
with the following characteristics, may submit grants addressing the following
issues, with a deadline of mumble, and the following logistics"... is really
really big.

------
walczyk
Darpa is incredible because it's run by scientists and not managers. A lot of
small tech is also run like that, it's so refreshing.

------
godelmachine
There's an organization closely resembling DARPA but with exclusive focus on
computing. Wish more gets written on it →

[https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...](https://en.wikipedia.org/wiki/Intelligence_Advanced_Research_Projects_Activity)

~~~
nl
I've done some work around the edges of IARPA. It's not exclusivly computing
BTW.

It's pretty much DARPA with smaller budgets. Some pretty interesting projects
- the one about increasing human intelligence with magnets is pretty far out!

~~~
hutzlibu
"the one about increasing human intelligence with magnets is pretty far out"

Sounds very esoteric to me. Is there a link somewhere? I would like to know,
if there is substance to it, or that someone was able to talk quite good.

~~~
nl
It's actually both magnetic (transcranial magnetic stimulation) and electro-
magentic stimulation (transcranial direct current stimulation):
[https://www.theatlantic.com/magazine/archive/2014/09/prepare...](https://www.theatlantic.com/magazine/archive/2014/09/prepare-
to-be-shocked/375072/)

There's a reddit dedicated to people trying it on themselves:
[https://www.reddit.com/r/tDCS/](https://www.reddit.com/r/tDCS/)

I wouldn't say the results are exactly conclusive, but there did seem to be
some results showing that pain can be reduced in some circumstances.

And while "magnets" sounds like some kind of crystal therapy or something,
ignoring those aspects it's actually exactly the kind of way-out things that
*ARPA should be funding.

~~~
hutzlibu
Thanks for the links.

"but there did seem to be some results showing that pain can be reduced in
some circumstances"

Even though that sounds not more deep than placebo.

Apart from that I am also open to crystal therapie. You know, the world is
quite made up of quants and crystals can focus and polarize quants .. so if
someone has some claim with more substance than that and wants to do a
scientific research about it, why not. But so far I have seen only esoteric
"research".

------
xVedun
It seems that this would be a much better way of funding research in a general
sense, since that seems to be the general process of their problem solving.
The only thing is that to apply this to other fields, the barrier for getting
people that are equally motivated and intelligent about analyzing if the
research is 'going' somewhere is high

------
twarge
I would say the key difference between DARPA funding and other research
functing is that DARPA PMs have a crystal clear map of the leading edge in a
field that's ready for advancement and can target it perfectly with both money
and real competition.

------
ilaksh
I think it comes down to the fact that the military has a lot of money and the
projects have goals that are somewhat directed but also long term. Don't think
it's more complex than that really.

------
b20000
it works because there are no google bros

~~~
semi_good
This is correct. The privately run version of this setup Alphabets Project X
was wrecked by lousy hiring decisions. Who the heck recruits people like this?

“ As the Times reported, DeVaul, whose title is “director of rapid evaluation
and mad science,” told a young female job candidate during a 2013 interview
that he was in a polyamorous relationship. Later, when he saw the woman at
Burning Man while she was still waiting to hear back about the job, DeVaul
asked her if she would take off her shirt for a back massage.”

[https://slate.com/technology/2018/10/google-x-sexual-
harassm...](https://slate.com/technology/2018/10/google-x-sexual-harassment-
allegations-employment.html)

