
Software eats software development - goronbjorn
http://cdixon.org/2014/04/13/software-eats-software-development/
======
fauigerzigerk
_> In the pre-Internet era, tools like Hypercard and Visual Basic allowed
hundreds of millions of semi-technical people to become software developers.
Since then, there hasn’t been much work in these areas, but from what I’ve
seen that might change soon._

I doubt that there will be much progress on this in the foreseeable future.
Not for general purpose programming anyway. It's not true that there hasn't
been a lot of work in these areas since Visual Basic. Not a week goes by
without news about some tool that finally allows everyone to create UIs
easily, wire up some logic and one-click-deploy everything on the web and
mobile devices all at the same time.

But the bottleneck is somewhere else entirely. It's thinking in terms of
models. Formalizing and abstracting what we know intuitively about the world
and about the things we want to automate.

In my opinion, there have only been two inventions that really "democratized"
access to thinking in models: Spreadsheets and SQL. Both are from the 1970s.
There has been no progress since then, perhaps with the exception of some of
the visual things you can do with modern game engines. Maybe R and Matlab
deserve mentioning but they are for people who know exactly what they're
doing.

I think the next step has to be abstracting away some of the intelligence
required to analyse data and make all the tedious micro decisions about how to
transform it into something fit for the task. What we need to make progress is
some kind of AI enhanced version of Excel.

~~~
david927
* Not a week goes by without news about some tool that finally allows everyone to create UIs easily *

That's because they start at the easy part (the UI) and move down until they
hit that this is a hard problem. But there's reason for hope: there are
breakthroughs happening in this space (reactive like spreadsheets, declarative
like SQL). The results are really exciting.

~~~
perfunctory
> there are breakthroughs happening in this space

could you elaborate on this one. Any links?

~~~
david927
I don't have any public links but send me an email, it's in my profile.

~~~
Edmond
I am curious too...I'll send you an email.

I am the developer of HiveMind (crudzilla.com), it is a web app platform that
aims to tackle web app development in a way that moderately skilled developers
could do a lot with.

So I am definitely curious!

------
wpietri
When I was something like 13, I went to my first computer trade show, where on
vendor was touting tools that would let non-programmers write software for the
minicomputers of the day, eliminating the need for those expensive programmer
guys.

Ever since, I've seen people selling similar notions. Whether or not they
work, I expect we'll see a number of similar tools on the market. As things
like magnetic healing bracelets demonstrate, not actually working is no
barrier to selling a product.

~~~
stormbrew
Programming for non-programmers is an oxymoron to begin with. Even using Excel
is programming.

I've become convinced that one of the reasons this field has failed so
spectacularly in the past is because people are trying to achieve something
that is literally impossible.

In order to communicate with computers you have to become computer literate.
Or at least, it's far easier (and will remain so for a while yet) for a
flexible human to become computer literate than for an inflexible machine to
become human literate.

Finding better ways to make humans computer literate is and will for a long
time be a much better path to make computers more accessible, and it's largely
what's worked to get us to the point we're at now.

~~~
ebiester
Let's operationalize it then:

We need tools that bring the learning curve down and allow people to develop
their own basic tools to make their lives easier, while storing the data on a
central location.

Amazing things have been done in Excel, but have you ever seen a department
have locking issues? "Can you close the spreadsheet so I can get in?"

Yeah, we need multiuser excel, or perhaps a google docs equivalent that can be
hosted internally.

~~~
Perdition
>We need tools that bring the learning curve down and allow people to develop
their own basic tools to make their lives easier

This won't achieve much because the issue is that most people can't think
about the problems they face in a structured way.

Modern programming isn't hard, all you need to learn is a handful of control
structures and how to do I/O, and there are a multitude of tools to take away
the tedious bits.

But if you can't describe the problem you are trying to solve in a structured
way than no non-AI tool will help.

P.S. I remember being considered a Wizard in a non-technical university course
because I looked up some Excel tutorials and built a spreadsheet with some
basic calculations and graphing. This wasn't technically hard, but most of my
classmates lacked the ability to comprehend the relationships between
different cells and tables in Excel.

~~~
ericHosick
Currently, programming requires critical thinking and coding.

Enabling people to become critical thinkers is a much bigger problem than just
programming: programming will always require critical thinking.

Making programming a computer easier is a difficult problem to solve but very
solvable as long as we assume that programmers are critical thinkers.

~~~
Perdition
See I don't think coding can get any easier for the level of problem we are
talking about here. The kind of small problems we seem to be talking about are
the kind of thing that is within the realm of "scripting". You don't need a
deep understanding of Computer Science or advanced data structures to write
good enough code for those problems.

"Coding" will never go away before AI because at some point you need to
formally express the design you have thought up. It doesn't really matter if
you do that with arcane syntax, symbols, or "natural" language.

~~~
ericHosick
> "Coding" will never go away before AI because at some point you need to
> formally express the design you have thought up. It doesn't really matter if
> you do that with arcane syntax, symbols, or "natural" language.

It sounds like you are saying, and I'm not disagreeing with you, that the act
of speaking required coding.

That is, speaking is taking input from the senses and memory, converting that
input into a natural language (coding/encoding) through thought and saying the
final thought out loud.

How does this help in understanding the differences between typing out syntax,
manipulating symbols, expression using natural language or AI using it's own
coding approaches?

~~~
Perdition
No, I am saying that coding will always require some kind of formal
expression, and that the difficulty of the grammar used is not the major
barrier (as these issues can be dealt with by compilers and IDEs).

Attempts to create "natural" language programming haven't worked because
actual natural languages are too informal and imprecise for simple algorithmic
comprehension.

A system that can comprehend natural language and turn imprecise
specifications into a program should be an AI.

------
pcurve
And enterprise software development has been going in the opposite direction.
I work for a $100 billion company. Some days, I want to stab myself and throw
myself out the window.

~~~
chaz
In these situations, I believe there are two types people:

A: "Wow, things are really messed up. I'm out of here." B: "Wow, things are
really messed up. There's opportunity _everywhere_."

Either can be right, depending on the person and situation.

~~~
beat
I'm a B, for sure. I'm trying to create new tools to deal with the "things are
really messed up" problem... in part because I can create a working business
and get myself out of the enterprise, and in part because I genuinely want to
make this industry better. What I like about what I'm building is that it
changes the human/computer ratio in debugging and problem-solving for complex
systems.

Have computers do what computers do well, so people can do what people do
well. That's the core of my approach to building software. A big part of the
reason enterprise development sucks is because debugging is so _manual_.
Automate parts vulnerable to automation, and engineers won't just be more
efficient - they'll be happier.

------
AdrianRossouw
All those tools you talk about still need to be maintained.

Anything built for "non-programmers" as you say, is also many orders of
magnitude more complex, and at some point that complexity will grow to a point
where it is cheaper to replace the whole system entirely rather than upgrade.

API's and stuff are easier on some level, but distributed systems are also
much more complex. Upstream API's change, or get EOL'd or change license.

actually, instead of listing all the reasons : every point you made is just
going to end up with more software being developed. it all feeds into itself.
endlessly.

------
bobbygoodlatte
This also means that design is playing a proportionally larger role in
startups.

A lot of the engineering work needed to build the next Instagram/WhatsApp/etc
has become commoditized. Design is what's left.

~~~
shortstuffsushi
Wow. That's actually something that hadn't really struck me until reading this
comment. While there is development to be done in ever project still, the
upfront amount is growing smaller and smaller with the (P|I|S)aaS techs
available to companies. A lot of it does really come down to making the front
end of a product.

------
beat
Here's why this is, at least on a certain level, wrong...

 _Software complexity expands to exceed available tools._

This is a Big Truth. No matter how fast hardware gets, we'll beat on it until
it's burning up. Give us better tools, and we'll just write more and more
complex software until the tools break down under the complexity. Give us
better processes, and we'll make bigger projects.

Better tools don't make it any easier to write software, never have, never
will. They just change the kind of software we can write. You don't buy a
Porsche to drive 55 more effectively, and you don't use better software
development tools in order to work less.

~~~
mattgreenrocks
Why aren't developers responsible for managing the complexity of their
project?

~~~
syntern
They do - if they have ownership too. But project managers won't like to give
up the ownership, they just push for more features in less time. There goes
your hope for responsible complexity management...

------
venomsnake
You know every time we create some innovation that unloads the burden from
programmers we invent some other crap to waste their resources on.

Circa 2010 enterprise java managed to take all benefits from freeing memory
management and waste them on absurd abstractions and xml permutations (you
throw some settings in xml and see what happens)

The universe is conspiring against us shipping real code.

~~~
narag
LOL, so true! I'd say the opposite is true: software isn't eating programming
in any significant way. As much, it's eating the overhead you describe.

Breakthroughs? Show me!

------
tim333
Competitive market forces will keep programmers busy for quite a while. As
soon as it becomes easy to do things like make basic online stores with
Shopify, stores will want more fancy functionality to differentiate themselves
from their competitors.

------
netcan
Economically these are not different from technology in general. We can do
more with less. There are two side to that statement.

We can do with less. The first two points are especially relevant to this side
of things. The most "talented" can be extremely productive. What if Google
search & Gmail could be built and run by 14 talented individuals? Duckduckgo
seems(ed) like an experiment in this direction. Doing with less is not
necessarily painful or revolutionary though. Often, it's smooth. The answer to
a lot of "how did people do ___ before computers" questions is "secretaries."

We can do more. That has been the overwhelming result of technological change
up to now. Better tools -> More stuff gets made. Most of the resources freed
by "doing with less" go into doing more. Agricultural machinery frees peasants
to work in factories. Industrial robots free laborers to work as social
marketing content creators. Progress.

------
nmrm
It's not hard to imagine rote programming becoming a universal skill among
high school graduates. I could even see this happening within our life-times;
it's really only a matter of societal will and available teaching staff. The
basics are... basic.

If this happens, then APIs and libraries that make development easy and
intuitive for end-users could easily become as in-demand as well-designed
websites and mobile apps are today.

~~~
JackMorgan
I can see general programming being added to a school curriculum as most
people think it's a guarantee at a good career (for some reason).

I don't mind this mindset, and would enjoy teaching such a course.

I look at it this way though, we theoretically teach every single American
student how to write, has that caused an increase in high quality literature
being produced? Theoretically, every high schooler can read, I'm curious what
percentage of adults actually read literature for fun?

Most long form writing (which I'd consider to be analogous in complexity to
large scale programming) is not very common. The average adult's writing is
quick and "dirty". I'd expect to see an increase in programming similar in
size, scope, and usefulness to email, text, lists, and meeting notes. It would
be personal, private, and not meant to last. Tents, not marble palaces.

I think a much more valuable tool would be to teach everyone git (or something
easier but still with good branching). For quick and dirty, excel is fine, and
some improvement on it likely will come in our lifetimes. What I think is far
more valuable is a way to version all text a person produces, with branching
and history.

~~~
collyw
The problem with quick and dirty excel, is that the temporary gaps that it
fills become permanent solutions. The a couple of years down the line, too
many parts of the system are hinged around an unreliable error prone excel
sheet, when something better should have been written.

~~~
JackMorgan
Well that's just it, what is "better"? Currently, the software to replace a
typical excel spreadsheet is prohibitively expensive. (As someone who
currently makes a "spreadsheet replacer", I've heard our BA regularly comment
that it takes us tens of thousands of dollars to add in features he was able
to add to his spreadsheet in a few minutes.)

I recognize that we are building a cathedral meant to last, and he was
building a tent that was error-prone and easy to break. The thing that gets me
is how fast he was able to turn data into the answers he needed. Why don't my
tools let me put something that fast together?

I've become lately interested in the idea of a "better excel" that could solve
the issues excel has: easy to "forget" a new row or cell, hard to version,
hard to share, stuck behind a complex gui, hard to add bigger data, etc. I
think such a tool could greatly increase the "general programming" of the
world far better than just teaching high schoolers to program. Most people who
would program either do so for someone else (trying to automate something) or
for work because they need to get an answer out of some data. I think excel IS
a programming language, it just has a gui instead of text. I think if that was
improved upon, it would meet a big need.

------
31reasons
Software eats software development is nothing new. Everyone who writes
libraries is doing just that for past 30+ years. In fact good software
developers take pride in writing software that require less code. Its in the
inherent nature of software development itself. Come to think of it, software
field could be the most egalitarian of all. You won't see lawyers lobbying for
less complicated laws!

------
CmonDev
"Developers have steadily marched upwards from Assembly to C to Java to,
today, scripting languages like Ruby and Python."

Downwards in terms of language design though, you can add JavaScript to the
end of that sequence to make it even more obvious. Otherwise we would've been
using a some sort of sane syntax version of Haskell by now. If software is
"eating" 'whatever' then computers should be doing more for you not less,
hence even more pre-runtime checking, not less.

"... allowed hundreds of millions of semi-technical people to become software
developers. ... these tools act as a force multiplier for the software
industry."

Multiplying the code mess professional developers will then have to maintain?
Example: converting Excel and Access spaghetti into sane programming models in
finance.

~~~
drill_sarge
Who moved to Java? Write once - debug everywhere.

~~~
tluyben2
Guess you don't code in Java. As that's bullshit. But maybe you refer to GUI
!= HTML applications. And even there it's not true.

------
AKora
It's interesting that there is no mention of the so called "citizen
developer". According to some Gartner reports very soon a fairly large amount
(approx. 25%) of all business applications will be "written" by so called
"citizen developers". These are mostly business people with minimal or no
programming knowledge but are able to use software tools to create other
"software tools". A good, VERY BASIC example is building an Excel "app" with
some VB scripts...

------
yason
Programming for non-programming works best when they don't know they're
programming. It's easy if you don't know you're doing it.

------
mbesto
> _General purpose tools for non-programmers: Since then, there hasn’t been
> much work in these areas, but from what I’ve seen that might change soon._

Really? Loads of examples. One just off the top of my head is Scratch:
[http://scratch.mit.edu/](http://scratch.mit.edu/)

~~~
blahedo
I love Scratch and teach it in my gen ed class, but I'm not sure I'd quite
call it a "general purpose tool" in the same way that Hypercard was. I could
certainly see it evolving (or rather, forking) into such a thing, but as it
stands it's a little too kid-education-animation-game-oriented.

------
rubiquity
> _Developers have steadily marched upwards from Assembly to C to Java to,
> today, scripting languages like Ruby and Python._

Since when does being a dynamic language and not having to have a main()
function make you a "scripting language?" I didn't know people still said
those words.

~~~
giblaz
>scripting language

It's just a categorization. Not an insult.

~~~
marcosdumay
It's a categorization with no meaningful difference. There's no honest reason
to create something like that.

------
alphydan
For those in machine learning and programming languages, what would be the
major hurdles in achieving voice programming (or at least scripting). A way to
interact with a computer so that one could ask things like:

"Computer, search google's rss news for arduino articles. Sort them by lenght,
in the top 100 results search for the world circuit. Copy paste the paragraphs
containing those results into a text file. Send it to Jane".

Is it really that far? Wouldn't such a program enable some kind of
_programming for non-coders_ (of course assuming one would need to learn some
rules, and have a clear idea of what one was looking for).

~~~
Mangalor
The problem is it's already hard to be very precise about what you're saying
when talking to a computer just with text, so we'd need functions specific for
each of those voice requests, more reliable voice recognition, and also some
AI intelligence to understand the English you're using, so even that simple
request would be difficult unless we pre-scripted the functionality for that
whole phrase.

It's not impossible, the technology just isn't there yet.

------
fredgrott
Question, does this follow the same trend as other Relationships..for example
Mathematics can be said to be the 'software' of high-level physics..do we see
the same trends in that there is a high-level in Mathematics where new
theories are being created but a lower level where a large amount of people
use tools to apply these new theories?

If so, than are we facing the same exact problem just in a different field?

My take on it, is that we never fully mastered teaching how to come up with
abstractions and new models..tangential proof is the large number of religions
created to come up with new models of explaining how the =universe works.

~~~
nmrm
> do we see the same trends in that there is a high-level in Mathematics where
> new theories are being created but a lower level where a large amount of
> people use tools to apply these new theories?

Definitely.

If you allow more longer-running examples, this is exactly the situation of
calculus. The Calc I-III that many use regularly is the application; Real
Analysis is the theory. Ditto for any "applied" sort of math you can think of
(stats, crypto, machine learning, etc. -- all basically just use theorems
which are the "abstract interfaces" to pure mathematics.)

> we never fully mastered teaching how to come up with abstractions and new
> models

And we never will, as long as the human race continues progressing :-)

------
just2n
> Software is eating the world, and doing so using smaller and smaller teams.
> WhatsApp was able to disrupt the global SMS industry with only a few dozen
> engineers. Small teams can have a big impact because software development
> (and deployment) has improved dramatically over the past decade.

While this is true, I'm coming to realize more and more that a _lot_ of would-
be highly needed disruption is just not feasible because laws and monopolies
are standing in the way, much of it due to lobbying and sometimes patents.
It's sad.

------
DannoHung
Halide makes me think we can give a bunch of that power that Moore's law gave
us back to actual problem domains without making programmers any less
effective.

------
illumen
Pretty much his entire list is wrong.

"Deploying a commercial website ten years ago required significant upfront
capital"

Actually... you could get a very large website up 10 years ago on almost
nothing. $30/month dedicated servers were available that could do 10,000
connections at once. Which means you could serve millions of people cheaply.
You could also get lots of virtual shared hosts for $5/month or less.

Heck, even 15 years ago you could do that. Maybe not everyone realised it till
later, but it's not a new thing at all.

"Startups created simple APIs that abstract away complex back ends. Examples:
Stripe (payments), Twilio (communications), Firebase (databases), Sift Science
(fraud)."

There were plenty of services around on the internet 15 years ago with APIs
too. Including payments, communications, databases, and fraud. I know because
I developed some of them.

"Open Source. Open source dominates every level of the software stack,
including operating systems (Linux), databases (MySql), web servers (Apache),
and programming languages (Python, Ruby). These are not only free but
generally also far higher quality than their commercial counterparts."

Guess what? Open source was around 10 years ago.

"Programming languages. Developers have steadily marched upwards from Assembly
to C to Java to, today, scripting languages like Ruby and Python. Moore’s Law
gave us excess computing resources. We spent it making developers more
effective."

Ruby and Python were both around ten years ago. The same as perl, php, and
haskell amongst others.

"Special-purpose tools for non-programmers. These tools let non-programmers
create software in certain pre-defined categories, thereby lowering costs and
reducing the demand for developers. Examples: Shopify (e-commerce), WordPress
(blogging), and Weebly (small business websites)."

um... blogging, e-commerce, and lots of other non-programmer tools were around
in the 90s.

"General-purpose tools for non-programmers. In the pre-Internet era, tools
like Hypercard and Visual Basic allowed hundreds of millions of semi-technical
people to become software developers. Since then, there hasn’t been much work
in these areas, but from what I’ve seen that might change soon. By allowing
more people to program, these tools act as a force multiplier for the software
industry."

Have you heard of the internet? Seen the reams of cut and paste code out there
from semi-technical people? Whole industries are centered around letting less
skilled people make software. This hasn't decreased, but increased. There's a
whole "everyone should code" movement hitting us.

Is this the quality of article we want on this website? Please delete my
account!? I'm done.

~~~
1stop
Is this the quality of comment on this website... please delete your account!

What were the "plenty of API services" 15 years ago. I don't remember that
many, you are talking pre google days... Like flashing "Under construction"
websites... If there were APIs i think they were probably an accident ;)

Ruby and Python were both around ten years ago, and No body cared.. Python was
primarily used as a CGI layer, and then later in Zope... Ruby just wasn't used
(as far as I recall). After Rails, and Django, people started paying
attention, and then google gobbling up the python community worldwide changed
that.

What were the Blogging/E-commerce tools around in the 90s? Do they compare at
all with the ease of use/scale of those quoted?

~~~
illumen
The e-commerce shop sold to yahoo in 1998 by pg. The guy who made this
website. [0] Ebay was also doing _lots_ of business in the 90s.

Paypal was founded in 1998, and it wasn't the first e-commerce company by far.
[1]

Perl, php, TCL via AOL and yes python were definitely used for much of the
web. Google founded in 1998 used a little old language called... python. [2]

mod_python was released in 2000, which came after the non-cgi version of
python the Netscape Enterprise server... which also used JavaScript as a
server language. [3] However, _lots_ of people were using perl for web
services.

SOAP designed in 1998 was one of the libraries or protocols where people
published APIs, along with Corba, and xmlrpc amongst others. [4]

[0]
[http://www.paulgraham.com/yahoo.html](http://www.paulgraham.com/yahoo.html)

[1]
[https://en.wikipedia.org/wiki/PayPal](https://en.wikipedia.org/wiki/PayPal)

[2]
[https://en.wikipedia.org/wiki/Google](https://en.wikipedia.org/wiki/Google)

[3]
[https://en.wikipedia.org/wiki/Mod_python](https://en.wikipedia.org/wiki/Mod_python)

[4] [https://en.wikipedia.org/wiki/SOAP](https://en.wikipedia.org/wiki/SOAP)

~~~
1stop
... You still didn't mention a single API.

And yes, python was in complete disuse that only obscure academics used it
(Which is what google was founded by). I mentioned in my comment it only
gained traction after google and django before then it was niche Zope and CGI.
All your citations don't actually refute it, they in fact strengthen it...

So show me an API that existed in 1998... all you showed me was that SOAP was
'designed' in 1998...

Are you really citing "Netscape Enterprise Server" as proof python was in wide
usage?

Paul Graham sold a business to yahoo, wasn't it in lisp? Did it have an API?
does it actually prove any of the points you made, or is it just a statement
about "There was business on the internet" which I never refuted.

You were really scrounging to come up with pretty much nothing, so thanks for
further illustrating my point :)

------
applecore
_> Since the pre-Internet era, there hasn’t been much work in general-purpose
tools for non-programmers, but from what I’ve seen that might change soon._

What startups might he be referring to?

------
nahname
Developers keep pushing the envelope to make tools easier to use and people
keep coming up with increasingly complex requirements.

Neither side is ever content.

~~~
Mangalor
This. This is the reason why I never say "We can build X". We can never know
because the requirements always get pushed an order of magnitude higher
whenever we can actually build X.

------
michaelochurch
"Software is eating software" is true, but not good. It's not good at all.

Take a look at the manner in which so many companies use project management
software like Jira: to create a Big Brother system. Developer's velocity drops
below 15 "story points" per week? There's your bad guy! Some companies use
time-tracking software (designed to help an individual manage her own time
better, and certainly not to abuse workers) to take micromanagement to a new
level. Then there's LinkedIn, which may have been started with good intent but
makes it astronomically more difficult for people to reinvent themselves
(giving even more power to corporate management, which is thus empowered to
fuck with peoples' careers long after they leave).

The real bad guys have been using us (programmers) for decades to wage their
war on the workers. That's most of what they want us to do. Our effective
purpose is not build flying cars or cure cancer, but to vaporize jobs for the
poor and deliver the proceeds, efficiently, to the rich. Since we are also
workers (we're the upper-working class) it's not surprising that we'd see our
own weapons used on us.

~~~
artmageddon
> Then there's LinkedIn, which may have been started with good intent but
> makes it astronomically more difficult for people to reinvent themselves
> (giving even more power to corporate management, which is thus empowered to
> fuck with peoples' careers long after they leave

Just curious, how do you mean?

