

Plan to write big software - and you have already lost - bensummers
http://olabini.com/blog/2009/10/plan-to-write-big-software-and-you-have-already-lost/

======
adatta02
The central thesis of the post “Optimizing for your software project becoming
big is the same as optimizing a car to hit a rock wall - you are optimizing
for failure” is garbage.

I can't imagine any sane engineer not realizing that writing say an operating
system is on a different level of magnitude than writing some HTML peppered in
with a little Javascript.

Edit:

"This is the reason I like agile. It emphasizes small, working pieces all the
time. If you work with code this way, you can’t really become big. Instead,
your project will be forced to be modularized and divided into smaller, more
logical components that are highly cohesive and decoupled from each other."

Using Agile really doesn't guarantee you any of this. You're still very free
to abandon separation of concerns and end up with a tightly coupled mess.

~~~
Tamerlin
"Using Agile really doesn't guarantee you any of this. You're still very free
to abandon separation of concerns and end up with a tightly coupled mess."

This is what I find usually happens with agile development. The software tends
to start out small, but because it's not well thought out and because it tends
to be rushed, even the small stuff becomes spaghetti startlingly quickly -- as
happened on a current project, where after less than three months, we managed
to cram THREE separate object models into a UI with a whopping TWO screens.
(Imagine my frustration when what was six lines of code when I last worked on
it now had over 500.)

~~~
joevandyk
so you kinda skipped the code review and refactoring parts, right?

~~~
jamesbritt
I'm guessing you were modded down for implied snark, but it's well worth
asking how a small project becomes a big project with no noticing, or noticing
but not taking appropriate action.

It happens to the best of us, the feature creep, the must-haves, the rush to
ship. So, does agile actually fail to address this, or is this something
subtle that manages to evade detection despite better practices and intents?

~~~
run4yourlives
_but it's well worth asking how a small project becomes a big project with no
noticing, or noticing but not taking appropriate action._

Simple, nobody pays you to notice how much of a problem the software that
works perfectly has become. They pay you to improve feature x by a very small
amount, or to add feature y.

So, sure Agile works fine, in the same world that most every other methodology
works fine.

It's not the methodology that makes good software, it's the people behind it.

------
colomon
This article embodies what I hate about the Agile / TDD crowd. They have a
bunch of good ideas that appear to be helpful in their problem domain(s). But
then they make these grand pronouncements as if their problem domain
represented all of programming.

It's particularly bad here because in the process of telling every programmer
with a different problem domain that they don't know what they're doing, he
appears to concede that dynamic languages are not good for handling big
projects...

~~~
mediaman
Do you disagree with those who say dynamic languages are bad for big projects,
which the author appears to concede by arguing big projects are bad in the
first place?

~~~
enjo
Its not an argument I've ever really understood.

If we're applying the argument to Perl, then sure.. I get it. With Perl, it's
more a question of syntax. By supporting so many ways to accomplish the same
thing, it can be very unwieldy when you have many programmers all exercising
all of those options.

It's not a question of dynamic in that case. It's a question of a (in my
opinion) fundamental shortcoming in the language.

However, languages like Python or Ruby both seem pretty well suited for large
scale development. We've seen large scale projects written in both, and it
works just fine.

So no, I don't agree with the fundamental premise that dynamic languages are
bad for 'big' projects. If by 'big' we mean memory/processor intensive... well
lets have THAT debate instead.

------
raganwald
Well, cars are engineered to protect the occupants from trauma if they hit a
brick wall at moderate speeds, but that isn't the same thing as optimizing
them for hitting walls, so the analogy doesn't do anything for me.

And likewise some software is big, period, regardless of the language used.
The easiest way to measure the size of a piece of software in advance is to
imagine the size of the test suite. I have a poorly thought-out idea about the
minimal complexity of a software program being the complexity of the automated
test suite that validates the program...

Anyways, some software is bigger than other software, but again that doesn't
necessarily mean that given a large test suite to be satisfied and
anticipating a larger program or collection of programs you necessarily need
to discard certain tools that work well for smaller programs.

Steve Yegge made this point with a much more entertaining analogy: He talked
about pushing dirt, which illustrated the fallacy with _some_ tools that are
espoused as being appropriate for large projects: namely that they are really
good at solving the problems introduced by the tools themselves.

[http://steve-yegge.blogspot.com/2007/12/codes-worst-
enemy.ht...](http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html)

My takeway from Steve's article was that if you avoided tools that introduce
certain problems, you don't need other tools to solve those problems.

------
spazmaster
OK, I like his notion that 'becoming big is failure'. But... what about
websites like Amazon.com? That’s BIG with capital letters.

How did they lose? They operate successfully due to marginal costs on all
fronts including their website.

I would like to hear some ideas out here on projects like Amazon.com.

~~~
zaphar
Just because the product is big doesn't mean its a monolithic codebase. Amazon
is powered by a lot of different pieces. Decoupling let's you keep the pieces
small and yet grow and scale to huge sizes.

~~~
Tamerlin
It's also a maintenance nightmare. A LOT of that code is terrible, there's a
lot of tight coupling, and a heavy reliance on developers with pagers to keep
the mess operating.

There's a reason that they have such insane turnover so few people who stick
around after they collect their signing bonuses...

~~~
spazmaster
thanks, that's some interesting information. that does strengthen the OP's
case.

Loose coupling should be possible but a rarely seen feat I reckon for websites
of this scale.

I wonder how "loosely" coupled and well architected the 2.0 sites are like
digg and facebook?

~~~
aaronblohowiak
You can read about facebook's architecture at highscalability and other sites
of that ilk.

------
sanj
_Optimizing for your software project becoming big is the same as optimizing a
car to hit a rock wall - you are optimizing for failure_

No you're not. You're optimizing a car that can survive running into a rock
wall. If that's what the car needs to do, you're doing exactly the right
thing. If, instead, it needs to get people places, or be fuel efficient, or
drive over 120mph, then you're probably doing the wrong thing.

~~~
arnorhs
Yeah.. true, but indeed the best scenario is not to hit the wall... Drive
slower, more energy efficient and take the best route and try not to fall
asleep while you're at it

------
megamark16
I like the old mantra "Fast, good, and cheap. Pick two." If you build one big
project all at once, it may not do anything you want until it does everything
you want. Whereas if you start small and keep your features separate and
cooperative you are more likely to get some of what you want right away, and
over time get everything you want working together.

~~~
evgen
The danger here is that in optimizing for fast and cheap you may be
prematurely optimizing yourself away from being able to do "good" without a
complete re-write. Fast and cheap has a seductive nature of getting good
feedback on broad concepts, but it also tends to lead to very shallow
implementation of these concepts. Like a lot of evolutionary algorithms, it is
easy to get caught in a local maxima and not realize that you have only
reached the top of a foothill while a competitor that can see where the
mountain really is will have made less progress but will end up miles ahead if
you discover that you climbed the wrong slope...

~~~
wgj
I agree that whatever you optimize for early, you are probably stuck with
later.

There are more than three vectors. "Good" could mean reliable, secure, highly
functional, or many other things. Picking any three vectors and analyzing the
tradeoffs helps to figure out how to apply resources.

~~~
bombs
Scope, money and time are what I know them as, with quality as an optional
fourth (that could be included as part of scope).

------
idlewords
Having an intuition for the scope of a task, and for the likelihood of being
able to complete it using existing components, is part of being a good
programmer. His post would seem to imply that the only acceptable way to write
big software is to stumble into it by accident.

~~~
Nelson69
It's not like there isn't close to a 3/4 million lines of code in JRuby..
Nothing quite like practicing what you're preaching. (The naive code count is
something like 250,000 lines of java, 560,000 lines of ruby, 14,000 lines of
yaml(?!) and about 2400 lines of xml. About have the ruby is in tests. I just
piped 1.4rc1 through "wc -l")

It's almost like he's trying to prove the converse of his thesis. Not to be
mean here, but he's a fairly junior level developer, he's a younger guy who
done some things but not done that much. I've worked on some very large,
highly profitable projects and we always knew use-cases and requirements long
before we got in to the project. There are tools that allow not-great
engineers to produce good stuff. There are also great engineers which are
capable of dealing with the astronomic complexity of a large project and it
turns out that people pay a lot more money for software that solves problems
that they can't otherwise solve. If there is a simple solution that satisfies
the requirements and beats the competition, then more power to you.
Unfortunately, many complex problems have complex solutions.

------
LiveTheDream
"Big" or "small" software is relative to the problem at hand. The essentially
author posits that "big == tightly coupled", which doesn't have to be true
whether the team uses agile or not.

I think the real thing that the author denounces is blanket statements like
"this project will be big so we must use Java" without really thinking about
the project's potential structure or the best language.

------
nadam
I don't agree with the post because there are projects when you know in
advance that the project will be big. You can probably avoid to make it HUGE,
but you cannot keep it small. Trivial examples:

\- Writing a web browser \- Writing a professional image manipulation software
like Photoshop \- Writing a professional 3D modeler like 3D Studio \- Writing
an IDE like Eclipse Idea or NetBeans

and so on...

~~~
arnorhs
Yes there are projects that you know beforehand that will be huge, but that
doesn't mean that they should be made or that that is the best way to go about
them.

For instance: Browser - if you're going to start to build a web browser, it
doesn't have to be huge... unless you're going to have your own rendering
engine, your own JS engine etc... You'd probably just be addressing some set
of features that were missing/misdone in other browsers Writing photoshop -
what, are you going to remake all the features of photoshop? And do you think
your software will be better? That's a fail. You'd probably be starting by
doing some image manipulation program that does a limited set of features that
photoshop can't do etc...

I get your point and I totally agree that if you _WERE_ going to remake
photoshop, it's totally going to be huge... but that doesn't mean that it's
the best path to go down.

~~~
arnorhs
and why do people disagree with me ? downvotes but no argument?

~~~
nradov
You're making a marketing argument, not a technical one. Trying to build a new
Photoshop competitor from scratch is probably a losing proposition for a
variety of reasons, regardless of how you run the project or structure the
code.

However, there are a large class of problems out there which do require a huge
chunk of software if you want to address them in any useful way. No matter
what you do to reduce coupling between modules there is a certain level of
irreducible complexity. For those cases you need to do a lot of up-front
design work or you're going to end up with a huge mess. And no, refactoring is
not the solution here.

~~~
arnorhs
You're right, I was also blinded by my own ego when I wrote that comment.

But all that huge chunk of software will definitely be built using smaller
projects - and the more independent the better.

------
edw519
_This idea came to me as a tweet, but was way too long for a tweet._

No it wasn't...

"A lot of small things tied together intelligently is often better than one
big thing."

~~~
owinebarger
That idea has a name, and it is "Unix". qmail remains my favorite example of
this philosophy put into practice.

~~~
notaddicted
Would you recommend the qmail source as good learning material?

<http://news.ycombinator.com/item?id=879101>

------
messel
This a concise yet necessary shared observation on software design. The
problem invariably becomes feature creep or inheriting someone elses working
code.

Not every problem can be optimally fast, and also optimally sized for
simplicity. As an alg coder of quite a few years of course I prefer and love
simple easy to read, remember, and modify code. But then someone comes along
and says, we need this fairly quick code, parallelized, ported to these other
platforms, and faster than real time. Dynamically size and share memory as
needed to scale to any number of available nodes, and have it driven by a GUI,
but make those buttons overlay an elliptical earth model…..

As you can imagine, writing simple (and reusable) code is a desirable thing
for programmers. Unfortunately we don’t dictate the terms and requirements,
real physical systems do. Sometimes deadlines trump both of the above, and
getting something to work, means messy throw away code. I guess we could all
agree never to write software like that, good luck getting consensus (herding
cats made easy).

------
maddalab
<i>Dynamically typed languages are fine for smaller programs and simple web
applications, but if you’re building something big, something that will be
several millions of lines of code, you really need all the tools you can only
get from a statically typed language</i>

From my experience I am only speak to the benefits provided by a statically
typed languages as you try modularize the components. That isn't to say that a
dynamically typed languages do not provide an alternative or similar benefit,
I just do not have large project experience with them.

Not certain how he draws the causation from using agile to the resulting
software being modularized and cohesive. It just is a case of correlation on
the projects he has worked on.

Both the arguments for dynamic and static languages as presented here are
vacuous.

------
tybris
> Optimizing for your software project becoming big is the same as optimizing
> a car to hit a rock wall - you are optimizing for failure

Cars do get optimized for hitting rock walls (i.e. what happens in case of a
crash).

~~~
joevandyk
No, they are not optimized for hitting rock walls.

They are optimized for keeping costs low, getting reasonable performance and
gas mileage, and meeting government safety standards.

A car that is optimized for hitting a rock wall would get about 1 mile per
gallon and have a top speed of 1 mile per hour.

------
reefboy
yeah. and if you write small, anybody else can likely do it, too. so, you
write big and complex, you have barrier to entry. but it's hard. you write
small and easy, you dont have barrier to entry. which is hard. welcome to
planet earth.

------
didroe
I'm so glad car manufacturers don't bother with airbags, or ABS, or traction
control, or side impact bars, or...

~~~
sofal
He said "optimizing a car to hit a rock wall." Believe me, if your car was
optimized for hitting rock walls, it would look very different than it does
now.

------
arnorhs
Upvoted. You don't even have to read the article, the title says it all, and
it's true.

