Oh neat, maybe he can explain why the new Bay Bridge was practically a decade over schedule and billions over budget.
Or the Burj Khalifa was about 2 years overdue?
Or the California High Speed rail is projected to be twice the budget they thought, and they've barely broken ground on it?
So the idea that "Real Engineers" building bridges and railroads and skyscrapers get it right in plans and schedule is total bullshit. We've seen larger software projects, with more people working on them than these examples, hit their deadlines (Windows comes to mind -- except Vista).
And by the way, those "Engineering" disciplines have only been around for, what, hundreds, if not thousands of years. What they do for a living is mostly explored territory. Software, at least at a scale to be considered engineering at all, has been around for about 3-4 decades... maybe. And we're still hitting up against major unknowns all the time at every level of the practice, big and small.
I was with you until you got to this line. Let's be brutally honest: the limiting factor for most of today's software projects (i.e. CRUD websites and iPhone apps) is not technology. It's developer competency, experience and self-discipline. I'd go so far to say that most working software developers don't even understand the fundamentals of the technologies they're using -- it's why you see people replacing their relational databases for key-value stores, then trying to re-invent database join algorithms from scratch in the application layer. Or why every new generation of "engineers" re-discovers asynchronous programming in a new language (then promptly writes a framework in Blub...because Blub is so much better than last year's Blub.)
Truthfully, it's pretty damned hard to find gigs working on the frontier of technology. Most projects are predictable, tedious slogs to fit together well-understood technologies without shooting yourself in the foot. And yes, requirements shift and clients get fussy, but that's true of any engineering discipline. You learn to pad the schedule to compensate for the shifts. It's part of being a professional.
As far as I can see, commercial software isn't limited by "major unknowns", so much as an industry-wide unwillingness to learn from experience. But hey...that tends to happen in industries when you're considered "old" at 30....
I'm sorry, I needed to get in an afternoon run and actually deleted a paragraph that explained this a little more that I couldn't get just right before posting.
I completely agree with you. There aren't a lot of technical unknowns for most things people are trying to do.
Thing is, I wasn't actually talking about technical challenges with that point, rather, the requirements that are given at the beginning of a project. The requirements for the new Bay Bridge, for example, are pretty darn clear: make a new bridge from Oakland to Yerba Buena Island.
The requirements for making G+ attract Facebook users... errr... uhh...
Or how about the requirements for the next Call of Duty to be fun...
A great many of today's software projects -- especially those discussed here on HN -- aren't straight up engineering problems, they're more design and UX problems. Even for something like the software for the Joint Strike Fighter, you're more into UX than hard-hittin' avionics for a lot of it these days.
And I think that's what makes it hard to predict software development schedules. If we were all building ATM software again and again, we'd be knocking it out of the park. A competitor to Snapchat that is able to sway users over... that might be harder to define up front.
Of course out here in the real world, an "alter table" statement takes no time at all even on a large table, and can be done while the system is running.
Now I am not saying that MongoDB has no uses at all; but I am saying that the people around it don't really have much of a clue about databases in general and aren't in any sort of a position to be choosing technologies.
Mongo is a way of storing 'stuff'. You don't have to define the form that the stuff takes before you shove it in the database. This makes things easier by lowering the knowledge barrier to getting started, which is a double-edged sword.
A bog ordinary 12 story building will probably come in close to schedule and budget because it's a well-understood problem domain and there is a large pool of existing examples to draw data from.
But the Burj Khalifa required solving some genuinely novel problems, of the sort which only turn up at that scale. When a project involves substantial R&D, schedules become much less certain.
If my business is churning out Wordpress brochureware, I can give a pretty good estimate of how long a given job will take and how much it costs.
If on the other hand my customers require me to develop novel, publishable algorithms, then any estimate I give will need to have very wide bands.
- Accumulation of technical debt is bad.
- The latest trendy programming language won't make your hard problem all that much easier to solve.
- A complex system can't be properly integrated and tested in two weeks (as recent news reports have shown).
- The CEO setting an arbitrary deadline doesn't magically alter the laws of physics.
Is it always? In dollar amounts, I really do not know that it always is. "Don't let the perfect be the enemy of the good" and all. As a developer, I am always going to be the one push for better tests, more efficient systems, etc. but pg's post a month or two ago about scaling without automation (very rough paraphrasing) points to the fact that this does not always hold.
> The latest trendy programming language won't make your hard problem all that much easier to solve.
Maybe not the latest, but I would think that it was way easier to develop a desktop app in C++ than Fortran, develop a website in PHP than C++, develop a web app in Ruby than PHP, etc. Tech definitely improves over time. Jumping on every third bandwagon seems to be worthwhile.
> A complex system can't be properly integrated and tested in two weeks (as recent news reports have shown).
> The CEO setting an arbitrary deadline doesn't magically alter the laws of physics.
These are not engineering problems, but political/administration problems that have been around forever and in every endeavor. I am certain that stone masons building cathedrals dealt with the same sorts of incompetent management, despite a dedication to craftsmanship and engineering.
Mortgages are debt, are they bad?
Yes. Yes they are. Many people make a huge bet on the housing market with borrowed money, without even realizing what they're doing.
The majority of people involved in making decisions and programming right now haven't been doing this professionally for 30+ years. And for those who have (I'm closing in on 20), the culture that they grew up in was not shaped by people with experience programming.
The examples of "programming" and "lessons" from IBM/etc in the 1960s hardly affected anyone outside of those few small teams with respect to learning lessons re: software development.
IMO, "modern" development didn't really start until the early-mid 90s, when networking really started to take off at the small business and consumer level. This has meant that people who'd never dealt with making software decisions before now are - most of the clients that I have who are in their 30s and 40s have had 'website' duties thrown in their laps, and it's just another item on a checklist.
The lessons and timetables are still new to these people - certainly the mentors and bosses they had 15 years ago never had to deal with this stuff, so they were never exposed to it. The last 15-20 years have been huge in terms of businesses getting crash courses in software experience (whether they wanted it or not). It'll be another 15-30 years (imo) before these maxims are taken to heart by organizations when planning, because they'll have had 2-3 generations of institutional successes and failures to draw on. MOST orgs don't have that yet.
A correction: They've broken no ground on it. An AP article today reports that the "California High-Speed Rail Authority is in escrow on just one parcel of the 370 it needs" for the first 30 miles. http://abclocal.go.com/kabc/story?section=news/state&id=9320...
And the entire effort is in trouble: "Sacramento County Superior Court Judge Michael Kenney ruled that California's high-speed rail authority had violated the letter of the 2008 ballot initiative authorizing $10 billion in bonds for the 500-mile train's construction." http://online.wsj.com/news/articles/SB1000142412788732410820...
I actually agree with the parent comment re: engineering but wanted to make sure that folks didn't think ground had been broken already on California's high speed rail plan. I wasn't sure and had to check myself.
So in an abstract since, and in a world where engineers are the only ones involved in construction, I understand his point.
And software development is never involved in any of that?
While I suspect software is more often plagued by problems I do not see that it is another kind of problems than other forms of engineering.
A few examples do not an argument make. You don't think in the "law of long numbers" sense.
If you did, you'd find out that MOST engineering projects do end on time and budget, and even if they miss, they miss far more gracefully and with extremely better predictability of outcome that MOST software projects.
That you can find some counter-arguments doesn't mean much -- what TFA proposes holds in general, and holds much better than the alternative propositions (that engineering is as fucked or more fucked than software). Of course there will be several counter-examples, it doesn't have to be perfect to be near-perfect compared to software schedulling.
>And by the way, those "Engineering" disciplines have only been around for, what, hundreds, if not thousands of years. What they do for a living is mostly explored territory.
That just reinforces the original posts thesis.
For software, if you need another copy, that's a trivial task. It's more on schedule and budget than any skyscraper or bridge could ever hope to be. Once built, we can use it everywhere. The only thing that requires any real effort, is building that very first skyscraper.
FWIW, I've had the "Real Engineering" discussion before -- way more in depth than you illustrated here. My grandparent response above is probably a bit of pent-upness towards those discussions more than anything about your article in particular.
BTW - thank you for mentioning (indirectly) that civil engineering is the world's oldest profession. ;-)
My preferred term is "artisan". A software developer is applying techniques acquired primarily through experience, supplemented by training and outside knowledge. The artisan has a clear vision of the end result in high-level strokes, but the actual process and implementation is strongly dictated by experience and intuition.
Obviously there are different branches of software developer that have more characteristics of engineering, others of science, but I would hazard to guess that 80% of software development is essentially a guided artistic process.
You're not a scientist, and not an artist. Being a scientist and a programmer is extremely different because of a simple, seemingly minor thing: the lenght of the feedback loop. As a programmer, your feedback loop is sometimes an hour, sometimes a few minutes, sometimes a second (IntelliSense/other IDE help), sometimes weeks. But for scientist, it's sometimes forever. So it's for very different personalities.
But that's semantics really. We're talking about what sort of techniques do you employ. It doesn't really help to say that programmers use "programming" techniques because it says nothing about the common attributes with other industries / professions.
Which is where "artisan" comes in. A good developer is, IMO, more like an great chef - excellent technique, a wealth of experience across many disciplines and the best tools are the (ahem) recipe for success, not "good process".
Also computer science might be the worst -- there is zero science in CS. Science has to do with nature!
 R. Feyman: http://www.youtube.com/watch?v=lL4wg6ZAFIM
Just the same software developers are not engineers since they don't have to uphold an industry certification process nor are they held to public safety and ethic standards.
First, software engineers exist in a broader culture. The connotations of the various words in our respective languages are not in our control. "Engineer" and "gardener" have very different connotations. In a profession that sometimes struggles to gain respect, it would not behoove us to use a term that (wrongly in my opinion) is associated with lower social and professional status. If we could rewrite the English language and define "gardener" as a person with rare, hard-to-learn, and highly valuable skills, then perhaps we would do well to embrace the term "software gardener." But we can't.
Second, software is engineering. This article seems to define engineering as Big Design Up Front. While that may describe part of what engineering can entail, it's not a complete or general description. If I had to describe engineering generally, I'd say it has to do with understanding and manipulating complex systems, analyzing and fulfilling requirements, distilling vague notions to concrete implementations, and building things that work. Software fits all of that.
 As I hinted above, I don't think it's fair, right, or accurate to denigrate actual gardeners as compared to software engineers. I'm stating the way most people perceive social status, not the way I'd like them to.
But come October, they started to roll off the line...20+ kips of concrete a pop and then stacked three high. To this day, the sensation is fresh; "So that's what the fuck it really looks like."
The reason it's fresh is because it returns every time I am in the presence of some lines I put on paper manifested.
The picture of the bridge is not the bridge. The engineer, like the hacker, has better intuitions regarding the way in which the picture will correspond to its instantiation - but it is still no more than a [hopefully] informed intuition.
 The idea that there is an Engineer rather than a thousand authors of a successful bridge tends to be a bit absurd in modern construction practice. Such heroics are still far more likely in software development.
If you're building a skyscraper or a bridge, an engineer has to sign off on the design. If the design subsequently proves to be defective, said engineer is personally liable. In many places, calling yourself an engineer when you are not licensed to sign off on designs and be liable in this way is against the law.
We can debate whether or not extending this principle to software development is a good idea. What we can't debate is that software development does not currently work this way. You, the software developer, are not an engineer. If you think a design is bad, you do not have both the legal authority and responsibility to stand up, declare it so, and force the design to be changed to something saner.
But saying you're not an engineer because software projects don't succeed like projects done by real engineers - that's both silly and wrong.
That's what makes it a stupid point: it depends on a specific definition of engineer that is not used (and is not possible to use) with software engineering, and that definition is oriented around chartering/certification of the professional themselves.
That he buttons the essay up in such a self-serving way is just icing: taking "engineering" too literally for the context, then calling himself a gardener.
I wish we took the engineering metaphor more seriously. In particular, I'd like to see some real consequences for malpractice on the part of practitioners -- this would help us in our struggles with management to get projects done successfully.
I've seen so many cases, for instance, where people screw up simple things, such as generating primary keys, and keep making the same mistakes over and over again. This has got to stop.
I agree as long as we also see some of the benefits - required registration with a professional body - along with a requirement that certain projects cannot use uncertified practitioners (in other words, better pay for Software Engineers).
The only legitimate argument I can find regarding the use of the word "engineer" is that there are actual engineering licenses that can be obtained. But if we said that you had to have such a license to be called an engineer, that would screw up the naming conventions in a ton of industries.
Perhaps it's less applicable to civil engineers, because of the scale and safety requirements of their profession, but let's look at electrical engineering. An electrical engineer will build prototypes; they will build models; they will use computer simulations to predict how systems will behave; they will build testing and QA infrastructure. It's definitely harder to iterate—once you've built a million widgets, shifting circuits around is going to be expensive—but it's not qualitatively different to developing a software product.
As in any engineering discipline, software engineers must optimise for a system's required qualities. In some cases, reliability might not be that high up the list. In others—let's say medical systems, or those used in space travel—software engineers have rigorous systems in place to ensure quality, and in many cases substantial up-front planning and certification will be involved.
Ultimately this seems to say that software engineering is different from (specifically) civil engineering. I think that's broadly rather true – the nature of civil is such that generally there are stringent safety requirements, and indeed massive costs involved in making subsequent changes. The majority of software does not have the same requirements, and so it's no surprise that the process is somewhat different. And I agree that not all development is engineering, of course, but to dismiss the entire field is a bit of a shallow argument.
To me it feels as if we are just at a very early stage. It takes huge amounts of time to learn software engineering, putting aside mastering it and humans haven't been practicing for very long. So I'd guess that we're just clueless enough to not be able to feel like engineers yet. We learn most of our practical abilities from experience so thats an argument for craft, but once we get a better understanding of what we are actually doing, and formalize it, our job will probably be very engineery. There is just millions of problem domains and no formal education on the domain of writing software yet (no a CS degree is not it). And I think there is a class of applications that are already so well understood that they do get developed in an engineering fashion already.
All analogies eventually break down if you take them too far. (If they didn't then you wouldn't be comparing something to something else; you'd be comparing something with itself.) But software development is so different from so many other practices and processes that most comparisons to gardening, building, finances, etc. don't provide much real value.
Its especially tiresome when it's claimed that "sofware isn't like that, it's like THIS". In the article, I find that neither gardening nor engineering is an especially enlightning analogy to software development.
Saying it's just "whatever engineers do" is meaningless.
"Engineering is the application of scientific, economic, social, and practical knowledge in order to design, build, and maintain structures, machines, devices, systems, materials and processes. It may encompass using insights to conceive, model and scale an appropriate solution to a problem or objective. The discipline of engineering is extremely broad, and encompasses a range of more specialized fields of engineering, each with a more specific emphasis on particular areas of technology and types of application."
Application of scientific/practical knowledge: check. Design, build and maintain systems/processes, check. Using insights to conceive, model and scale a solution to a problem, check.
Sorry, but this is just a nonsense discussion imho. Stop devaluating your own grade.
I'm not a Civil Engineer. So what? What does a Civil Engineer have in common with an Electrical Engineer or a Chemical Engineer? Very little day-to-day.
Or an Engineer on an oil rig? They'd laugh that Engineering means getting it right every time.
I can also tell you a Civil Engineer has very different problems if you're building a skyscraper in Asia, somewhere waterlogged, as opposed to somewhere geotechnically active. If you picked up your average skyscraper and dropped it in New Zealand, you'll have problems. Saying the techniques are the same is a gross generalisation.
I agree with the author though, that part of the problem is using engineering as a metaphor (at least, I agree up to the point where he just replaces it with his own, equally bad, metaphor). Software development is not like engineering, it is engineering, and the actual practice of it is as different from civil engineering, mechanical engineering, electrical engineering, chemical engineering or any other branch of engineering as those fields are from each other. None of those, by the way, bear even a passing resemblance to the description of 'engineering' contained in the post.
So yes, let's quit with the metaphors. Instead, let's talk about what engineers actually do, which is design:
Its rather a case of each engineering profession is a regulated industry with an accreditation body, that certifies you as an engineer. Mechanical, electrical, electronic, chemical, industrial, civil, all have standards organisations, usually different ones in each country.
See IEEE for instance...
Also the engineer term is a protected designation in many countries, just as doctor and laywer is...
So no. engineers aren't guys that design stuff, and until the software industry is as regulated as the rest of the engineering industries are, software engineer is the wrong term. The only people who can call themselves software engineers perhaps is working in the medical device, avionics software etc. industries.
But again comparing modern day professions to the 19th century equivalent does not bode well for your argument. If you'd prefer the medical industry return to its standards and practices of the 19th century, your welcome to visit such a doctor.
I prefer my modern day certified medical professional.
Engineering is the same. Licensing most likely raises the quality of engineers, but that does not mean an unlicensed person who does the job of an engineer is not an engineer.
My point being, in this day and age, to call yourself a docter, you must be licensed. Bringing up history does nothing to change that fact. I would state that should also apply for engineers, and already does in most countries.
Merely the fact that train drivers are called engineers in the US proves my point. It dilutes the term to be basically meaningless.
There are, in fact, many places where engineer is not a protected title and to claim otherwise is just wrong. Many large countries like the US, UK and France do not restrict the use of the title engineer. They only restrict something like "Professional Engineer" or "Chartered Engineer".
There is a great deal of academic study in the field of "software engineering". That I agree with, but that academic research is being done by degree'd people, either with computer science or computer engineering degrees. Even still they don't practice engineering, they are researchers or scientist. Much like in the other engineering professions, scientists study and advance the engineering fields.
They only restrict something like "Professional Engineer" or "Chartered Engineer".
Ok that I agree with, but I restate, writing code does not make you an engineer, anymore than performing an operation makes me a doctor. Being board certified, makes you a docter. Same with engineering.
I would hope we would have gone a bit further than the Renaissance era in how we define professional fields. Would you also call Leonardo da Vinci a doctor, since he did breakthrough anatomical work?
In that same vein, would you call a random person digging up graves and dissecting corpses a doctor today? Would you let them operate on you?
I wouldn't let either operate on me, but I also wouldn't let a licensed doctor from 1920 operate on me either. Modern doctors have vastly superior medical knowledge compared to historical doctors, but that doesn't mean that historical people who practiced medicine aren't doctors.
Me thinks the logic isn't quite sound...
I have no interest in worshiping the "real" engineers with their protocols and whatever. An engineer is qualified to build a bridge because they know how to build a bridge. Their professional ethics and protocols are secondary. And we certainly have ethics of our own, even if it isn't codified and we don't wear special rings.
I do not 'grow' nor 'tend' code, trusting some unknown universal force to keep me on the right path.
I engineer buildings, bridges, roads, tunnels - all sorts. Sometimes they're very small and sometimes they're very large but they're all subject to the forces pushing and pulling on them - my job is to build them and keep them upright as well as I can subject to the laws and constraints of the environment.
I am not a gardener. I am an engineer building, modifying, and destroying complex structures daily in a controlled dance so as to keep their superstructure useful and safe.
Only to the naive eye could such complexity be reduced to such apparent simplicity.
Can a civil engineer say exactly how long it will take to design a bridge? No. Might putting more engineers on that project speed it up? Perhaps!
Coding is not so much constructing a program, but more creating detailed descriptions of every aspect of the system's operation for someone else (an interpreter, compiler etc.) to follow, and that is very similar to engineering design.
I'm not going to get into a who has bigger stones pissing contest, but lets just say the expectations and requirements for skyscapers are different than software.
You'd do just as well to call engineers "traffic gardners" who don't know how to make a bridge scale.
Therefor in my opinion software development "done properly" has more in common with engineering than gardening. :-)
That said, Chris has a good point that calling what you do engineering doesn't make you an engineer.
And the lowest bidder for contracted work often sucks, or finds ways to bill more than the original bid. :)
Gardener fits, but we design the plants in essence....
To me Programmer, developer, even software architect seems closer than gardener.
Building software with an engineering approach is a real thing. Not everyone subscribes to it, but doesn't mean it isn't a real thing.
For the record, I do a lot of gardening too, and building software has fuck all to do with that.
I also don't get how everyone who argues against software-as-engineering talk like buildings and bridges never fall over. They do.
2. It is quite dumb to assume we build software in the same way that bridge is built. That's essentially what Water Fall method would do. You build the blueprint precisely, build a prototype and then build the actual bridge. I don't see why the author would assume we think we were classical engineer. I never assumed we would be using the same technique all the time.
It's a very important thing to understand and explain that software development is _not_ an engineering discipline -- at least not yet. We strive to make it another branch of engineering, that's why we have design patters, methodologies, etc. Engineering is usually boring, but predictable, and that's a very important property software lacks at the moment.
I use old tools (vim, terminals), like engineers use old tools (protractor, pencil), to make comparatively much larger artifacts (search products, management tools), like engineers (buildings, bridges).
You're not judiciously applying the metaphor. You've only rephrased what it means to be "self-taught"; it goes without saying that we don't allow unlicensed people to engineer things. It also goes without saying that no one lives or dies by a faulty HTTP request header.
More analogies: Should we not have software architecture, because it isn't real architecture? Should we not refer to systems programming, unless the practitioner has studied systems theory? Should we not have windowing systems, because no real windows are involved? Almost everything in computing is termed by metaphor with 'real' things.