After 20 years doing this job I can confirm that building software is really hard. I’m not sure how you really give people experience with that. My courses at uni did little to prepare me for building software.
Once you have a codebase in place, things are a whole lot easier. There are far fewer decisions to be made. Really you have little choice in how to do things as the current structure dictates your options. Discussions with customers are easier too, as you’re no longer talking in the abstract.
But it’s really challenging reaching that point.
Edit: having said all that, this does look like an interesting course
I think the problem is you really need to feel the pain of a few badly organized projects to understand why we need things that seem like extra time sinks. It's hard to design a course that deliberately goes about building a project the wrong way without seeming contrived, and yet it's easy to just start writing something that ends up being a big ball of spaghetti.
> I think the problem is you really need to feel the pain of a few badly organized projects
I'd go further than that. I've come across a few programmers who had _only_ built new systems, or maintained systems they had built. You could cajole them about coding standards, documentation, unit tests, clear naming until you were blue in the face. It is pointless.
So you force them to maintain someone else's code base. Be prepared about the rants about how bad code base is, whinging about having to learn all the unfamiliar libraries, the total lack of comments, yada yada. But somehow the connection between how that programmer worked and how they worked is never made.
Then 6 months later, after a some of the intimate knowledge of their own code they were using to make maintain it tractable is gone, you move them back. Then, finally, it hits home. They improve steadily thereafter.
Yeah, I was surprised there is no mention of drift and technical debt. These are the things that make software engineering hard. Someone new will always suggest a rewrite, which is tempting but is almost always a trap and massive timesink.
You could have them build a project one requirement at a time. That's how projects usually get out of hand in the real world. But I'm not sure if a semester is long enough.
I had a class in grad school called Database Implementation. We wrote a simple SQL database from the ground up, with a SQL parser as the first assignment. I had worked in industry for a few years, so it was fun seeing that "I have painted myself into a corner" look in some people's eyes. And the "I have to rewrite everything" scramble. :-)
That was also when I learned you could deadlock AIX 3.2.5's NFS filesystem using mmap. (I was also working as one of the dept's sysadmins, so putting 2 and 2 together was easy.)
I think you can do this. You take an existing badly done code base, and ask the students to make significant changes and/or additions. After they've felt that pain for a semester or two, you discuss what causes the pain, and what choices led to that point.
> Really you have little choice in how to do things as the current structure dictates your options.
I really like this framing. The architecture of the code should reflect the flexible and rigid parts of the domain, and how they interact. A good design will let small changes in the domain become small changes in the codebase -- and large changes in the domain will at least not be arbitrarily larger in the codebase.
Software architecture feels like a very code-centric and technical thing -- and it's not like it isn't, exactly -- but it's really driven by a solid understanding of the system into which the software will be placed. Often, the pre-existing system is a bunch of humans, and people who previously interacted with humans will now interact with software. Also often, the pre-existing system is some mix of humans and software. And even if the pre-existing system is pure software, some of the internal or external boundaries will undoubtedly shift.
> But it’s really challenging reaching that point.
We as a profession need to acknowledge that not everything is "solved" by software, and that we really need people with design experience, human factors experience, and the ability to distill the domain into something that can be addressed by software engineering. I think "reaching that point" is predominantly led by these factors, and not technical excellence in the strict sense.
It's amazing they have a screenshot of FreeCol in there! That's a Java game I've submitted bugfixes to, like fixing multiplayer, in just a few hours, despite having barely any Java experience, because it's so debuggable and ordinarily architected.
> But it’s really challenging reaching that point.
Choosing opinionated and well supported languages & frameworks is essentially the lesson there, and they'll probably finish that in one week.
The real difference here is that this course does not aspire to cram leetcode, which is what you're actually evaluated on if you get into Software Engineering. In that framing, they are adopting an extremely contrarian and risky position: if their students perform worse in job hunts after taking the course, and entrepreneurial / indie / artisanal software development remains hit-driven, students would conclude the course is bad in the long run.
> Choosing opinionated and well supported languages & frameworks is essentially the lesson there...
There are strong benefits to opinionated frameworks, and reaching an initial version of a code base is absolutely one of them. But the trade-off comes in the next version, when you get customer feedback, and the devs start saying things like, "No, we can't do it that way because that isn't how the technology works."
That doesn't make it the wrong answer for a MVP or a V1... but this is an academic course. I'm hoping they cover the longer-term consequences of early choices in software development. Because that is what is really hard - not getting a first working version, but maintaining delivery speed over the long haul as you realize the limits of your choices.
> Once you have a codebase in place [...] There are far fewer decisions to be made. Really you have little choice in how to do things as the current structure dictates your options.
And for these exact same reasons working with an existing codebase can make further software development a challenging process. It's all a matter of how well those past decisions line up with the present and near future which may not have been apparent (or simply not considered) at the time.
> Once you have a codebase in place, things are a whole lot easier. There are far fewer decisions to be made. Really you have little choice in how to do things as the current structure dictates your options.
You still have the choice to extend the current code, or build more code around it.
For example, you have a ticket system and are now tasked to build full-text search for it. You could try to do it within the existing code base, or to add an exteral search engine, sync that data to it, and build a shallow integration between the two.
That said, you are right in that deviating from the current philosophy of the code base is a big risk, and so a big decision to make.
> After 20 years doing this job I can confirm that building software is really hard. I’m not sure how you really give people experience with that. My courses at uni did little to prepare me for building software.
I'm kind of amazed at just how much experience it seems to take to build a robust & generalizable model for making reliably good decisions about code complexity management & problem solving techniques.
Pragmatic programmer does a decent job of it. Not perfect, but decent enough.
I'm of the school that thinks it's something you need to practice. Many software positions especially in large companies are very limited in scope, you basically just maintain and patch existing systems, rarely having to build systems from scratch. Building these systems from the ground up is what is hard but you learn what works and what doesn't the more you do it.
If you want to learn how to build software, join a start up or small company that has a healthy appreciation for what software rewrites can provide. Run away from companies that follow the never rewrite mantra. Usually saying something like you'll just repeat mistakes, its like well no shit this thing was built before I was born, if you rebuilt it with me here I could avoid those mistakes.
I'm not saying you should always rewrite, its usually better to refactor, but some design failures are hard to refactor out, have double binds or are too critical for stability woes. But systems that live forever should have unlimited quality assurance budgets for refactor and repair, very few do in this industry. Managers seem to expect systems to live forever all while constantly patching with very small refactoring budgets. It's completely insane.
Like running a machine in a factory 24/7 and never doing maintenance but expecting it to work perfectly like new forever with no future cost incurred as eventual debt.
> Managers seem to expect systems to live forever all while constantly patching with very small refactoring budgets. It's completely insane.
Preach it! I don't know where this mentality comes from, but it seems to treat software as something that just magically solves problems and only needs a little upkeep to chug along.
The problem is that problems change, and a different problem is no longer necessarily a solved problem. It takes a lot of design effort to make something that can easily be adapted to changes in the problem space, and it still requires effort to make sure the software doesn't specialize itself over time into a corner.
Yup! And then sometimes it's actually advantageous to specialise systems into a corner, it has benefits, but these decisions need to be considered and made with care, and then it needs to be an understood limitation going forward that it can't be rolled back at a later time without a high cost.
Thinking about the above further I notice the things I've learned over the years also require the ability to link problem to cause, or in my case it came in reverse, cause linked to problem.
I ran into problems for years, learned how to fix them, and then during code reviews (even in self review), discovered the cause behind these problems.
Sometimes these were causes that would only become problems at a later date because of knock on effects.
I don't know how you teach that. Possibly antipatterns / code smells which usually hint at underlying problems that may be elsewhere / at a different level.
Refactoring by Fowler and TDD by Beck cover a lot of these in good detail.
Refactoring is an amazing book, but I think most people would be better served by starting with working effectively with legacy code, as it shows you how to get to a place where you can refactor confidently.
The examples are kinda dated but the approach is pretty timeless.
I tend to disagree but this might be my biases at play.
Refactoring starts from first principles, almost all examples include the reverse refactoring as often you need to undo optimisations a few steps before you can move forward in a different direction.
I was recommended working effectively with legacy code by the same types of managers that expect forever returns off the same code. They like to point out how that book encourages small patching without refactoring as refactoring is often fruitless in short term money terms, whereas Fowler tends to advocate refactoring as a healthy and necessery part of software development.
I can't say I hated working effectively. I found TDD by Beck more useful, and working effectively with unit tests to also be worthwhile.
All of these books cover a lot of similar ground.
If you're going to read only one or two I suggest TDD > refactoring and avoiding the others.
(Just my opinion based on what I got out of them reading after 10 years on the job, I suspect juniors might have a different take)
However, I have ended up inheriting a pile of crappy code driving business value a bunch of times in my career (data science is much worse for this), so for me, Working Effectively was really really useful as it provided ways to deal with the craziness.
As soon as I finished WELC, I ordered refactoring, and devoured it - but it is less practical for my situations, where there are no tests and nobody sees why they would even be useful.
In terms of good and impactful reads, TDD is probably the most bang for my buck I've ever gotten, in that it's super short and very very good. Kent Beck is hilarious also, which definitely helps.
Of course it can be taught, it has to. The current model of expecting people to just pick it up on their own by going to school or a bootcamp isn't working.
I think a large part of the problem is that in contemporary workplaces it's becoming rare for senior talent to spend serious time developing juniors and doing so as an integral part of their job rather than merely within the scope of some mandated "knowledge transfer" phase.
To many places are beset with pushy "deliverables" treadmills and suffocating project management. It's hard, especially in large organizations, to just "think" and be creative. Juniors need to see that, emulate it, and get meaningful feedback beyond a dumb burn-down chart.
Art schools that teach you visual art and craft is a thing. So I don’t see why it would be impossible to teach the art of building software. We just need to change our methods.
> This is a stark contrast to Engineering where everything is proven with math.
You've just described the field of formal methods. For software with strict requirements of correctness, correctness is sometimes proven mathematically.
It's not that a rigorous approach to software development is impossible, it's more a question of when it's really worth doing. At present, formal methods are very costly to use. I don't think it's always necessary to go all the way to full formal verification for software development to count as rigorous, though.
Related reading: They Write the Right Stuff, from 1996. [0]
> efficiency is obscure at modern coding levels
Depends on the domain. There are plenty of domains where efficiency doesn't much matter, on modern hardware. Game engines are still tuned for efficiency, and will continue to be.
Okay I'll explain one more time, maybe a second write will clarify.
In Engineering, everything is defined with specifications and based on specifications you can calculate if something will work. If you need something to work at 100 degrees C but it will never get to 110C, you don't buy a material that works at 110C, you find the cheapest material that works at 101C. (Factor of safety aside)
You can prove in math your design will work
In Programming, unless you are working at switch levels, how could you prove your code is most optimized?
It's not going to be. Abstraction has removed the possibility of doing that.
The closest thing I've seen to Engineering in the Programming world is Industrial Engineering. This is the way Engineers optimize production environments. You can see this in Programming in various time/load aspects. But industrial engineering is an "After the fact", and as much as I like optimization, it has a reputation of not being Engineering.
Hope that makes sense. Science and math vs optimization.
It's possible, but extremely labour-intensive, to mathematically prove the correctness of a program. I mentioned this in my previous comment.
> In Programming, unless you are working at switch levels, how could you prove your code is most optimized?
'Most optimised' is an entirely different problem than 'your design will work'.
Complexity theory lets us do something like this, but at a more abstract level, rather than at the level of real-world performance on a particular machine.
As you say, real-world code is almost guaranteed not to be perfectly optimal in terms of performance. We really never need to aim for this though. Market forces push for high correctness and performance, to varying degrees across different different problem domains. Performance matters in game-engines and for high-frequency trading, and in those applications, the software engineers put in great effort to optimise their systems, using fewer abstractions.
Sometimes the software engineering challenge may be a life-critical hard-real-time system, such as in medicine or aviation. Software engineers are able to deliver such systems. It's not of great concern that these systems aren't perfectly optimal in terms of their performance, provided the programs give the right outputs and always meet their deadlines.
The generation of perfectly optimised code has been researched, branded superoptimisation, [0] but it's little more than an academic curiosity. I doubt it will ever be possible to scale it up to be practical for large programs. (I'm not sure if/how they deal with the way most modern processors are terribly complex and don't have easily predictable performance.)
> Abstraction has removed the possibility of doing that.
It's very often a sound decision to trade off on performance in order to gain on some other dimension, such as development velocity, or maintainability, or indeed correctness, given the project's resource constraints. As tools improve, the Pareto frontier advances, and we get a little closer to being able to 'have it all'.
An example of this might be the use of Rust for web development work, which can apparently greatly improve performance (or greatly reduce the computational resources needed). It can bring the performance of C++, while retaining many of the advantages of safer, higher-level languages like Java and C#.
Whether it does this well enough to really grain traction, we'll have to wait and see, but I think it's a good example.
> The closest thing I've seen to Engineering in the Programming world is Industrial Engineering.
Industrial engineering is an entirely different discipline than software engineering.
For examples of 'proper software engineering', the obvious candidates are avionics (as discussed in the article I linked above), and development methodologies involving formal methods, such as with the Tokeneer project. [1][2]
Software engineering is an optimisation problem in the same sense that other engineering fields are optimisation problems. There are tradeoffs to be made on many dimensions, just like in any engineering discipline.
Again, mathematical tools can be used to prove the correctness of programs.
I think you are missing the science and math aspect of this. This isn't a contest. Physicians make lots of money by following tradition and practicing medicine as an art. There is huge debate that Physicians could lose their monopoly powers if medical is a science.
>Once you have a codebase in place, things are a whole lot easier.
You sound blessed to not have run into huge spaghetti code with no comments, no testing, violates best practices, and doesn't work for the purpose, and also doesn't compile.
After this job doing bug fixes and feature adding, I'll take new code ANY day.
Oh don’t get me wrong, I’ve worked in plenty of bad codebases. One job I walked into and on day one the two other senior devs said “congratulations, your share is these 1M lines of code”. It was a total mess. It was back in the day when coldfusion didn’t have functions and instead you called modules that accessed and updated variables in their parents/grandparents directly. There’s nothing more devilish than code that can touch any other part of system. By the time I left there I understood the domain well enough to rewrite that whole system away. I feel your pain.
I’m talking more about how to start things right so you don’t end up in that place. It’s really hard.
And my point still stands that in your codebase, it’s done. The ship has sailed. There’s some comfort in knowing that you’re just going to have to make the most of the restrictions you’re working within (small comfort, I know).
After 20 years doing this job I can confirm that building software is really hard. I’m not sure how you really give people experience with that. My courses at uni did little to prepare me for building software.
Once you have a codebase in place, things are a whole lot easier. There are far fewer decisions to be made. Really you have little choice in how to do things as the current structure dictates your options. Discussions with customers are easier too, as you’re no longer talking in the abstract.
But it’s really challenging reaching that point.
Edit: having said all that, this does look like an interesting course