I’ve worked on a lot of personal projects over the years that were never finished or were quickly abandoned. I used to feel bad about this, feeling like I had wasted my time. But then I realized that I learned a ton by doing all of this work. Often I’d have a need for what I learned a year or two down the road. So now I take the attitude that if I learn something, time spent programming is never wasted.
There's a joke that every web developer makes a personal page by first writing a custom bespoke CMS over a couple of months and then only using it for one "Hello, World!" post, but honestly, I've found that's been a great use of time for me. When I do real work projects, I've ended up saying "Didn't I solve this for my blog a year or two ago?" and then I dig up the code and reuse it. It's a great learning experience, and I think every web developer should do it.
> There's a joke that every web developer makes a personal page by first writing a custom bespoke CMS over a couple of months and then only using it for one "Hello, World!" post
I might be able to top that. About eleven years ago, I went through this process:
1. I wanted to redo my web site. But I hated using PHP which is all my shared host allowed.
2. Maybe I'll design a little programming language that transpiles to PHP. Then I can program my site using that.
3. I have the parser working and producing an AST. You know, writing a little interpreter that walks that AST is practically trivial. I'll do the PHP backend later, but let's give that a try.
4. Whoa, I have a working language! Neat! But, man, that tree-walking interpreter is slow. Maybe I could translate the tree to bytecode and write a little bytecode VM...
5. Bytecode is so fun! But I'm relying on the .NET VM for most of the heavy lifting like the GC and object representation. I wonder how that stuff works. Maybe I should try writing a little VM in C++...
6. That wasn't as bad as I expected. I learned a whole bunch. Maybe I should write some of this down.
So, uh, now I work at Google on a programming language and wrote a textbook on building interpreters [1]. I never did finish that website. (I just ported my blog to Jekyll at some point instead.)
That's one hell of a story! Myself, I only got as far as implementing a CMS with a virtualized filesystem with symlinks (and I think it was even rudimentarily distributed); lots and lots of pointless wheel reinvention, but I've learned tons of stuff and ultimately managed to use it to skip a subject at university. What I didn't manage to do is to use it to build my website (ended up keeping my Wordpress blog, later replaced by a DIY static site generator).
I've seen you around here a lot and enjoyed your comments, but until now, I never realized you're the author of "Crafting Interpreters" and "Game Programming Patterns"! These are some of the best resources I've ever read on-line! Thank you for doing those!
Literally laughing out loud now. Yes, this is a pretty fair observation.
I read once that a famous author got started because they read some trashy novel and was so incensed that it had been published that they decided they could do better.
> I read once that a famous author got started because they read some trashy novel and was so incensed that it had been published that they decided they could do better.
Jim Butcher with Dresden Files, or someone else? Because that pattern-matches to "highly promising book recommendation" for me.
"... struggled mightily until deciding (out of frustration) to reverse course and follow Chester’s rigorous writing techniques to the letter, doing all the plot outlining and character background sheets and so on that the English literature folks often scoff at as the workmanlike tools of genre hacks."
My "how I got a job at Google" story is even more bananas.
Roughly around this same time, I was a very burned out game developer at EA. My job was doing data analytics on telemetry data reported by games. Basically, any time you started a game of Madden, the game would phone home to an EA server and tell EA which stadium you picked, which teams, game mode, etc. EA wanted to use that data to know what to prioritize in future versions. I was working on some rudimentary code to scrape those log files. Nothing that felt like making games in the least.
At the same time, my employment contract prevented me from doing game stuff on my free time. Most game companies essentially own your brain and don't let you do your own game-related stuff on non-work hours. So here I was nominally in the game industry and because of that, I was doing less stuff that felt like making games than ever before. It sucked.
In a fit of pique, I made a throwaway account on Reddit to whine about this. A complete stranger saw and said, "Well, if you know C++, Google is hiring and I could give you a referral..." I already had an interview lined up at ArenaNet in Seattle at the time. The recruiter for Google rushed to get an interview with Google going while I was in Seattle. I wasn't super interested because what the hell do I know about working on search engines and server stuff? However, I wasn't about to turn down the opportunity to see a Google office. My plan was to stay in games.
Then I bombed the ArenaNet interview because the interviewer didn't know what amortized complexity was and I couldn't do a good job of explaining it. (They asked what the cost to append to a growable array was and I said "constant time on average" and they looked at me like I was an idiot.)
Somehow, I managed to squeak through the Google interview. I had no idea at the time how few people get offers. If I did, I probably would have been a lot more nervous and fucked up the whole thing.
So, yeah, I got a job at Google and moved across the country because of a whiny throwaway account I created on Reddit.
I too got a job from posting too much on Reddit. Another user told me to apply to work with him, and that became my career when academia didn't work out.
that's really cool. After some code I posted here on HN got featured on top 10, another Brazilian guy from Google asked me if he could refer me. I accepted right away. Stopped everything I was doing to study for the interview. I Applied for 3 job openings using their internal tool (two outside Brazil and one here), but never got called to an interview. Probably for the best, I suck at programming with a timer on.
I haven't progressed that much in my CMS, :-) still in-progress. It is not as simple as I think. In addition to the typical blog implementation, I also add extra features.
For example, I included multi-lingual functionality as I wanted google to give me better result when people searched the keyword in my city.
As I had never been satisfied with my design, I found a way how to safely separate content and layout /color/etc. Then I was curious if anyone visited the website, a simple analytics was built. Wait, I also wanted to know in real time of any visit, so I set mobile notification if anyone visited the page or clicked on specific button. This list keeps.... haven't mentioned about my form builder, dynamic popup, and chatbot.
Thanks for sharing the experience, I thought I was that unique.
Exactly. If I’m curious about a technology, no matter what it is, I know that if I build something with it or even just set it up myself and play around with it, I will learn a lot more than if I just read the README or a tutorial written by someone else.
It used to be said all applications evolve until they can handle email, but maybe that should be CMS, or CMS and email. I am fairly certain that 'bespoke CMS' is consuming enough developer-hours to largely account for the 'developer shortage' we have. If you supply an extra dozen developers, eight of them will end up working on another bespoke CMS.
Off the shelf CMS ends up being a mix of feature factories and lock-in traps, though, which probably explains why NIH is so particularly pernicious in this case.
As a corollary, time spent answering StackOverflow questions or blogging is also a great way to learn. Writing code & the blogging about it is the best mix for me.
Shift your focus. You're not writing for others (strictly) but for yourself. Express your ideas, thoughts, what you've learned. I write a lot, but don't publish, and it's for that purpose. The act of writing is an act of communication, which forces you to formulate your ideas into more coherent forms (and to try and remove ambiguity). If you've just learned a topic, or are learning it, and want to write about it, it makes you think about how to express what you've read or studied (ideally try to do it from memory versus pulling up the references you've been studying).
It's similar to the benefit of teaching, my classmates and I always learned more (in undergrad) by teaching each other in a quasi-seminar format in the computer lab during our study sessions. Someone would naturally take over after a bit as the "teacher" for the session as they found that communicating what we were studying to the rest of us clarified their ideas. Blogging can be similar.
Unless you're at the absolute top of the field, it's very unlikely you'll ever add something new to the discourse. That's not a problem, though.
There are two answers I can see: The first is to keep learning, and leave breadcrumbs for the people who come after you. And maybe, at some point, join the top of your chosen field. It'll be a while, but the breadcrumbs are incredibly valuable to anybody coming after you.
The second is both more interesting, and harder: Find a couple of fields that you're good in. Not an expert, but good. Then write about the intersection. That is almost by definition a much smaller field, and you're much more likely to be at the top of that field. (For a made up example, how many people are really good at music, systems thinking, and gardening, and write about that intersection?)
The things you can generate in that particular field are adding to that particular field - and more importantly, they likely contain unfamiliar insights to people who are only fluent in one of those fields. You've added something new to both your field, and their discourse.
Most of us probably alternate between 1) and 2), and only very few of us will ever be recognized experts in a well-defined field and further that field through our expertise.
That's OK. We're still contributing to the discourse, we just target slightly different people.
I suffer from the same problem. Something to consider is that we benefit from having a diverse set of materials on the same topics. Maybe you explain something a little differently than someone else does, or in a way that resonates better with a certain type of person. Julia Evans[1] is a great example of this. For much of what she publishes, she’s not explaining something that you couldn’t figure out from reading the man pages or a lengthy tutorial, but her work is still hugely valuable because it’s concise, and visual.
The other thing to remember is that writing is just as much for you as it is for others (maybe even more). The act of writing down an explanation of how to do something or how something works ensures that you understand it was well as you think you do. I don’t publish nearly as much as I should, but I often write things down in a Markdown file because it helps me organize the thoughts in my head and lets me identify any gaps in my knowledge.
Examples of common patterns add incredible value. A 1000 page document may contain everything you need to know to implement a protocol, but a well written example will be far more useful for most.
> how do you overcome the perception that you have nothing new to add to the discourse?
Perhaps make a tutorial? There are various neat tools (and combinations of tools) that lack approachable documentation. Won't be ground-breaking, but might be helpful and worth reading.
Great question. My blog is to help people answer questions they Google. I'm the top result for searches like "spark partitionby". The content isn't anything new. They're just useful posts for people that are busy and Googling for answers.
Writing the blog post is what forces me to actually learn about the topic and commit the concept to memory.
Unfortunately most of the things I've learned from my incomplete side projects are useless. Some may make for interesting discussion sometimes, but ultimately nothing that will help me in any tangible way.
A few months ago, I was poking around ptrace details for an incomplete (unstarted, really) side project, and I came across a mailing list message suggesting that debuggers should be using perf events instead of debug registers for doing hardware tracepoints.
A few days later (maybe literally the next day, I don't quite remember anymore), a coworker was having an issue where we needed tracepoints to catch who was overwriting bad memory, but the debugger was causing the thread race to not happen properly. So I looked up how to use perf to set hardware watchpoints, and we caught the issue almost immediately after that. I would not have thought to try that had I not been poking around with my side project.
Some of my key skills as a developer are my ability to debug issues quickly and my ability to spelunk through large code bases. These skills were developed largely through poking around code bases for incomplete side projects. Sure, the details of how CVS works is probably never going to come up, but when you stumble across time synchronization issues for code, or the different filesystem semantics of NFS--those details tend to be more relevant, and you might not think to look them up before embarking on the "useless" side projects.
Keep at it. There’s plenty of stuff that I learned that never had any practical applications, but the handful of things that did have been very valuable.
Real world example: I was building an app for myself to scrape some websites and Twitter to aggregate stuff I might be interested in. I had built an MVP that I wanted to get up and running on a web host somewhere so that I could let it run for a bit and then iterate. I was annoyed though, because while I was comfortable setting up servers, I hated all of the repetitive tasks involved in getting everything set up the way it needed to be. I’d tried writing shell scripts, but as anyone who’s tried that can attest, they are fragile and end up being almost as frustrating as doing it by hand. I’d heard about this thing called Chef that people in the Ruby community had great things to say about, so I set out to learn that so I could use it to set up servers for me (and my local Vagrant setup). Long story short-ish: I never did get my app deployed to a server, it’s still sitting on my hard drive almost 10 years later. But my Chef skills turned into a bit of moonlighting to bring in some extra money, and then ultimately let me make a career change and get out of a job that had become toxic. I don’t do Chef any more, but I’m still in a job I enjoy and I make a lot more money than I did.
I’m still going to get back to that app though. One of these days.
I've been writing code professionally for quite a while now. I'm not trying to land my first coding job. The vast majority of programmers working on side projects are already established in the industry.
I've learned a few novel things from my unfinished side projects, but I've never really used them in my day job.
> The vast majority of programmers working on side projects are already established in the industry.
I strongly disagree. There are teenagers writing custom skin mods in Minecraft by Googling and copy/pasting. There are non-tech professionals making hello world android apps but are too novice or uninterested in getting a professional developer position. There are photographers who write plugins in Photoshop to create copies of their photos at various resolutions. You won't find any of them making github contributions but they're there.
And a lot of the skills you learn from side projects are internalized without being able to explicitly state it, like getting a feel for conventions in the framework or language you're using, or how much abstraction to use in your solution.
Hmm. My views of Java/C++ (mainline productivity langs.) have changed a lot because of screwing around with other languages. I think I 'code better' because of it.
At first it was a rough implementation to see if the tech stack could do what I want; later it was "let's polish to make it usable by others"; later it was "I need to make it good enough that paying customers would be very pleased". Numerous times I kept wanting to just call it finished, but there were always cool features to add.
I've learned a great deal, and it's immensely satisfying to work on. I've explored face recognition and server capabilities (learning a lot and having a good time). And eventually will likely add both features to the released product.
I'm sure there is great value in small exploratory projects. But I really hope that people polish things up a bit more (at least a screenshot on the GitHub page!) and share it with the world in a more-finished form. There's value in that too.
An old colleague tried to convince me that I shouldn't write my own job queue last night because "it's all handled in the cloud". It's things like this that make me a better programmer whilst also allowing me to run software that does exactly what I want.
I see this on HN a lot. The “what the purpose of writing this project? X and Y would suit your needs just fine.” The problem is, you can’t learn unless you do. So even though X and Y may have suited their requirements, they can’t truly learn how X and Y work unless they write their own version.
And in exceptional cases, your similar program will be just different enough in a few key ways that it will open up new use cases.
Redis is just a hashmap (and LRU). A load balancer is also just a hashmap (and routing). There are plenty of interesting questions about which particular hashmaps to use in each case.
Also dragging in a second library also means you are pulling in their issues too. Sometimes a quick little function is all you need. Sometimes though it is so wildly better than what you have it is worth doing. It is about time and cost. One project I wrote a whole subsystem to do a particular thing. On a revision of the OS they had the same thing but way better. Tossed out every line of code I had written and used that. But without writing that thing I would not have know if it was worth it or not.
> One project I wrote a whole subsystem to do a particular thing. On a revision of the OS they had the same thing but way better. Tossed out every line of code I had written and used that. But without writing that thing I would not have know if it was worth it or not.
This sounds like it could be a good story to learn from if you wrote it out.
Not very. It was 2 custom controls. The OS came out with built in ones that did the same thing but fit the tooling better and had a few extra things my code didnt. Great learning exp on writing custom controls though! It brought forth my 3 rules of windows programming. Always read the docs (then re-read them). If MS has a bit of GUI that does something there is usually a control in there to do it. For every function in windows there is usually an 'anti' function, they are to be called FILO.
I wouldn't say "just a hashmap". That would be like saying Kafka is just a log. But I see your point. You could replace some of the very basic functionalities with those data structures.
I think implementing patterns that are "solved" in SAAS or FOSS is a really awesome way to learn. I grow tired, however, of convincing teammates that no, I don't want to support their homegrown HTML templating library in production, having to worry about bugs or security vulnerabilities.
Editing to really affirm that I agree especially with personal projects. I have a lot of projects where pieces I want to build approximate some percentage of something like AWS Lambda, but I want to do a custom version because it's more fun and will make the project easier, since I'm not planning for anybody else to ever maintain it.
> I don't want to support their homegrown HTML templating library in production
On the flip side, that's how projects like Spring, Spark, jQuery, Angular, React, Kafka, Cassandra and Mongo came to be: somebody did something that somebody else was always doing, but a little bit better, and ended up making a career out of it.
Sure! And I think the degree to which you have the "room" to do that in your job depends on the company. I'm not here to spend my novelty budget on a routing framework that can't handle optional parameters.
Totally agree. Implementing something yourself also helps you better understand how established similar systems work. It also takes out the “magic” from things. For example when you look at a compiler first, it seems incredibly complex. But once you write your own simple compiler you can quickly see that a compiler consists of components that can be understood .
David Graeber the anarchist antropologist who recently died had the same argument for UBI as a way of unblocking society from the stagnation the bureaucratic system has imposed on humanity. https://www.youtube.com/watch?v=MN9S0HD8VH8
Christian Kellermann wrote this nice "Code it yourself Manifesto" [0] that always rings true with me. It is more important now than ever before. I believe OP is talking more or less the same thing.
Related. Such a pleasure to read a site designed to be read with a link that works the way web links work. So many articles are on sites filled with utter garbage, flashing adverts, notifications, and annoying newsletter requests that hitting simple links is just relaxing.
This whole topic made me realize perhaps I should set up an RSS feed with sites like Dr Drang's and this one just so I can have regular access to good non nonsense content.
I've often thought this of IoT devices in my home. Because I'm a programmer, it would sometimes be easier (and more fun) to tweak the devices to suit my needs vs. same amount of time doing product research. As an aside, Does anybody know of an open-source IoT ecosystem?
https://www.openhab.org/ is a pretty established one (written in Java). Although if you're interested in one that isn't established and is very open to tweaking I've just refactored my home automation system and if other people were interested it would help my motivation to keep working on it (written in Go): https://github.com/alittlebrighter/igor
I did feel disappointed after spending around two years of evenings and weekends building a 3d libretro front-end with an extensive Lua API (vintagesimulator.com) and then having the people in r/lua basically tell me they were not going to download my "malware" (unless I disclosed the source code) and I should kind of go to hell when I complained about the way they responded to it.
But I did enjoy building it. And I sort of console myself with the idea that I could resurrect the project in the future if I port it to Godot Engine and/or set it up for VR.
> they were not going to download my "malware" (unless I disclosed the source code)
That's a competely reasonable response (assuming you were not in fact disclosing the source code). I assume everyone on r/lua uses pure-FOSS GNU/Linux systems that they compiled themselves and have javascript (and parts of CSS) disabled for all their web browsing?
I spend a lot of time working on projects about the way we program, because it could be a lot more fun. Those projects are not always fun, so I definitely want them to succeed and it's not about the journey.
