Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Can build CRUD apps, what now?
135 points by evnix 15 days ago | hide | past | favorite | 83 comments
I have been building good amount of CRUD apps for the past 10 years or so, but I feel stuck, like a massive wall in front of me.

I started coding in 2010 when I was in High school using JScript and Microsoft PageBuilder with the Help Manual(chm file) that it came with (I had no internet connection). Later I picked up PHP, JS, Java and a whole lot of other technologies which allowed me to be an amazing CRUD app developer, My resume is filled with the latest trendy words like Docker, Scala, Kafka, Spark, Hive, react, K8s or any new framework that seems to make a buzz. All this was amazing but I recently realized that I am not so smart and I can't seem to build anything other than CRUD apps and it is quite depressing.

There is a world out there about compilers, run-times, drivers, emulators, VMs and OSes that I can't seem to grasp, it is just too complicated for me. My Computer Science degree and the newer degrees do not focus on these, they have become 4 year long coding boot camps that focus on getting people job ready with AWS, React, Ruby or whatever trendy. This is true for the majority of the not so prestigious universities out there.

I see a pattern and I feel the older generation is way more capable and knowledgeable when it comes to Computer Science in general. I am pretty sure a lot of people are or were in the same boat as me. I don't really have the time(Have wife & kids) to take a compiler course or set out to build my own OS, what would be the easiest way to learn these lower level things?

Things I have tried so far,

1. Reading the Dragon book(I find this would be useful only once you have built a bad compiler) 2. Paying people older than 40-45 to teach me some C/C++ and some tricks of the trade (expensive but I gained a huge amount of knowledge in a short time)

any other tricks, or comments about your own journey would be helpful.

I don't want to be dismissive but I think we all go through this. My dream of building robots was kind of crushed when I realized as a mature student coming into a cs program after eight years of working blue collar jobs, that I might not have the chops to get into the electrical engineering program. A few years later I'm graduating in the middle of the dotcom crash. Hey, building contact us forms and customizing or building CMS software landed me a job so let's do that.

Ten years after that I was thinking "great, I've spent 10 years validating email addresses on contact us forms" (the reality was quite different, of course, but I was feeling like I'd lost the thread). But then I started doing a little more with ecommerce and got good at mentoring people and so on.

Now, it's ten years after that. I am managing a bunch of programmers and still write a little code. And I'm wondering how I will stay relevant for the next twenty or so years but so far, so good.

And I keep learning more and more. Not about compilers and Docker (this does come up, I guess) but about projects and people, business, domain knowledge about the area we build software for, and so on.

On paper, yes, I'm nothing. Just some average dude. But I really like my job, my team, and my skills.

I'm not saying this will happen to you but what I am saying is you shouldn't feel inadequate because you can't build a C compiler in assembly or something. That's the kind of thing only someone truly, truly interested would do. It's not something you do because you should or because it will make you a "real programmer".

Find an industry or business that fascinates you and figure out what they need to see to hire you and go for that.

Agreed on this. People pay you to solve business problems, not to write code. Not all business problems - in fact probably fewer than half - are solved by code.

I took plenty of OS and compiler classes in college and I was still struggling to grasp some concepts. I think the issue was I was able to understand each individual class in isolation, but when I tried linking everything together, something just didn't click.

I came upon this book/course: https://mitpress.mit.edu/books/elements-computing-systems-se...

From a Hacker News post IIRC, and it helped more than I can express. At least for me it filled in all the little gaps in the computing stack and really made me "get it". I could relate to the old timers complaining about how everything is so bloated now.

The course is very hands on and you build a "full stack" from logic gates to an operating system that can run programs. It's truly amazing, and I think I learned more from this book than most of my college courses.

I believe it's also offered for free on coursera and has a website: https://www.nand2tetris.org/

But if you really want to know how a full computer works from the ground up, I think this book would be well worth the time.

I haven’t done Nand2Tetris (yet), but I can 2nd the idea of learning the fundamentals of computer architecture. I was 12 years into my Software dev career when I took my digital logic class. It was a real Bear for me, but the lab stuff was fun. At the end of it, we had to build a (simulated) 4-bit microprocessor and it was really cool to see it working. Even though I’d been using and working with computers for 30+ years, it was the first time I really understood how computers worked.

This is a great book/course. I started programming at 13 but didn't truly get a grasp of how a computer worked until I took this course at 24. Got me started with embedded systems and all kinds of other fun stuff.

There are three questions: how do you want to make money? how do you want to be entertained? how do you want to sell yourself?

It's very easy to get stuck in the world of CRUD because it pays the bills, but as you noted there are a lot of technologies.

Shifting out of that world requires risk tolerance because there will be failure after failure. That wall you speak of is the nature of hard things and the time you can dedicate bashing your head against that wall.

My advice for climbing/breaking the wall:

Pick a small project that you find interesting: build an OS, game, programming language, VM, etc... and just do it. The key is to cross the hump of understanding and get something simple working, and the good news is that conceptual understanding isn't that far away. The tool to help you get unblock is writing. Write the comments of what you are trying to do, and break it down until it becomes code-like. Read books on the subject and bounce back to writing what you read.

However, going from conceptual understanding to professional environment is different because the concepts start to inter-relate and the complexity manifests.

Prior writings on this:



Do you want to do something non-CRUD because you do not enjoy building CRUD apps, or do you feel like you _have_ to do something else because CRUD is too "easy?"

Most of the world, however, does not work this way. As an example, take Restbucks (https://www.infoq.com/articles/webber-rest-workflow/). You can see how there's a web component to this, though it's much more complicated than your normal CRUD view. I've discovered new challenges in dealing with these problems ranging from building a rich domain to handling side effects.

This. Listen to this person right here. Remember that what’s most valuable to businesses is often not what’s hardest technically.

> My resume is filled with the latest trendy words like Docker, Scala, Kafka, Spark, Hive, react, K8s

Guess what? I'm in the older generation, I have a CS degree from the late 1980s. I'm comfortable with hardware, compilers, run-times, drivers, operating systems, etc. but all that stuff you listed is like a mystery. I don't know what it is, why or when I would use it, etc other than at a very high level.

I mostly write CRUD apps.

I mostly write CRUD apps.

That's where the volume is. There's a big demand for programmers to do web stuff. That demand may or may not continue. It's so stylized that it should have been automated by now. Have an exit strategy.

There are many things in computing that are hard and more interesting, but they tend to be low-volume. From self-driving cars to industrial control, there are many areas of specialization. But not many people do those things. You can get stuck in a niche where there are no jobs near you.

If you're stuck doing web apps, at least learn to do them in Go. The performance is pretty good and there are fewer layers.

Someone recommended Rust. I write a lot of Rust, but for non-web stuff. It's useful for things where you'd otherwise have to suffer through debugging C++. Don't write web apps in Rust; it's the wrong tool for the job.

A user-level understanding of machine learning is useful. At least learn how to start up TensorFlow.

How compilers work, and especially how they optimize, is fascinating, but there's not much demand for people to do that. Distributed database internals are similar. As is networking. These areas are very important, but most of the big problems have been solved and they don't need many people.

Operating systems are also interesting, but things seem to have settled down to either Linux or Windows and mods to them.

The blockchain/distributed finance/cryptocurrency world has a sizable following, but it's mostly the make-money-fast crowd, some of whom are crooks and some of whom are suckers. Be cautious.

> That's where the volume is. There's a big demand for programmers to do web stuff. That demand may or may not continue. It's so stylized that it should have been automated by now. Have an exit strategy.

This is highly variable. The more rote aspects of web development are indeed automatable: the parts that feel like repeating boilerplate because that's what they are. Creating duplicative controller actions, state updates, etc.

But then there are the parts that would be extremely difficult to automate. Creating rich, unique UI elements with lots of custom logic. Anticipating the user's needs before they realize they even have them. These take a lot of skill and experience, and are truly fun to work on – assuming they're your jam. Lucky for me, they are.

> it should have been automated by now.

I thought that's what coding bootcamps were for?

> There is a world out there about compilers, run-times, drivers, emulators, VMs and OSes that I can't seem to grasp, it is just too complicated for me

Regarding VMs and OSes:

Since it seems that you have been working as a web developer, I think a soft transition for you would be to get into modern devops practices:

- start with vagrant to spin up a linux virtual machine and start playing with it. E.g., how to setup ssh keys, how to install packages in an automated an idempotent way (e.g., using Ansible), etc.

- while you are dealing with virtual machines, you'll learn also a bit about networking (e.g., what's a private network? what's NAT?)

- when you know enough you could start using Terraform to spin up virtual machines in the cloud. There's some magic the moment you run one command and you have a linux machine up and running in some cloud provider! Even better: you run another command and the machine is totally destroyed :)

In such a journey you'll be exposed to different and potential new concepts for you: vpcs, virtualization, a bit of security, firewall rules, ssh, tcp/ip, etc.

This might help: https://www.nand2tetris.org. If you go through this, you will build a computer from the ground up starting with boolean logic. It touches on hardware design/computer architecture, building a compiler for an oo language, and building a basic operating system.

Not sure if useful for anybody, but here is my story. I have no real cs-background, I just liked computers in high school and participated in some local programming contests. I started working on my 4th year of uni. It was java enterprise. So yeah, a lot of crud. But even then we managed to build abstract stuff to make crud building easier and fun.

After around 6 years of crud I decided to try something different, and went to yandex (it is like google, but in Russia). First time it was just the same crud (lot of rest api), but after some time I managed to land on some infra projects - kafka, streams, microservices. And it became much much more fun. You really build stuff from the scratch, you design it, test it under high load. Anyway my tips:

1. Don't go for buzz word, find job where you can actually choose technologies. Soon you will understand how to see through the hype. And cost of choosing wrong tech.

2. Stop trying to learn what 'everyone think is cool'. Learning compilers won't make you better developer, esp if you are not into it. And Dragon book is not the best intro to compilers anyway. If you want to learn something fun, find what it is fun for you in the first place.

3. If you want to become better developer, try to improve things you do at your day job. Not something like 'we should use k8s or we die'. More like 'we will have to make a lot of these crud in near future, lets spend some time to build framework that makes building and support them 2x faster and cheaper'. Look at small and achievable improvements first.

4. Invest time to understand how things work. And build your own opinion about technological decisions.

> ...spend some time to build a framework...

Yes, no. Do this and then leave the company. Then I or some other poor sucker gets to maintain your framework that makes you 2x faster but everybody else 2x slower. Yay fun.

Build a framework only if its reeeeeaaaaallly necessary. Whis is most likely never. If you feel the urge to write one, by all means build one for your own side projects, but dont shove it down everybody else's throat

I went to a pretty unremarkable state school and took compiler construction classes, OS classes, AI classes, deep networking classes, etc. Pretty standard course load for many CS BS programs - then (16 years ago) and now.

I really don’t think most modern CS programs are glorified bootcamps. Not sure where you’re getting that from.

I went to two top-ten public research universities and can confirm they were glorified boot camps, unless you get really lucky in your course selection.

Some of the worst offenders were:

* A graduate-level parallel computing class taught in sync with a 1000-person online class aimed at OMCS students. For the level of depth we went into, I may as well have just taken a Coursera course.

* A PhD seminar on compilers, led by a tenured professor who was reusing slides from 2002 and seemed not to know anything about modern compiler theory or practice. "Huh? What do you mean a 'lattice'?" he said. Thankfully his TA cared enough to develop some projects around LLVM, which is the only useful bit of knowledge I took away from the course.

* A natural language processing course with a severely outdated bibliography, where the projects required us to make changes to convoluted "starter code" riddled with bugs. Half of the final exam was manually computing the probability table (by hand, on paper, without a calculator) for a quite complicated probabilistic context-free grammar.

* A machine learning course where the lecture slides were written by the TA, and where the professor was often seeing the slides for the very first time five minutes before lecture.

And that's just the courses I was able to take.

* For my entire four years as an undergraduate at a large public university, there were no classes offered on the topics of programming language theory or computer graphics. They simply couldn't attract research faculty in these areas, and instead of hiring a lecturer, they thought it better not to offer the subjects at all.

* When I became a graduate student at another large public university, I was excited at the chance to finally take some PL + graphics courses. But nope, during my first year, all of the PL + graphics faculty left to join other universities / industry.

To be fair, I don't think an intro level compiler class needs anything that's changed since 2002. Almost any software class that did a competent job teaching you about the state of the art circa 1995, you'd be in great shape. A few exceptions maybe are machine learning and other "AI" fields, where even the principles have dramatically changed.

Everything newer you will learn on the job.

This was a graduate-level research seminar, whose purpose was ostensibly to prepare graduate students to conduct research in this area. Unfortunately, my undergraduate compilers course covered more material than this one.

I think these days, a proper compilers course should also be a comparative study of programming languages. There is a lot to learn from comparing how C++ vs Python vs Haskell go from text -> machine code. There are both practical and theoretical aspects to this, and I would expect a graduate-level research-focused course to cover the more theoretical side.

> For the level of depth we went into, I may as well have just taken a Coursera course.

This may be true for graduate programs, but I wholeheartedly believe that undergrad Comp Sci programs should be completed at a 4-year school mostly because the course work is less specialized, much easier to teach and generally attracts pretty decent pool of instructors at most schools.

Additionally, bootcamps are not online courses and shouldn't be conflated. A 6 to 9-months bootcamp goes through material at such a high pace that it often becomes counter-productive. I have a friend who paid $10K for a 6-months bootcamp, got burned out and learned very little in the end. The entire experience basically turned him away. I have another friend who started Comp Sci program at a community college and he's been absolutely happy with overall experience and pace. I have helped both of them and would generally recommend staying the hell away from bootcamps, especially with no prior programming experience.

* NAND2Tetris - https://www.nand2tetris.org/

Teaches you how computers, languages and operating systems work from basic principles

* Little Schemer, Seasoned Schemer

Teaches you about the power of functions, recursion, various CS topics and lisp's syntax

* K&R C Programming Language

Teaches you about lower-level programming

I am one of millions (I assume) who learned to program with K&R and gcc (using a Linux distro I got from a CD-ROM in the back of a Linux book). I have very fond memories of sitting with it, reading things over and over, knowing the answers were there if I just took the time to understand.

I taught myself to code, started in 2010 after 10 years in IT infra/networking. I’m not a professional developer and it’s a hobby for me - I did a humanities degree with no comp/sci background.

Initially I built a simple crud app to track water changes for my aquarium. For a while there I had it online as a free SaaS app too. It was using Ruby on Rails, and I had no idea how to use JavaScript to make the UX better.

Fast forward a few years and I was getting fed up with voicemail, so I read up on APIs and figured out how to use GCPs transcription service, pointing to audio files in twilio (I diverted my mobile to twilio to act as my voicemail). Actually pretty simple even though it might sound complicated.

Fast forward to last year and I built a free directory to raise the profile of primary producers who were missing out on foot traffic as a result of COVID, closely followed by another project I’m close to launching. These last two projects were in nodejs as I figured I had better finally figure out JavaScript.

When I was completely stuck on something I did a lot of googleing and watching YouTube - that happened a lot, particularly when trying to get my head around promises.

Moral of the story, don’t be put off with CRUD apps, they are very rewarding. Just build stuff you enjoy when you have spare time (for me evenings after child in bed).

I strongly recommend classes at Bradfield School of Computer Science which teach about as fast as possible (https://bradfieldcs.com/) and their self-study guide (https://teachyourselfcs.com/).

I also got a lot out of Bob Nystrom's "Crafting Interpreters" book

Its such a shame they don't offer accreditation. I'm a prime candidate for their courses by the loom of it, but won't undertake without something to show for it.

What is accreditation for? Do you think you’ll be more likely to get a job from having this accreditation?

My personal path hasnt shown that to be true, but maybe I’m the odd duck.

Web dev nowadays don't need to learn how to build compilers, vms, etc. For me they are separate branch / subject, same with gaming which is also a separated one.

Depending on your field / industry, having good knowledge of accounting is magnitude times better than compiler knowledge.

Learning about optimization (both db and app), write good / clean code, debugging are useful skills wherever you go.

Unless you want to change course, it's perfectly fine.

> Web dev nowadays don't need to learn how to build compilers, vms, etc. For me they are separate branch / subject, same with gaming which is also a separated one.

Disagree. Most JS frameworks require toolchains these days, particularly in the form of Babel (parsing/compilation) and Webpack (linking). If you understand the way compilers and linkers work, then these web tools are more useful.

Sadly, a lot of web devs just think it's too complicated and so copypasta their configs without understanding, but lots of frustration.

> If you understand the way compilers and linkers work, then these web tools are more useful.

I'm interested with it. Got any examples at what can they be used further if you know compilers and linkers? So far without understanding it I can use it for compiling react.

However I didn't said that those skills aren't useful. They just aren't requirement for CRUD / web devs. And I'd argue that learning business side is far more useful than technical side (after you've learnt up to certain level)

> And I'd argue that learning business side is far more useful than technical side (after you've learnt up to certain level)

This is an excellent perspective, and a recommendation for OP I hadn't thought of. Learning how accounting or finance works, how sales works and then how to think strategically about what each faction involved in a business is interested in (including customers and various types of employees) can be very helpful. Appreciating and starting to understand design can also be helpful.

> I'm interested with it. Got any examples at what can they be used further if you know compilers and linkers? So far without understanding it I can use it for compiling react.

I'd take a look at the sources on GitHub at first. Maybe clone them, set breakpoints and start debugging.

In React, the compiler (based on Babel) is going to translate JSX into (virtual) DOM operations, plus handle some of the JS features that aren't supported in the vanilla browser implementations. Webpack is going to read in these translated sources as well as the React runtime (which has the logic handling virtual DOM operations) and any other assets you use (images, CSS, etc.) as represented in JavaScript. With everything in the same namespace, Webpack can then begin "linking" the "objects" and resolving references. It will then begin writing them all out as you specify in the webpack config.

This is highly analogous to e.g. `gcc` (C compiler) and `ld` (linker). The compiler takes sources and translates them to something the machine understands, the "object" files. But the references to anything outside of each object file are all broken. The linker takes all of the object files and resolves the references, including to runtime dependencies like "shared libraries"/DLLs, and then it organizes the final outputs in a way the programmer specifies (usually there's a default that just works, but for some types of outputs, you write "linker script").

Also like with webpack, there are all sorts of kind of lousy tools that aim to automate the creation of the build scripts, like Automake or CMake. These tools do save time in most cases, but they are leaky abstractions that are quite complex to understand vs. the relatively simple (though also difficult) compiler and linker, which are fairly easy to reason about with some experience. React-scripts is I guess the latest analogous tool I've seen that tries to hide away webpack, though I've also seen create-react-app and razzle.

The beauty is, if you understand how people were dealing with some of these issues in compiling C or FORTRAN or whatever 50 years ago, it's easier to see what the pitfalls and strengths of the current web toolchains, despite the complete dissimilarity on the surface. The patterns are quite similar.

EDIT: it might be useful to find a tutorial you like on regex and parsing. (Strict) regex are equivalent to "deterministic finite automata," which are conceptual, abstract virtual machines that transition between states based on inputs. Parsers are more related to a class of automata called "pushdown" automata (though there's a lot more variation, and parsers are not actually implemented as PDAs despite some loose equivalence). Understanding the hierarchy of automata (DFAs up to Turing machines and then the purely imaginary ones like "oracles"), as well as the distinction between deterministic and non-deterministic and how those can be made equivalent (for some trade-off in real world efficiency) is also useful. It might also be good to study and explore BNF-type grammars and parser generators. Some of these exist in JavaScript, like Ohm. Ohm is particularly useful because it has a visual browser-based editor that explains exactly how a parse is being performed given some test input and grammar. Once you kind of understand that, you can experiment with other kinds of parsers (Earley, LALR, recursive descent, etc.).

Writing a small calculator from scratch or building your own JSON parser could be useful tools.

One nugget of wisdom is that if you are decently intelligent, most of your failure to understand “complex” topics is really due to poor quality explanations from other people. Few engineers are clear writers. Buy books, sort though university course syllabi, watch lectures. That is, consume materials meant for teaching beginners.

For consuming non-beginner material, it’s okay to read a blog post and not fully understand everything. Things will click together over time.

One model of motivation is that you get motivated with autonomy, mastery, and purpose/belonging.

I understand the 'mastery' bit is less about whether you can do it (clearly you rate yourself well at that) so much as the feeling that you're improving at it.

"just CRUD apps" suggests that you're not feeling you're mastering it, and that you don't feel like it's the best fit for what you're interested in doing.

I think any professional task you can competently accomplish without having to learn anything is going to be boring. I think e.g. Scala, K8s are complicated enough that there's room for learning more about it. Not necessarily "do hard stuff for the sake of being interesting", so much as "being able to evaluate whether this is the best tool for the job".

> I see a pattern and I feel the older generation is way more capable and knowledgeable when it comes to Computer Science in general.

One thing to consider is that the older generation has simply been at it for longer than you have, and this is an industry where the vast majority of experience is gained on the job.

Another consideration is that due to a number of timing factors, a certain demographic (the elder Millenial/younger Gen X cohort) came of age at a time when computers were just recently cheap enough to become a mass-market good and viable for home use, but before they were commoditised and standardised, and before there was an internet full of helpful information, so if you wanted to learn how they work under the hood, you needed to get your hands dirty.

Well, one idea is you could help us get a compiler up and running for our home-brew CPU project: https://github.com/hsc-latte/hs32core-rtl

We are at the stage of lacking a lot of software help. We have an assembler, but that's it.

We're expecting some initial chips back in a few months from using the Google/Skywater/Efabless shuttle program: https://efabless.com/open_shuttle_program

Currently we are simulating the CPU in verilog simulators, as well as fpga's. There's a lot of fun very low-level work to be done.

Hey, this is pretty cool. Did you guys consider writing a LLVM backend for your ISA?

I'd love to help.

do you have experience with LLVM? you can get an idea of our ISA encodings here: https://github.com/hsc-latte/caravel-hs32core (which is probably the better repo to flaunt)

This is real cool stuff and was wondering how I missed this.

Do you need someone to write embedded tools? I am getting into FPGA for RISCV with the Arty board and was wondering if you need any help regarding this.

What are you trying to do? Just stimulate and challenge yourself? Or are you trying to get a better job? It does not sound like you have a passion for compilers and OSes. Neither do I, so I don't try to deeply understand them. There are plenty of other areas that do interest me. Don't force yourself into something that is not particularly interesting to you.

Game development has really intense math and engineering stuff, and it's extremely fun and addicting for some people. If you can get yourself into that world, it'll set you on a track to learning what you're asking for here.

Of course, it's not for everyone, but I'd say it's worth giving it a shot. Definitely start with C/C++ if you haven't already, and try not to stay with existing game engines like Unreal or Unity for too long (unless you're specifically trying to start a career in game development; a bad idea)

FWIW, I think Unity/Unreal can provide that feedback loop to really enjoy building games. There's still plenty of opportunity for mathematics at the scripting level, and the advantage is you won't spend days or weeks trying to draw a square in OpenGL. I took that path and vastly prefer the engine approach.

You don't have to be smart, you just have to surround yourself with people smarter than yourself. The rest will happen automatically.

Have you tried joining one of these projects you mentioned as open source contributor? I know at least some of them are always happy to get more people to join and help. And not just with code contributions. They also have (CRUD) websites that need maintenance, docs to be written, events to be organized, and training/marketing material to be produced. You might be surprised, how these seemingly smarter people end up being super happy about your contribution, as usually what is easy for you will be hard for them, just as you might see it the other way around.

And because you said you have wife+kids, your time might be limited: there are also ways to get paid to participate, at least around the bigger projects.

I often feel the exact same way. I have had to really be honest with myself to understand these feelings. What I have realized is that this feeling can mostly be reduced to fear. Fear of becoming obsolete, fear of being not so smart, fear of not being a "real programmer". It also doesn't help that a lot of people in other fields really do look down on web developers in this way. At least for me, this induces a fear response and I get anxious and it is almost entirely out of insecurity. I start to "stress learn" things that I feel would make me a "real programmer" even if I have no real use for or interest in them. It's pointless, of course, but I have found it to be a viscous cycle that's hard to fight sometimes.

A few things have helped me. The first is that you need to embrace what you do and love what you do. It doesn't sound like you hate being a web developer, more that you feel you should be doing "harder" things. Web development is far more than CRUD apps, so it might be worth looking for another job that will challenge you more as a web developer.

The second thing is that you need to understand what you do _deeply_ instead of _broadly_. I have for a long time considered myself a generalist, but this has its drawbacks. When you really understand a tool set deeply and become an expert a lot of the lower level internals really reveal themselves. You end up learning things that you've always thought you wanted to learn in isolation, but instead there is now a useful context in which to learn them. Not to mention that this helps with the feeling of being "not so smart" because you will form informed opinions as a result of your deep level of understanding.

The last thing is that you do need to feed your curiosity and the only way to do that is through projects. You want to build a compiler, build an emulator, a game engine, a database, or an OS? You really just need to do it. I have dabbled in each one of these things. Some of which I got very deep into, others not so deep. Each time I learned a little bit more. The hardest part is picking a project and running with it, especially when there are so many different options, but it is the only way.

Sometimes it's important to take a step back and appreciate what you do. It's very easy to beat yourself up and compare yourself to others. If you really are unhappy with what you do, then you need to start deeply diving into something that you do find interesting. Nothing in this field is easy and it all takes time to gain any level of proficiency at.

I know it's not for everyone, but being able to build CRUD apps is pretty useful if one wants be an indie hacker. So if you think you'd enjoy running an (indie) online business, I'd encourage you to take a look at Indie Hackers [0], Microconf [1] or Y Combinator's Startup School [2].

[0]: https://www.indiehackers.com/

[1]: https://microconf.com/

[2]: https://www.startupschool.org/

- Take CS50

- Take Nand2Tetris

- Take Casey Muratori's upcoming course StarCode Galaxy.

- Get some inspiration from Jonathan Blow and Casey Muratori on YouTube, and start the rabbit hole there.

- Be prepared to question a lot of the dogma that you have already absorbed.

I wouldn't be so hard on yourself. It's unfortunate that there are a lot of CS degree programs that are basically vocational prep teaching today's tech du jour at the expense of teaching the fundamentals, but this is not a reflection on you, and it's never too late to learn and master the fundamentals. You might not have the time for college classes, but the nice thing about pursuing CS as a hobby is that you set the timeline.

(Note: There is nothing wrong with vocational schools for those who need to learn specific technologies. In fact, many people with CS degrees could benefit from courses in specific technologies. However, the purpose of a bachelor's degree in computer science is to teach students the fundamentals of computer science, and I believe undergraduates students are done a disservice when they are not taught the fundamentals.)

I highly recommend going through the book Structure and Interpretation of Computer Programs (https://mitpress.mit.edu/sites/default/files/sicp/full-text/...). This may be your first exposure to functional programming in Scheme. Even though it was originally designed as the textbook to MIT's introductory CS course (before switching to Python in the late 2000's), this book is useful for anyone interested in computer science, and it's easier to read for those who already know how to program. By the time you get through the end of this book, you will have written a Scheme interpreter, a logic programming interpreter that can handle unification (don't worry about what that means right now), and a Scheme compiler.

Once you finish that book, then it should be easier for your to delve deeper into the topics of your choosing. Want to learn more about computer architecture? I highly recommend getting the latest edition of Patterson and Hennessy's Computer Organization and Design (https://www.elsevier.com/books/computer-organization-and-des...). Want to learn about operating systems? I highly recommend Remzi and Andrea Arpaci-Dusseau's free Three Easy Pieces (https://pages.cs.wisc.edu/~remzi/OSTEP/). For compilers, you already have the dragon book, which should be easier to read once completing SICP.

Firstly, this is the first I've heard about newer CS degrees not having classes on compilers, etc. and are now just 4 year boot camps that focus on AWS, React, etc. If true, thanks for this insight! Helpful to know about the current landscape of CS education. (at least for the not so prestigious universities as you mentioned)

Secondly, I'm not sure if you've spent time in different industries? While you may still be building what amounts to CRUD apps in different industries, your work can be very different and you can have very different learnings. For example, I worked in social games (i.e. FB) and mobile games a decade+ ago, for several years. Engineering was all about high traffic, high performance backends; squeezing every bit of speed possible, handling large amount of concurrents and DAU while keeping systems up. Over the next couple of jobs, eventually I landed in real estate transaction tech, which is almost a polar opposite -- very small amount of traffic relatively, but needing to be very robust and well-designed at every level from API to data store to schemas to ETL. Design patterns and concerns etc. are completely different; to me it's a whole different world of software engineering. I imagine if you move the opposite direction, you would see just as much drastic differences.

> any other tricks

I would look for jobs in software companies who make stuff you want to learn to make, but for some parts of their products they also need these Docker/Scala/Kafka/etc. from your resume. If you are good at these things, they gonna pay you well i.e. enough to support a family. I think over time you’ll inevitably learn their lower-level stuff you have to integrate with. You can then change position within the company should you want to.

This will probably be an unpopular opinion but most software is just connecting a couple layers of abstraction to form a new, novel layer of abstraction that you get to design. You do this all the time with CRUD, what makes you think writing drivers in C is all that much different? If you know to any extent how Node's single threaded event loop works to process HTTP requests, then you know how drivers work too, there are just different best practices, APIs, etc. These things you mentioned are mostly practical and can be learned by doing.

That's different though than "Computer Science in general," which is far more theoretical, mathematical, and logical. These things, while they can be learned by doing, are much easier to learn by being taught. Paying someone to teach you C/C++ isn't going to make you better at Computer Science in General, it will just teach you a new programming language, which is practical. Computer science in general are things like distributed systems, discrete math, algorithms, etc.

I do think it was easier to develop a full understanding of how IT systems work a few decades ago. We played around with Windows 3.11, XP and early Linux distros, setting up networking, etc in the old days™.

Nowadays the level of abstraction is higher (smartphones, FaaS, VMs, UI frameworks).

My tip to OP would be to set up a home lab on hardware from eBay. Run some services on it and manage it from the metal up.

Yup. This advice is spot on. I started with just setting up my own server at home, and over the last 15 years, it's now a small data centre with a with a 24U rack and all sorts of tech. Eventually, I was able to quit my job and go solo with consulting. The home data centre has been really useful in keeping my cloud costs down. Since much of the infrastructure was already in place and properly setup it pretty much takes care of itself.

Haha. Same here. I started out hosting client backups in my parent's shoe cabinet. 10 years later it's a proper company with dozens of servers. (they now live in real data centers)

Couldn't have done it without fooling around with my home server first. An early ASUS Eee PC in that case.

> My tip to OP would be to set up a home lab on hardware from eBay

Not OP, but this sounds like an interesting thing to do. Which hardware would you recommend?

(Also, for the uninitiated, what exactly is a ‘home lab’?)

Whatever is cheap and common, I'd say. If you plan to fool around with setting up Linux and maybe *BSD servers, you need hardware support for that.

Home lab for me would be a place to build and run servers. I.e. basic tooling to assemble cases, change CPUs and RAM and a wired network connection. A community hacker space may work too and there will be folks to ask for advice.

You could also use micro PCs[1] or a Rasperry Pi, but those are too different from actual servers IMO. That's why I'd prefer a larger form factor to start with.

1: https://www.servethehome.com/introducing-project-tinyminimic...

I would totally recommend a used HP Z230 (either SFF or normal size depending on how big you want it to be). You can put in a Xeon processor, ECC RAM, and you have multiple PCIe slots and SATA ports.

What is what bothers you most about the current situation? Is is that you feel you haven't learned enough for yourself and want to getter a better understanding how other domains work? Or is there are fear that you are missing out on some interesting career opportunities?

If it's the latter than I can tell you with high confidence that you are not missing much there. Most companies need CRUD apps, simple Frontend apps, etc., and there is a high demand for engineers with the relevant skills and therefore also salaries are fine. Building OSes, compilers, embedded devices, networking stacks is super interesting, but there isn't that high of a demand since it's rarely what drives business. Therefore even if the topics are sometimes more complicated, they aren't payed that different.

if you want to break out of the crud trenches, one nice way to do it might be to learn some functional programming. to that end, give "realm of racket"[1] a try - it takes a pedagogical approach, and you might find the earlier chapters a bit trivial, but also you should never feel out of your depth, and you should be able to follow along and have fun while learning some new ideas. after that you should be in better shape to tackle some more serious areas of programming and computer science, but i feel the easy intro to racket will stand you in good stead.

[1] https://nostarch.com/realmofracket.htm

> There is a world out there about compilers, run-times, drivers, emulators, VMs and OSes that I can't seem to grasp, it is just too complicated for me.

Do you want to become a specialist or a generalist? You can dive into one or two of these topics and specialize the rest of your life into it. By the way: a specialist in any of these topics would probably not know any of the fancy buzzwords you've listed in your resume. As a matter of fact, I think you'd have more career opportunities instead.

Keep in mind that the older generation had a "few layers" to explore and could easily understand each slice until the bare metal. These days, it is much more broad and in-depth.

So you want to specialize in something, or become a generalist?

Regardless of where you focus your attention on next, you might want to consider filling in your understanding of really low-level foundations with books like “Code” be Charles Petzold and “The Elements of Computing Systems” by Noam Nisan and Shimon Schocken.

Stop thinking in terms of technology, language, and tools. Learn to think in terms of product. Imagine the end state of something original and awesome, then build it. It’s not about what you know or don’t know, it’s about action and goals.

Sounds like you have breath but not depth. Go deep into the technologies you've mentioned.

To be honest, it sounds like youre disillusioned more with application development and not the particularities of the technologies you work with.


>I see a pattern and I feel the older generation is way more capable and knowledgeable when it comes to Computer Science in general. I am pretty sure a lot of people are or were in the same boat as me.

The older generation are in this place because back then none of the technologies you listed existed and so they had to know the more low level things.

A book that might help some of your knowledge gaps (at least directionally) is The Imposter's Handbook[1] by Rob Conery.

If you're specifically aiming to get away from CRUD take a look at the various Domain Driven Design books by Eric Evans[2] and others. This is the opposite direction to your ask though (more high level than low level).

If you want to go deeper in Java, read Effective Java[3] and Java Concurrency in Practice (JCIP)[4]

Lastly, if you want to experience the reasons why some of that early comp-sci stuff happened the way it did, playing with expensive hardware is the least effective way to find understanding of constraints. Instead consider grabbing devices with significant limitations like an Arduino, ESP32, or similar (I'm partial to the Wio Terminal[5] and M5Stack[6] as interesting easy to get started devices, but others would suggest various Arduinos or Raspberry Pi devices). Realistically you're going to learn much more by coming up with a self-challenging project and completing it than just by learning it for the sake of learning.

[1]: https://bigmachine.io/products/the-imposters-handbook/

[2]: https://www.amazon.com/Domain-Driven-Design-Tackling-Complex...

[3]: https://www.amazon.com/Effective-Java-Joshua-Bloch/dp/013468...

[4]: https://www.amazon.com/Java-Concurrency-Practice-CONCURRENCY...

[5]: https://www.seeedstudio.com/Wio-Terminal-p-4509.html

[6]: https://m5stack.com/

In Rails-land there is a term DHH used (not sure if he coined it) called "conceptual compression". All of these OS, compiler, run-times, emulators, etc. have been conceptually compressed down so far that a typical application writer doesn't have to dive deep into those to solve problems. In fact, in Rails you might not get past the ORM much into actual SQL. You can spend an entire career solving good business problems writing CRUD and related apps.

I would consider myself in older generation, having graduated around when Google was operating in a garage. I took assembly programming and played with circuit boards and logic gates in college. I don't have specialized knowledge of OS writing or compilers but I have enough from a CompSci degree (or maybe it was building gaming rigs in the days of dealing with IRQs) to know if I see issues in those areas. A lot of the concepts in computing haven't changed in ages, yet the buzzwords have.

I would only suggest diving into those areas deeply if you are looking for a job/career change. I say career because working on writing HW drivers or other things is almost a different career than writing business apps since the job opportunities will be different. I've run into folks that got "stuck" into deeply specific areas of computing and have no buzzwords on their resume. They got laid off and are having a hard time finding work.

For someone such as yourself who clearly has some practical ability to write and understand code, Michael Sipser's Introduction to the Theory of Computation is an absolutely marvelous text to fill in gaps in theoretical Computing Science that may be missing from your education.

at the crossroads of everything (machine learning, graphics, constraint optimization, cryptography, simulations) I've wanted to do with coding beyond basic crudapps is linear algebra. I'd recommend learning it well enough to implement the operations in code.

> There is a world out there about compilers, run-times, drivers, emulators, VMs and OSes that I can't seem to grasp, it is just too complicated for me.

If you've already determined that you can't do something because it's just too complicated, there's a strong chance you'll be proven right. In my experience, all of the things we software people deal with down to networking, hardware drivers, task scheduling, you name it, were created and explained by humans. Nature constrains us with time and space performance and quantum mechanics. If another human was smart enough to create this stuff ex nihilo, certainly with some work you can be smart enough to understand it.

> My Computer Science degree and the newer degrees do not focus on these, they have become 4 year long coding boot camps that focus on getting people job ready with AWS, React, Ruby or whatever trendy. This is true for the majority of the not so prestigious universities out there.

No one is going to give you a checklist for how to be creative or pursue your own goals. You have to do it. If you're capable of asking this on HN, I think you're capable of setting a few goals. Just write them down and put the deadline a long ways away.

> I see a pattern and I feel the older generation is way more capable and knowledgeable when it comes to Computer Science in general. I am pretty sure a lot of people are or were in the same boat as me. I don't really have the time(Have wife & kids) to take a compiler course or set out to build my own OS, what would be the easiest way to learn these lower level things?

I'm in my thirties. When I was in school, there were a lot of kids who were just punching the requirements and doing what they were told to get by. Any time they got stuck, they'd have to ask the professor or TA in office hours. Others of us learned that we could solve a lot of problems ourselves if we cut them down to their essences and wrote small programs to see what was actually going on. That heuristic was very helpful.

Once I really did get stuck after doing this, and went to office hours. My professor (an old school Unix graybeard) was so thankful that I had put in the leg work of examining the problem with "toy programs" as he called them that he gave me an exceptionally detailed explanation for what was going on, and how I could show this to myself. He made the comment that a lack of curiosity was a problem among my peers, but that it can be grown and cultivated like other skills.

One other thing the older generations did better than we do is look at the history of our field. If you don't know who Alan Kay, Doug Engelbart, Nikolaus Wirth, Alan Perlis, Ivan Sutherland, Leslie Lamport, Margaret Hamilton or Ted Nelson are, go find some information about them (most have talks or demos on YouTube). Then find out the rest of the missing pantheon of computing. Our industry goes through fads in sort of a fashion culture, and in my experience it leads people to feel burned out our stuck exactly as you described. There are some enduring truths which our field does a poor job of passing onto posterity. Turns out more people had interesting ideas beyond Dijkstra and Turing.

> 1. Reading the Dragon book(I find this would be useful only once you have built a bad compiler) 2. Paying people older than 40-45 to teach me some C/C++ and some tricks of the trade (expensive but I gained a huge amount of knowledge in a short time)

My advice: pick a small, interesting problem and tackle it for some set amount of time each day, maybe after the kids are in bed or your wife is enjoying a bubble bath or whatever. It doesn't have to be long - maybe a half hour to an hour. Just do it consistently.

I've found that GitHub, YouTube and HN itself have loads of interesting project ideas and tutorials. I find those much better than just reading a book. In those cases, the book becomes supplementary material, and is usually most useful at exploring theoretical issues, edge cases, history and other more detailed topics.

If you are going to read, always do the exercises. Also, keep in mind that a lot of the books recommended like the Dragon Book or SICP or whatever else are typically a very thorough treatment, but may not be great for a great read. (There are exceptions to this: I find Tanenbaum's OS books to be highly engaging, interesting and helpful. He is both a practitioner and a theoretician, so I find his perspective valuable.)

Not only is my understanding better with experience (rather than just reading), but I find that I feel accomplished, which is a virtuous cycle. And I also find that I learn enough to be able to discover other interesting things for myself.

> pick a small, interesting problem and tackle it for some set amount of time each day

In the specific area of programming languages, a good example might be to write a toy Forth interpreter.

Whatever interesting technology you take on, I recommend approaching it like a customer driven CRUD project. Define some requirements for it, and then learn enough of the technology to meet the requirements but not much more. That way, you'll be more likely to actually finish the project rather than bite off to much.

Try recreating popular applications & see what it takes to make them live!

As an expert in CRUD - is security a financially viable field for someone who knows nothing about it past SQL and JavaScript injections?

In general - something you find interesting. Ideally on a frontier. E.g., Smart contracts, Web3, dapps.

I learned a lot by running, tinkering with and contributing to open source projects.

How do you find interesting Open source projects?

My progression into open source is documented in an interview (link below), but in short by running Debian, contributing to Debian packages I use and other packages, following Debian mailing list discussions, Debian community blog posts and IRC channels etc. The same probably applies to other FLOSS distros too. For a while I noted down a list of interesting software, but the open source world has exploded since I started that list so I stopped maintaining it. Also reading HN often brings up new open source projects I didn't hear of yet.

https://zgrimshell.github.io/posts/interviews-with-floss-dev... https://bonedaddy.net/pabs3/about/#interviews https://wiki.debian.org/PaulWise/InterestingSoftware

Learn Rust. Pick a project, something native, and build it in Rust.

I wrote a shitty database. That was a great way to learn!

Write a Tetris clone.

> My resume is filled with the latest trendy words like Docker, Scala, Kafka, Spark, Hive, react, K8s or any new framework that seems to make a buzz.

This grab-bag of technologies is kind of all over the place. I see two possible paths forward (these aren't exclusive):

First path: everything you've listed is open source. Pick something, and start digging into the code to figure out what makes it tick. Try to fix a bug from the issue tracker, or review a PR. At some point you'll dig down far enough to encounter a tool or technology (also open source) you're unfamiliar with, and you can learn that and keep digging.

Lather, rinse, repeat.

I've found that most recommendations are to study things bottom up (eg. NAND to Tetris), but for many people it is very difficult to see the point of the lower levels, they are just too many layers of abstraction removed from anything they might find interesting. Top down learning tends to be self-directed, less formal, and ad-hoc, but the motivation for learning is much clearer (ie. you're digging down from the interesting thing made possible by the layer you're digging into), which tends to help a lot. Sure, you're always finding out that the abstractions you know are arbitrary, illusory, or even false in important ways, and in any case always hiding nasty messy truths underneath, but TBH that's how most real learning in our field happens anyway.

Second path, those technologies all have use-cases where they are (supposedly) useful. Start with a CRUD app, and try to figure out when one of those technologies makes sense to incorporate, and explore the edges of those use-cases, their failure modes, when it stops making sense to use them. Most CRUD apps in the wild are, well, pretty cruddy, which means that there is a ready market for skills to get them working again after they've broken down for some reason: too many concurrent users, too high volume of transactions, too large volume of data, too many monitoring alerts, backups take too long, transactions fail, specific queries always time out, inconsistent state, search autocomplete has started freezing the page, newly entered data takes 10m to become visible to users or to another app, hosting costs have started increasing non-linearly, a password reminder takes an hour to arrive in the user's inbox, and on and on.

When it comes right down to it, Twitter is basically a CRUD app. So are ebay, Wikipedia, Amazon, AirBnB, Yelp, Facebook, Reddit, LinkedIn, and so on. And yet not, as a Rails clone of these sites can quickly reveal once you start getting a few thousand users (real, or simulated). Sometimes solving a bottleneck can be done by swapping out a component or scaling horizontally or vertically, other scenarios require more extensive changes. Learning how to recognize and mitigate these inevitable growing pains can lead to some interesting places, and more interesting solutions and work.

Sure, CRUD apps as initially deployed, are boring. But they don't have to (and arguably can't) stay that way, especially as they evolve to encompass more users and data, not to mention functionality (eg. geodata, full-text search, stream processing, notifications, workflow, auditing, PII, security, etc.).

I hope this helps. Good luck on your journey!

I admit I was never in this exact situation, so I'm not sure if what i have to offer can help, but I can relate to the feeling of "something being outside of my capability".

What I don't understand from your post is what your end goal is? Why do you want to learn low-level stuff like building compilers or operating systems? Is it for work? Desire to improve yourself? What can help you better depends on the answer to this question.

I can offer a couple of pointers I hope they can help:

1. I was frustrated by my lack of understating of low-level concepts and how it prevented me from understanding the more complex pattern. The thing that clicked from me was building a small CPU and related hardware (timer, VGA frame-buffer, etc...), I don't remember which site/guide I follow, but today ben eater videos or nand2tetris seems good choices.

I also read a lot about game console programming (GameBoy color in particular), a lot of that low-level concepts were really a lot simpler to grasp with the visual demo to demonstrate. I'm also constantly surprised how much concepts like interrupt, how a mmu works, etc... are useful even when writing high-level code like I'm doing now.

2. Stop considering Databases, compilers, and operating systems as a single unit to learn. They are made of a lot of smaller and simpler components that together form a system. And while when are together seem very complex, often when divided into smaller part became a lot easier to learn even if the total "complexity" remains the same. Take compiler, for example, there is code generation, optimization, and syntax parsing. Instead of trying to build or understand a compiler start by learning the basics of parsing, for example by building a simple chatbot (one of the first things I wrote was an IRC bot), evolve it to understand more complex queries (for example SQL).

Related to this: One thing I could not understand was how big relational databases, like MySQL, worked, a lot of the literature on the topic seem to have a perspective that didn't help me (like the math behind it). The ah-ha moment to me was with sqlite and seeing that at the base was a simple key-value store, and everything else was built on top of this. One takeaway, for me, was that when you feel that you don't understand something, maybe you are asking the wrong question! in my case I keep searching "how relational databases work" instead of what I was really interested in "how relational databases are implemented".

3. I would not start with a thing like "the dragon book", from what I see is an extremely complete book about "everything you need to know about compiler" (hasn't read it, so correct me if I'm wrong). Book like that can seem intimidating and you risk of reading a lot of material without really learning and I feel more like an "information dump". Search for smaller book or course that are an "introduction" to compiler instead of the "everything you need to know", they may not teach you everything, but you will have a better understating of "core" concepts that allow you to tackle more complex topics a lot easier.

Side note: in the specific case of compiler search for a book/course that focuses on building an actual small language! The book my professor gave to me only had a disjointed example for each topic. Something like "crafting interpreter" with a precise goal is MUCH easier to understand while teaching the same concept.

Hope this can help, or at least a starting point for a discussion.

edit: formatting

To me the important thing that turned the lights on regarding relational databases, is to think in terms of sets. If you are thinking in terms of rows and columns you tend to fall into iterative, imperative patterns. Think in terms of sets and frame your thinking about queries in declarative terms of sets, intersections, unions, filters, etc.

You can build a:

- library

- SW combined with the Pi

- find something to mine

- ML prototypes

Implement Scheme.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact