An interesting deep dive with code into how sidekiq job processing in ruby works. Articles like these are a great way of reading (or skimming?) code and absorbing patterns without having to puzzle everything together yourself.
Would also be interesting to compare this to GoodJob or basecamp's Solid Queue which work with relational databases instead of redis (and might be more specialised for use with Rails/ActiveJob).
It works like charm! I love sidekiq. It powers Microsoft’s one of the biggest database management services controlplane. It can easily handle hundreds of thousands of operations with a proper deployment.
There are no blog posts on the controlplane side, so I cannot really share any reading materials with you. Thanks to the NDA, I cannot really share the product’s name as well. However, considering the nature of sidekiq with ruby + postgresql, you might guess which of the products is powered by sidekiq :)
I still believe Rails is one of the best and most productive web frameworks out there and using it for a new project is a no brainer. I hold no special love for Ruby but Rails feels more like a DSL for creating websites than anything else. Sure it's not the fastest stack out there but the vast majority of websites will never outgrown it.
I think a part of the reason why it has become "boring" is that everything's already there and so you don't read every week about a new shiny thing like in the good ol' RailCasts days.
Its kind of ironic that for folks like me that ran screaming away from Java/J2EE to Rails are now seeing questions that assert Ruby/Rails is the new "boring" when it used to be Java. I think this is speaking volumes about the maturity model of the stack and eco-system. For almost any web application you can't go wrong with Rails.
People will knock the performance almost out of habit but there are major internet scale applications out there running this stack. Its horizontally scalable like many other web stacks and I would put developer happiness and ease of use as more important than some theoretical spreadsheet benchmarks.
Many developers just get bored and want to play with something new and shiny. Our industry has spent countless hours reinventing the wheel. I'm not saying there should be one tech stack to rule them all, just that the time spent inventing new frameworks for Resume Driven Development is kinda ridiculous.
I agree, and will add that even though it might seem "boring" to some people, there are plenty of unique fun requirements per project even when using the same stack. 5 different clients have gotten me to start new projects in the past 3 years, and I chose Rails for all of them. Different business domains, different budgets for MVPs (4 figures to 6 figures), sometimes multiple developers and sometimes not, but in all cases we were able to ship product quickly without burning too much of the client's money. Contrast that with some bloated, slow dev on traditional SPA apps I saw during my last stint in enterprise, Rails enables us to move at lightning speed. Seriously, I saw people burn way more money for far less output! Maybe SPA wasn't the only problem and enterprise was, but the point is for that a lean MVP for a wide variety of domains, it's hard to beat Rails if you already know it.
Everything being a DSL is the worst part for me. Given Ruby's lack of static typing, every task becomes "how does this specific DSL work again...?". And when there's a problem it can be non-obvious how to figure out what you did wrong. That said, once you learn the DSL I can totally see how it makes you ultra-productive.
I've been playing around with Ktor lately, and while a lot of it still depends on DSL's, it feels less magical and more like regular code, and I can easily step into the types/source of each DSL.
Just replace 'DSL' with 'API' in your comment and it describes using any large software framework. When you use on a framework to multiply your effort, you inevitably have to learn the ins and outs of the framework. In Ruby/Rails this is accomplished with a lot of class methods, but in other languages you would just be doing the same thing with instance methods or whatever.
Correct, but the key part I mentioned was "Given Ruby's lack of static typing". It's nearly impossible to hunt down source code in large Ruby projects due to mix-ins, dynamically adding methods to types, etc. At least in Ktor I've reasonably able to track down the implementations of most DSLs I've come across since it's statically typed. I tried the same in Rails long ago and gave up.
We chose Rails + Sidekiq for building PlanetScale's REST API + background workers. Project started 3 years ago. We are still very happy with the decision and wouldn't change it.
> Why?
To build fast. Our team knew Ruby well. Many of us worked at GitHub and trusted we could solve the problems ahead of us with it.
So I inherited a 5-year+ Rails app having never touched it before, and having spent two years dragging it from version 4 to 7 I actually get the hype. Everything I want to do for a boring REST app is already built in. Side-jobs, live-updates, ORMs, it's all there.
Would I use it for a new app, yes. But it is very opinionated, and the tools take a while to get working nicely.
We use Rails with some sprinkles of VueJS but we're a team of two and haven't had to hire in a while. Although when we did hire last (a few years ago) it wasn't really a problem finding people.
I would and do use it for new projects as well. The ecosystem is excellent, Rails itself is adding new/useful features all the time. I don't know what would happen if our app scaled to crazy huge traffic loads, I assume we would adapt. Right now we're around 70k monthly active users but a year ago that was double and we didn't really have any performance issues. We've been able to use Sidekiq and Kafka to ingest huge loads of data on the backend without needing to do anything too crazy.
So yeah I don't really understand why Rails has to prove itself over and over again - it works, it keeps improving still after all this time. Its great.
Yes, Rails is a great framework to be very productive in so a great fit for new project.s
Hiring has not been a major problem, having hired for ruby developers and for php developers (in another company). I found that the average level of ruby devs was much higher than php devs (probably has changed with the massive facebook layoffs).
Otherwise Django is quite good too and very similar philosophy to rails and ruby, but I like Ruby just a bit more in term of productivity.
Personally, if it's my own side project, I'm partial to Elixir and Phoenix, but definitely harder to hire for.
Rails is magic, both in good and bad ways. The good generally outweighs the bad. And it has things (like Sidekiq) that I typically need in a project, and I don't have to waste time putting it together and can worry about getting stuff done in my developer role, and code quality in my team lead role. I do use Rust in some performance critical stuff - I used to use C++, but Rust just feels good to write, like Ruby. We have typescript on the front-end, but everything's React and React just never clicked for me in the "this feels natural to write" sense, so I don't venture into that realm often.
We don't have trouble hiring, other than the usual "not enough money in the budget" problems.
If it was a choice between Rails and any of the uber-bloated Node/React SPA monstrosities I've been forced to deal with in recent years, I'd choose Rails in a heartbeat.
Fortunately, I don't have to choose either, because Phoenix exists.
Some customers of mine keep starting projects in Rails. They know it, it works for them and their customers are obviously fine with those services. One of them made the classic switch from Java to Rails on about 2011 and kept creating projects in Rails since then.
While Rails is great (it jump started my software career), I would not use it for new projects. Especially over Elixir/Phoenix which brings Rails like productivity to a much better architecture.
For example, you don’t need sidekiq or Redis as Elixir/Erlang is a concurrent, distributed platform. This simplifies operations and is cheaper to run.
> For example, you don’t need sidekiq or Redis as Elixir/Erlang is a concurrent, distributed platform. This simplifies operations and is cheaper to run.
This is simply wrong. You need a Sidekiq alternative (There is Exq which is protocol compatible with Sidekiq, others like Oban are available as well) because Erlang processes are not durable, nor do you get retry, concurrency control, etc. Everyone loves to claim Erlang is distributed and you can connect them so Redis is useless? but this is rarely utilized in the context of the web server. I have been using Elixir for more than 5+ years now, and I have never seen a valid use case for connecting two nodes.
In reality, the tech stack of the Elixir web app is mostly similar, but it makes life a lot easier if you ever have to deal with any kind of concurrency. Making concurrent requests is as simple as doing a map over a list and they just work, unlike other languages where you have to double-guess whether the library you use is thread-safe.
I chose to build our backend in Ruby (front end in Vuejs) 2 years ago when I started leexi.ai because I wanted to have a quick MVP on the market and we sticked with it because it's so good for what it does!
Rails, Sidekiq, Postgres, Redis, boring technologies powering thousands of customers.
I'd still choose Python/Django as the basis for a startup now. Would probably choose Ruby/Rails if that was where my expertise lied, although I still see lots of Python engineers (in London) and few Ruby engineers.
I think the pattern of a monolith that's very easy to rapidly develop on, plus specialist services around the edge that might be high performance/security/etc and are written in languages like Rust is a pretty good strategy. Personally, having seen the speed of development on a Django project, it's hard to justify much else for the boring CRUD core of a service.
This looks like a web service framework, rather than a web application framework like Rails/Django. Like it's closer to Sinatra/Express than Rails -
The speed of Rails comes with rapid development & strong conventions leaving the important stuff (like business logic and building something) to you.
Once you get to a size where scale is an issue with Rails, you'll have a startup that's working - it's a better problem to have than a startup that has perfect fast code but no customers.
They never implied "en masse." But yes, for what it's worth, those that would use Rails 10 years ago are now increasingly using JS based frameworks, such as NodeJS with express and/or Next.js. In fact, many people who are in web dev these days only know JS/TypeScript, they know zero Ruby at all.
Or maybe: those people that were using Rails 10 years ago are still using Rails now and those people that entered the job market in the last 10 years are using JS frameworks. That would match more closely what I see.
Exactly, that's what I meant, I should've worded it better. I see no reason for most companies who are successful to move off of Rails, but then again, many big ones have over the years [0], including Twitch, Airbnb, Hulu, Coinbase, SoundCloud, and even Twitter.
The point I'm making is that people leaving for nim is a HN thing to say that's unrepresentative of what's actually happening in the rest of the world.
I do and I absolutely plan to use it for any startups I run in the future.
It's the ultimate batteries included framework. It provides high-quality, sane functionality for nearly everything. There are lots of high-quality plugins that drop in without much functionality. And, many companies write Ruby libraries for integrating with their APIs.
I enjoy trying things out on side projects, but almost always find myself spending way too much time trying to figure out how to do something Rails provides out of the box.
Great Functionality:
* Basic REST framework. This part isn't unique, but it works just fine.
* Data modeling with all of the lifecycle and validation stuff you need.
* Database migrations. I'm always amazed by how many modern tools just ignore this part
* Queues and Jobs
* Tests that work out of the box.
* Dev/Test/Production as a 1st class mindset
* Binary/Blob Storage out of the box.
* Email support (though I don't see most people use the built in)
* Pretty darn good security defaults
* View engine (though, I mostly rip it out to write a SPA)
* Caching
Amazing 3rd Party Functionality:
* Drop in Admin panels that mostly just work (ActiveAdmin)
* Devise for authentication. Includes all of the lifecycle basics (forgot password, reset password, email confirmation, etc, etc)
* Several authorization options. Not strictly necessary, but helpful.
* Scheduled/Cron jobs
Complaints:
* Convention over configuration can make it _extremely_ hard to understand or debug things when you don't understand the convention.
* Some people complain about computational speed. It's valid. In my opinion, though if you're at the scale where it matters, you have a really good problem to solve.
* Incompatibilities between gems can be annoying and challenging to resolve.
> configuration file (...) this file will be evaluated before the Rails application or (...) which makes referencing constants defined in the application code impossible
That's not strictly true. You can use them just fine if there's a good reason to do it. For example this erb file works as expected:
<% require 'socket' %>
<%= Socket.gethostname %>
Useful if you have a specific provider for the config values.
It's true that you can write arbitrary code in the template and reference other gems if needed. However, to reference constants located in any directory of your application, such as `app/` or `lib/`, you'd need to either explicitly require these files, overtaking the eager- or auto-loading mechanism of your app framework, or boot the whole application. So, you're right that it's not impossible, but it requires some additional considerations.
Sidekiq is incredible. Been using it for almost a decade and it's just been absolutely rock solid. I put it on the same level as Postgres and Redis is my "GOAT-tier" software list.
When I saw he created a language agnostic sibling "Faktory", I started recommending it to all my non-Ruby colleagues looking for a background job processor.
A bit of an aside, I'm a big fan of how Mike Perham managed to start a business around Sidekiq. I think it really gives us Hacker News folks that there's hope starting a viable bootstrapped business around open-source infrastructure.
I'm sorry but I fail to find how this is relevant.
TSR was an old simple trick in achieving a likeliness of non-preemptive multitasking on an operating system that didn't have any multitasking, but only one single active process.
Sidekiq (or Celery or nearly any task runner system) are designed for multitasking operating systems and they all keep the master process running, spawning/forking worker processes (or threads or whatever primitives they use for concurrency) as needed. They don't use CPU interrupts to stay active (IRQs aren't particularly accessible to user space), as most queue systems don't exactly allow it - typically they have a sleep-poll-repeat cycle. Though if the queue system allows it they may sleep waiting for I/O (which, in turn, indeed, could be triggered by an IRQ), but that's the closest resemblance I can find - and it's quite a stretch.
Would also be interesting to compare this to GoodJob or basecamp's Solid Queue which work with relational databases instead of redis (and might be more specialised for use with Rails/ActiveJob).