
Understanding Kafka with Factorio - DonHopkins
https://hackernoon.com/understanding-kafka-with-factorio-74e8fc9bf181
======
DonHopkins
When I saw this title, I first thought it was about understanding the
cockroach-like aliens in Factorio, but it is actually about queuing and
congestion and parallel processing with conveyor belts and factories!

With the caveat that Factorio is EXTREMELY addictive, properly described as
"programmer crack", here are some other Factorio related discussions:

Factorio – a game where you can automate basically anything (factorio.com)

[https://news.ycombinator.com/item?id=11266471](https://news.ycombinator.com/item?id=11266471)

[http://www.factorio.com/](http://www.factorio.com/)

"This game is like crack for programmers." -kentonv (Tech lead for Cloudflare
Workers, author of Sandstorm.io, Cap'n Proto)

[https://news.ycombinator.com/item?id=11269098](https://news.ycombinator.com/item?id=11269098)

Raycasting engine in Factorio game – Facto-RayO v1.0 [video] (youtube.com)

[https://news.ycombinator.com/item?id=19878688](https://news.ycombinator.com/item?id=19878688)

[https://www.youtube.com/watch?v=7lVAFcDX4eM](https://www.youtube.com/watch?v=7lVAFcDX4eM)

Factorio to talk about horizontal scaling of software services (medium.com)

[https://news.ycombinator.com/item?id=19095271](https://news.ycombinator.com/item?id=19095271)

[https://medium.com/@chrisfauerbach/gamification-of-
horizonta...](https://medium.com/@chrisfauerbach/gamification-of-horizontal-
scaling-4a6c394c9e75)

Mapping Factorio with Leaflet (cloudflare.com)

[https://news.ycombinator.com/item?id=18185058](https://news.ycombinator.com/item?id=18185058)

[https://blog.cloudflare.com/mapping-factorio-with-
leaflet/](https://blog.cloudflare.com/mapping-factorio-with-leaflet/)

~~~
Baeocystin
Pro-tip to others who think 'Programmer Crack' is hyperbole- it is not.

Factorio does have a rather steep learning curve, and the UI is not the most
intuitive. But. Once how it works clicks, it clicks, and suddenly morning. And
I'm in my 40's, I'd thought my days of unexpected overnights while working on
fascinating problems were mostly behind me. Highly recommended, but be aware.

~~~
mabbo
I had to uninstall it last weekend because I came to realize my playing it was
no longer healthy. I mean, that sounds funny, but I'm being totally serious.
I've had a few games like that in my life, and I've come to recognize a few
signs that what I'm doing is not good for me anymore.

But wow, what a great game.

~~~
jjjensen90
I've only had 2 games like that in my life, EverQuest (also called EverCrack)
and Factorio (also called Programmer Crack). I had to go cold turkey or I was
skipping the gym and meals and not sleeping!

~~~
Baeocystin
One thing I credit Factorio for (and I am not joking at all when I say this,
despite how ridiculous the statement might seem) is learning to be a little
more understanding about folks that deal with addictions. I've never had a
problem not doing something to excess, so I never really understood people who
do. But when I first discovered Factorio a few years back, there were enough
times where I didn't do $ActuallyRatherImportantThingX because I was allllmost
done with some aspect of a megabase to give me pause.

There are lots of games I can play for a specified amount of time and be fine
with it. Pretty much all of them, really. Factorio, I get _annoyed_ if pulled
away in the middle of a project. So now I know to not even start unless it's
free time for the rest of the night.

------
gnulinux
For those who don't know factorio is truly a masterpiece of a game. It's
described as "programming the game". Strongly recommended to everyone here.

~~~
gambiting
So I must be the only person on HN who doesn't like it too much. I love the
concept and the factory building - but I found the necessity to move a player
character manually with direct controls to be incredibly detracting from the
process of building factories. I have no idea why they couldn't just include a
mode where you have your typical RTS-like mode where you just go and click on
things that you want to do without playing a game of "will this 5 pixels tall
character fit through this gap between two machines or not". Also, this might
be a personal preference - I found the game's style to be hard to read, it's
far too dark even when played at maximum brightness...it's just an endless sea
of dark brown.

~~~
kingbirdy
By mid-game your actions should be pretty divorced from the Player Character.
Once you have a bot network you can place down items anywhere you can see on
the map (i.e. have Radar coverage), allowing you to build anywhere you want
without having to be nearby. You can even use the bot network to expand the
bot network & radar coverage.

~~~
PetitPrince
Additionally, maintaining the stock of material for your bot adds yet another
fascinating layer of complexity to the game.

------
pul
I wrote this article a couple of months ago and didn't expect it to appear on
my news feed all of the sudden. Thanks for the love everyone. Any questions
are welcome of course!

~~~
Baeocystin
How do you like the .17 update? :D

More seriously, I am curious about what people who haven't played Factorio
think of the article. Since I've sunk $bignum hours in to it, I found your
explanations clear, but I'd like to hear what those who haven't think.

~~~
IggleSniggle
I haven’t played Factorio, and have just barely used Kafka. Therefore, if
there was any subtlety, it was lost on me. Nonetheless, this article was
clear. I thought this cute and I liked it, but I’m not sure that I gained any
new insights into Kafka. This is not a criticism!

However, one of the main advantages of Kafka, I thought, was that it’s so much
more than a queue, since data is replayable, etc. The article does mention
this as a shortcoming of the analogy, but isn’t this one of the main selling
points vs other message queues?

------
chrischen
The analogy helped me understand, but in factorio belts are required primarily
because everything takes physical space. What are real world use cases for
kafka? Is it primarily if your data is too much to simply store in a db?

~~~
pdpi
Push architectures put all of the control on the producer, but become quite
painful in the face of an unreliable consumer. Pull architectures put all of
the control on the consumer, but become quite painful in the face of an
unreliable producer.

Message queues allow the producer to think it's participating in a push
architecture, while the consumer believes it's a pull-based arch. Because the
message queue itself does _very_ little work, it gets to play a very well-
behaved consumer insofar as the producer is concerned, while behaving like a
well-behaved producer for the consumer. The queue's buffering then allows both
the producer and the consumer to misbehave to some degree while the queue
itself keeps the illusion of good behaviour.

Kafka specifically falls under the category of log-oriented message queues,
which are eminently useful for distributing any workload that looks like "tail
a log and process each line as it comes in" across a large number of nodes.

~~~
pul
This. With the slight nuance that it works better if you don't think of it as
queues, since that suggests messages being removed when processed.

The big win when introducing Kafka is that (apart from the schema) it
completely decouples producers and consumers of messages. That in turn reduces
the required coordination between independent teams within an organisation.

------
xmonkee
That's crazy! I installed it yesterday for the first time to try out the steam
proton thing and was literally up till 3 am. I didn't realise the programmer
crack angle till I read this. I'm screwed.

~~~
Itsdijital
You're screwed.

Also press Alt

And use "Q" while hovering over an object to put it in hand (if it's in your
inventory). Press "Q" again to drop whats in hand. I played waaayyy to long
without knowing this and it is unbelievably useful, saves mountains of time
digging through your inventory.

~~~
Baeocystin
Also, Z to drop something on the ground if you find your bots a little to
assertive about filling your inventory back up and you find yourself stuck. :D

------
amdelamar
I think the analogy works better with queues and distributed services in
general, rather than just Kafka.

------
Lowkeyloki
You lost me at the part about partitions. It might help if those gifs were
labeled.

~~~
pul
Yeah, I started out labeling them, but that turned out to be extremely
tedious. Does anyone know of tools that make annotating gifs easy?

------
fragmede
There's a similar web game -
[http://www.factoryidle.com](http://www.factoryidle.com), for those who
can't/won't play FactorIO, with a similarly addictive quality.

~~~
allannienhuis
I'm getting 403 bad request responses from cloudfront on that url

------
realityman
A better way to understand Kafka is to fill a room with rakes, turn of the
lights, and walk around.

~~~
jpk
This sounds like an idea founded on experience, but without including any
details about how you arrived at this sentiment the comment doesn't bring any
value to the thread.

~~~
breischl
Not GP, but early versions of Kafka had a lot of poorly-chosen default
settings and sharp edges. eg, it was tuned for speed and availability over
reliability, so it could lose your data if you weren't careful (not dissimilar
to early versions of MongoDB that way). It was possible/easy to set up topics
with no replication, so you'd lose data on your first server outage. KStreams
had no real error handling - it would just explode and not even really tell
you why. I wasn't as close to the operational side, but there were a lot of
problems there too.

IME it has gotten better over time, but it's still pretty easy to shoot
yourself in the foot if you don't understand what you're doing with it.

------
rs23296008n1
Isn't this a repost?

~~~
DonHopkins
Dang suggested reposting it since it didn't get much attention the first time.

~~~
rs23296008n1
Wow, I ask a valid question about a old post and I'm punished with multiple
downvotes. Lesson learnt.

~~~
NullPrefix
You were downvoted because you did not RTF FAQ.

~~~
rs23296008n1
But you're happy to abuse me. So that is acceptable then.

~~~
NullPrefix
How did you came to the conclusion that I am abusing you by pointing out that
you did not read the FAQ?

My message was not even unsolicited. You were asking why and I gave you an
answer.

