On small teams or projects, this attitude is probably fine.
One of the most frustrating experiences as an engineer is running up against a deadline and just staring at my CI with my finger on the re-run button because my coworkers can't be bothered to believe me when I tell them their tests are flaky.
Fixing your flaky tests is like putting your cart away - a little effort on your part makes a lot of peoples lives a little better.
I didnt need to use bazel, I like bazel and want to learn more about it.
I also have a small, but burning, passion for reproducible builds, distributed compilation and distributed caching.
Being able to build an entire OS and essentially anything I want on top in a reproducible and relatively organic way (with incremental compilation) is pretty dope.
You sound like the perfect Nix cult memb… erm, user. It’s everything you describe and more (plus the language is incredibly powerful compared with starlark).
But you speak from sufficient experience that I presume Nix is a “been there, done that” thing for you. What gives?
Nix can be used as a build system in the same way that bazel can. It already has all of the tooling - a fundamental representation of a hermetic DAG, caching, access to any tool you need, and a vast selection of libraries.
The only catch is that no one has used it to write a build system for it in public yet. I’ve seen it done in a couple of companies, though, as using Nix to only partially manage builds can be awkward due to caching loss (if your unit of source is the entire source tree, a tiny change is an entirely new source).
Nix can do it incremental
U could split it into multiple derivations which get built into one package
For rust there ist the excellent https://crane.dev/index.html project
Or you can also go to the extreme and do 1:1
source to derivation mapping
So for example if ur project has 100 source files it could be built from 100 derivations, the language/CLI tools are flexible enough for that
Don't know tho if there any well working smart nix tools which can make it well working /efficient, in theory it's very possible, just unsure about practicality/overheads
Nix is basically merely a quirky functional programming language that generates shell scripts to be run in a sandbox for the actual build. It is not a great tool for within-a-project building; its minimal unit of work has a pretty high overhead.
Decentralised caching, absolutely - unless I’m misunderstanding what you mean there. You can build across many machines, merge stores, host caches online with cachix (or your own approach), etc. I make fairly heavy use of that, otherwise my CI builds would be brutal.
Memorizing isn’t a term I’m familiar with in this context.
I am interested in making a system that can memoize large databases from ETL systems and then serve that on iroh or ipfs/torrent, such that a process that may take a supercomputer a week to process can have the same code run on a laptop and it will notice it's been done my a university supercomputer before already and grab that result automatically from the decentralized network of all people using the software (who downloaded the ETL database).
Derivations are just a set of instructions combined with a set of inputs, and a unique hash is made from that.
If you make a derivation whose result is the invocation of another, and you try and grab the outcome from that derivation, here’s what will happen:
- it will generate the hash
- it will look that hash up in your local /nix/store
- if not found it will look that hash up in any remote caches you have configured
- if not found it will create it using the inputs and instructions
This is transitive so any missing inputs will also be searched for and built if missing, etc.
So if the outcome from your process is something you want to keep and make accessible to other machines, you can do that.
If the machines differ in architecture, the “inputs” might differ between machines (e.g. clang on Mac silicon is not the same as clang on x86-64) and that would result in a different final hash, thus one computation per unique architecture.
This is ultimately the correct behaviour as guaranteeing identical output on different architectures is somewhat unrealistic.
I see. Perhaps the added benefit I am trying to create with this other system is that specifying remote locations isn't necessary, and is just inherited as the distributed network. Anytime anyone runs it, they're added to the network, so it scales with the number of users.
You should check out the ChromeOS Bazelification project[1]. It has those exact same goals. Not all packages are reproducible though because they embed timestamps.
Yeah I think the title is a little hyperbolic - the author is really saying "intermediate commits are useless, make sure you squash commit when you merge to main and have a good message there"
We are a small company but our autoscaling cluster for GitHub actions on aws will scale up to >500vcpus during the work day when there are a lot of prs going in.
I don't see it documented anywhere, what are your concurrency limits on accounts?
Looks like in theory it works similar to other solutions I've seen, although those rarely get the polish I see in the video (since they're internal tools).
What's interesting is how many engineering teams put up with low-polish tools. It's not just the pretty-factor. Most in-house tools cannot justify the expense on building them to product quality - which means engineers suffer along.
Github's "require branch to be up to date" setting helps, but doesn't really scale beyond a few engineers in one repo. We tried this where I worked, but after ~6 people, rebasing feature branches got way too annoying.
Hey congrats on the launch, clearly a lot of thought and effort went into this.
I'm pretty new to this space, and maybe this is a dumb question but how does this differ from Mixpanel? Would I use this for something different?
Mixpanel will store the data for you and do visualization. Jitsu just help you to get you data to your data warehouse.
Downside: you'll need to build all visualization by yourself. Fortunately that's easy with tools such as Looker, Mode, Metabase etc
Upside: you can do with your data whatever you want - built any reports, join with other datasets etc. You not limited by reports MixPanel team build.
In reality, Jitsu and MixPanel could co-exist. Jitsu support s MixPanel as a destination (e.g. you send data to Jitsu ; Jitsu sends it to MixPanel and data warehouse).
We are currently looking at Segment, Jitsu and others. While we generally liked Jitsu, this was kind of a big deal for us and made us lean towards Segment.
Mixpanel employee here, I actually just emailed Jitsu this morning to ask about adding Mixpanel as a destination. We're willing to work with them (or any of you!) to get the PR open. Feel free to reach out directly to josh@mixpanel.com!
Have you ever worked on something interesting or meaningful at a job? You list a bunch of random topics, none of which are inherently interesting or meaningful. I've worked on flashy projects - most recently self driving at one of the bigger companies doing it. Guess what? In an eng org of hundreds, at least half the people were still doing shit work that had nothing to do with self driving.
Beyond that, if I want to hire you for interesting work, it falls into two buckets: mission critical, or irrelevant. I can't hire you for mission critical work because I have no confidence you can do the job, so I can only hire you for irrelevant work. If I hire you for irrelevant work, it makes me no money, so I pay you no money. Why even have this arrangement? Just go start your own side project - at least then you own it.
Consider this: join a company doing something you don't like (making an absurd amount of money because of the industry), demonstrate/develop your expertise, identify the parts of the company that are interesting to you, then go work on those. People doing interesting satisfying work didn't just get there accidentally.
One of the most frustrating experiences as an engineer is running up against a deadline and just staring at my CI with my finger on the re-run button because my coworkers can't be bothered to believe me when I tell them their tests are flaky.
Fixing your flaky tests is like putting your cart away - a little effort on your part makes a lot of peoples lives a little better.
reply