It's one of the huge benefits of being both a statically-typed compiled language and offering a modern package system. A lot of languages have the types & compiling but no standard package ecosystem (Haskell, Go, Java), or the package ecosystem but not the compiling (Ruby+Gems, JS+npm, etc). Rust is the only language I can think of that has both, although I'm sure there are others. I'm not sure if there's tooling for compiler writers in other languages to compile them all, though.
And this feature will only get more and more powerful as the ecosystem grows. I think it's a seriously important tool to keep the rust language feeling stable.
Conversely, what stops other languages with source-shipping package managers doing this is that packages don't include tests. One notable exception to this is Perl, where packages can, and typically do, include tests, and these are run as part of the normal installation process. There are even some public-spirited souls who download and test everything that gets uploaded:
This incorporation of tests is both a consequence of and a justification for the fact that Perl inter-package dependencies aren't versioned, so you can never really be sure that you're going to download a combination that works.
I've seen some intense debates over the merits of the two package managers, but ultimately they both have their pros and cons, and are fortunately fairly compatible.
Far as I'm aware, every browser is running some version of the jQuery testsuite nowadays, though I doubt that's the only third-party library tested in such a way in each browser.
Also, Haskell definitely has a standard package system: Hackage. (Stackage is just a "view"/subset of Hackage)
Crater is the older tool that runs builds. CargoBomb is the newer tool that runs builds and tests.
I had a similar lightbulb moment years ago when someone pointed out that y'know, these days it's perfectly feasible to test a simple float32 function by running every possible float32 bitpattern through it.
(Not your point I know)
Conference talk: https://www.youtube.com/watch?v=v7wSqOQeGhA
that's not true! The standard "package" ecosystem in java is maven, and the maven repository!
One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (>40 merges/day) and ~1hr maximum test times.
Given a queue of patches A, B, C, D, start the normal testing against A, but also start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it's still "compatible" with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn't cause any slowdowns versus the fully serialized case.
It looks like bors already supports something like this with "rollups", where simple fixes can be marked for testing together: https://internals.rust-lang.org/t/batched-merge-rollup-featu...
I mean, it's pretty unlikely to happen, but it's possible. Plus it doesn't matter if all you cared about was that the HEAD of master passes tests, but not the whole history.
However, the current approach, if rebasing is used to squash every merge request into a single commit on top of master, allows you to have a linear history of the master branch where every commit is known to pass the whole test suite that was in existence at that moment, which, I think, helps a lot when bisecting.
I'm not sure if Rust compresses merge requests into a single commit on top of master, though.
What we often see is m+A fails because of a flake, and m+A+B+C+D+E passes and merges as a batch. We have a lot of flaky tests. :(
bors' rollup support only lets one mark things for rollup, which show up differently in the queue. It's effectively a priority=-1. This is for things which probably won't break the build, and contributors make a batch rollup every now and then (bors has a button for this). Generally when I do rollups I go through these and ensure they're green on travis.
I've considered giving homu (the engine behind bors) better support for automatically doing this (at least automatically showing travis statuses) but I don't have much time.
So far we haven't been able to stomach doing parallel integration builds because of the doubling of expenses.
As you see, our current solution is rollups, where a human batches low-risk PRs into one big PR. It works just ok, and is super high-maintenance.
In that case, it wouldn't double expenses. Assuming batches are slightly effective, it should reduce expenses-- with >50% success rate with batches of >=3 PRs, you don't have to do individual testing for the later PRs in the batch.
I think it would double expenses because our CI runs at capacity. To do parallel builds we would have to contract for double the compute resources. (With a different purchase structure for our CPU time you may be right about that, but not sure).
Using batch testing as a default would both reduce costs and increase merge speed, assuming that your commits usually pass testing.
The downside of using batching as a default is that it wouldn't test every commit in isolation. That means it wouldn't necessarily be safe to roll back to a particular commit if that commit was tested as part of a batch. E.g. if patches A, B and C were tested together, then it's not certain that patch A by itself would pass the tests.
For example, if the pass rate is 80%, then by batching 3 PRs together there's a 51.2% chance to pass with all three, which would save 2 test runs. and a 48.8% to fail, which would cost 1 extra run. That's a pretty substantial increase in throughput.
What optimization was that? Some searching didn't turn up any relevant results (other than this article).
"This is a known valgrind false positive and Rust already has a lot of suppressions for it. It's reported upstream on the LLVM bug tracker but is not really an LLVM issue, just a limitation of valgrind. LLVM is allowed to generate undefined reads if it can not change the program behavior."
"Just for the record, the undefined read doesn't change the behaviour because it's just a check whether free should not be called, i.e. either it jumps over the free call right away, or it performs the second check."
Hm, Travis CI on GitHub runs tests on a pull request before the merge. Is what they're doing really that unusual?
1. Your PR is tested with the merge against Master. All green!
2. Another PR is tested with the merge against Master. All green!
3. The other PR goes in first. All good!
4. Your PR goes in (merge happens, but introduces a bug). Oh no!
AFAIK, you have to manually merge the targeted branch into the PRs to be in an equivalent situation. Also, other commits might still be added into the targeted branch between the time the tests have been run and the PR is merged.
I'm curious which configuration that is, and how it could be accelerated.
> "env": "RUST_CHECK_TARGET=dist RUST_CONFIGURE_ARGS=\"--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler\" SRC=. DEPLOY=1 RUSTC_RETRY_LINKER_ON_SEGFAULT=1 SCCACHE_ERROR_LOG=/tmp/sccache.log MACOSX_DEPLOYMENT_TARGET=10.7\n",
iOS, it seems.
Looking at the other builds,
* s390x Linux
Seems apple is an overall slowpoke.
Notice how the active OS X builds reaches the limit of 216 jobs and the backlog of OS X jobs increases without decreasing. This is a capacity issue that is likely related to high OS X build costs.
Another thing to notice is that the backlog correlates strongly with time of day in the EST timezone, where the backlog starts to take off after 9am EST.
Their slowest build jobs are all on Darwin/OSX-based systems and (notably) this is a problem numerous other projects have -- slow jobs on OSX build systems. There is a large backlog of OSX build jobs that starts around 9am EST.
Alternatively, given that Mozilla has their own extensive CI infrastructure (for Firefox), perhaps it'd make sense to migrate to that instead of Travis?
> Given that Mozilla has their own extensive CI infrastructure
While Mozilla does sponsor a lot of Rust stuff, it's fundamentally a community project, and so many of us (including myself) would prefer to not to move closer to something that's Mozilla-specific.
Fundamentally the problem with CI testing on Mac is that Apple don't provide the components you want to build a large pool of machines at reasonable cost, so everyone ends up doing slightly mad things involving hundreds of mac minis (obviously the Windows situation also isn't great; this kind of infrastructure loves free software on commodity hardware).
> We are already paying Travis.
Have you heard anything from them about improving the issues with macOS? You've certainly got a case study in it being a problem.
> While Mozilla does sponsor a lot of Rust stuff, it's fundamentally a community project, and so many of us (including myself) would prefer to not to move closer to something that's Mozilla-specific.
Fair enough; I only suggested it because I had the impression that Mozilla's bits were all based on Open Source systems, and it would just be a matter of sharing hosting infrastructure rather than duplicating it.
Yeah, it's not really so much "it's not open source" as it is "Mozilla is the only organization who's really using this stuff". I used to worry about this a lot, but I think we've mostly gotten away from it now that a lot of organizations are using Rust. That is, if Rust was a Mozilla-only technology, I think Rust would have failed its mission. As such, I'm wary of things that seemingly tie Rust back into Mozilla too specifically.
We wanted to pay them (as an OSS project) for faster and more reliable builds for a long time but they always refused.
> The Mac queues for travis-ci.org and travis-ci.com are separate in terms of provisioned capacity, but use the same underlying infrastructure. We make sure we have enough provisioned capacity for our customers on travis-ci.com who are testing private Repositories on our Mac infrastructure. For travis-ci.org we provide a greater capacity, in terms of concurrent VMs, but have had to make some hard decisions on how and when to increase this capacity as the cost for helping the open source community is fairly high.
Tools like bors act as a "force-multiplier".
Practically every business system consists of a view, the logic and the database, plus some webservice and whatnot, but mainly its those 3 parts (the so called 3 layers).
Can you unit test, test the view layer? Not really.
Can you unit test, test the database by doing real test? Again, not really and its not always possible.
So, Unit test is mainly focused in the layer in between of the view and database and both, by their nature, can't be automatically tested.
I like the idea of Unit Test but, for business system, its moot.