> Why would you recommend that? [..] It's a bunch of work for no benefit.
nanobind/pybind11 (co-)author here. The space of python bindings is extremely diverse and on the whole probably looks very different from your use case. nanobind/pybind11 target the 'really fancy' case you mention specifically for codebases that are "at home" in C++, but which want natural Pythonic bindings. There is near-zero overlap with Cython.
Yes, I assumed everyone who cares about performance (or who writes large programs) is also a C++ or CUDA programmer. Don't tell me that is not the case :-)
IINA is basically a front-end for mpv. I use it. It's got nice features like force-touch support, but in my experience it's a memory hog—way worse than base mpv somehow.
Caution: if you rely on pybind11 or a project using pybind11 (many projects do, like NumPy/SciPy/Tensorflow/PyTorch..), hold off on upgrading to Python 3.9.0 for now.
A change in Python 3.9.0 introduces undefined behavior in combination with pybind11 (rarely occurring crashes, but could be arbitrarily bad). We will work around it in an upcoming version of pybind11, and Python will separately also fix this problem in 3.9.1 slated for release in December.
I was about to ask why would they do a release if they know it's broken and blowing up their flagship libraries? It's worrying when the interpreter starts to favor shipping quickly over being functional.
But nevermind, the 3.9 release was 15 days ago and the bug was identified 8 days ago. I guess there has to be a release before people will start using it and find bugs/regressions.
There's a saying to always wait for the .1 release, this is a good illustration of why.
I don't know if I would call the SciPy ecosystem the flagship libraries of python. I know it is very popular,and probably in the top 5 use cases, and I use some of them everyday. but in my experience they've always been off in their own world in relation to the rest of the python community. Most of the people I've met that use this stuff are either scientists or finance people that are just using it as another tool along with matlab, R, and excel. Which is what they should be doing, but in their minds they're not writing python code they're using Pandas, Jupyter, and Matplotlib.
But yes, never ever jump on a new release of anything, at least in production. Always stand back and wait for someone more optimistic to find the bugs
> I don't know if I would call the SciPy ecosystem the flagship libraries of python.
I mean, sure NumPy is integral to python finance and scientific computing...but it's also right at the heart of the popular Python roguelike tutorial, too.
There's Python code that doesn't use NumPy, sure (I've got some in production right now), but I can't personally think of a Python library with nearly as deep dependencies across nearly as diverse a range of use cases.
What makes pybind a "flagship library"? If it was very widely used this bug probably would've been caught by people testing the alpha/beta/rc versions of 3.9.
People using numpy are not in the fringe. If this wasn’t triaged during prereleases then it’s either that it doesn’t always cause problems, or there is an issue in the diversity of testers.
I personally don't like Python (I prefer Julia/Ruby/Rust), but all the new advances in machine learning are implemented in Tensorflow and PyTorch, so I came back to Python just because of these libraries and infrastructure on top of them.
I was just using the expit function from scipy yesterday by the way, because that's where it was implemented.
You want more fanout to use the entire cache line or even the entire virtual memory page. RB trees assume access to any address in memory has equal cost, which is extremely incorrect.
The red-black tree implementation typically used in *BSD development, <sys/tree.h>, is an intrusive data structure; tree nodes are embedded within the object you must dereference for key comparison. There's no need to optimize fanout because you've entirely removed that extra indirection.
As a general purpose data structure implementation for systems development, <sys/tree.h> is quite nice. However, for any particular task you can always optimize the data structure, or choose an entirely different data structure, to better fit the problem. Which is perhaps why the Linux kernel has so many different tree and hash implementations.
Indeed. C++ needs some better Standard containers. As it is, you need 3rd party library containers when you need optimal performance on ordered collections. Abseil and boost have alternatives.
The great advantage C++ has is that there is no temptation to open-code them. Library implementations can absorb immense optimization and testing effort, amortized over all uses, and delivered without compromise.
If you're not interleaving insertions and lookups, then a sorted vector can be really good.
If you need to interleave insertions and lookups, but don't need to traverse in sorted order, then a good hash table (not std::unordered_map) is normally the best option.
You only need a tree if you need traversal in sorted order and interleaved insertions and lookups, which is pretty uncommon. Even then you are almost always better off with a B-tree than a red-black tree.
Even with interleaved insertions and lookups, there are common scenarios that make a sorted vector still significantly more performant. I wrote about it when we published our open source SortedList<T> implementation for .NET and .NET Core [0], specifically comparing it to AVL and Red-Black trees.
Seems like these criteria miss the use case for mm_rb, one of the central red-black trees in linux.
If you need to be able to store intervals (i.e. virtual memory areas) and do lookups based on any address in the interval, not just the base address, I don't think a hash map is the best option.
IMO "lookups based on any address in the interval" requires "traversal in sorted order", although I could have probably been more precise in my terms.
I would be curious if anyone has ever profiled the impact of changing mm_rb to a B-tree. It might be very difficult if existing code that uses mm_rb depends on pointer stability, though.
Aren't splay trees terrible because they turn all reads into writes? (Which means the cache lines bounce between readers, instead of being shared as they would be with pure reads.)
Yeah, having reads rebalancing the tree in a multithreaded subsystem is probably not optimal. Might be that the Splay-trees have outstayed their welcome :)
FreeBSD’s tree.h used to have, iirc, both RB- and Splay-trees.
The extra-painful part about this whole situation is that it's apparently only due to Apple blocking NVIDIA from signing and publishing their drivers. (for whatever bizarre reason they may have to do that)
I know it stinks for the people who are acustom to Nvidia's workflow, but it is quite hard to feel bad for Nvidia. A bigger fish is doing to them what they do to smaller players.
I think there’s more to the story than Nvidia is letting on. It wouldn’t surprise me a single bit if Apple is blocking Nvidia due to something like QA issues — Nvidia’s web drivers are notoriously buggy under macOS and Nvidia seems completely unrepentant about it. Pair this with Nvidia’s unwillingness to share source and collaborate with other companies on development and you’ve got problems.
I currently run an EVGA 980Ti Classified in my hackintosh. It’s great hardware, but with the frequency of glitches that the drivers bring I’m increasingly inclined to sell it and replace it with an AMD card.
I've long been really excited about SCons but eventually decided to move away because it became unbearably sluggish for a large-ish codebase with >180K lines of C++ code split into many files. Another issue are cross-platform builds. SCons breaks every time there is a new version of Visual Studio, and it takes many months until an updated version restores compatibility.
That blog post is an impressive logical leap on the authors part.
I realise that the post is not from recently but the benchmarks are on seriously outdated software and hardwaee, those specs are over a decade old at this stage. His conclusions are also misleading, all of his graphs show that make is 2-10x faster than scons, and that make scales better than scons.he seems hell bent on proving there's no quadratic complexity, despite there being an order of magnitude of a difference!
On the other hand, I don't think it's reasonable to assume every build system will immediately support every compiler/IDE. At the end of the day, scons is an open source project and if that's the only issue stopping you using it, I'm sure they'd be happy to accept patches providing support rather than wait months.
As is cmake. Make is about to hit a mid life crisis, other projects like Ninja are faster and more lightweight, projects like WAF/Premake use a scripting language instead of a DSL, others liek FASTbuild claim to support distrobuted builds. At a certain point, sofradre has to be accepted for what it is, not what it claims to be.
I agree it's a peculiar system to use for benchmarking in 2018, but for comparison purposes it shouldn't matter much as long as the same software/hardware are used for both Make and SCons.
There's definitely some mental gymnastics and careful massaging going on to hide the performance problem. Would be a great example for a "How to lie with charts and graphs" article.
I think it is still invalid to consider for a benchmark. We don't consider a car track based on how quickly 10 year old cars can navigate it.
In the last decade we have gone from everyone needs an antivirus to Windows 10 is good enough for most people. We've had Spectre and meltdown, SSDs have become viable and been replaced by nvmE SSDs we've had a decade of OS development, filesystem development and hardware improvements.weve also had huge improvements in build ststems, Ninja was only relewaeed in 2012!
And yet, even after all of that, the TLDR from the article is "scons is an order of magnitude slower than make, but my hardware was the bottleneck before i figured it out scaled quadratically, therefore scons is not slow".
I don't think visual studio had ever claimed to have a stable API. 2019/2017/2015 have been good, but there was some definitely teething issues migrating to their current situation. That doesn't necessitate months of waiting - there is nothing stopping me or you adding support for an unstable API
Try WAF. It was created about twelve years ago as a fork of SCons because the author was dissatisfied with its performance problems. WAF is in my opinion a very capable build and configuration system but the main drawback is its small community. The person developing WAF isn't keen on "marketing" it.
The only place I have encountered WAF was at Bloomberg, where it was used to build the C++ foundation library (BDE) that Bloomberg uses in older projects in place of Boost and Std, both of which it predates.
WAF worked, and didn't seem especially slow, unlike Scons, but I frequently needed to code Python to make it do what seemed like pretty ordinary things.
Personally I see Meson as an attempt to do CMake right. A big differences between Meson and SCons is that SCons handles the execution of the build graph, while Meson delegates execution to Ninja. The nice thing about the former is that while the graph is being walked, new nodes can be added. The nice thing about the latter is that it's incredibly efficient.
Why not both? :-) While the architecture is more similar to CMake, Meson's language reminds me of SCons---but it is a custom DSL and not Python, which makes things less surprising. For example in SCons a single-element list can be replaced with with the element itself, but that is a feature of SCons's builtins: if you write custom Python code, you can have confusing results when you place a string in a place where you must use a list of strings.
Also SCons is Turing complete and this makes it a bit of a stretch to call the language declarative; Meson keeps simple loops but is not Turing complete and the maintainers are okay with plans/pull requests that make Meson even more declarative. In Meson generally "there is only one way to do it", in SCons much less so which is ironic given it uses Python as the language.
All these systems annoy me with their dependencies... the only one that i kinda do not mind much is premake because it consists of a single self-contained executable. However even with that you need to have premake available. I'd prefer it if these programs took a page out of Autotools and created configuration scripts that only relied on whatever the target OS has available (plus the compiler) without having the need to install a separate build system since after a while you end up with a bunch of different build systems since everyone prefers theirs.
Meson's only dependency is Python, with no external packages needed---only the standard library.
The only annoying part of Meson is that it only supports a few compilers (GCC, clang, ICC, MSVC), and requires porting to new compilers unlike Autoconf.
It takes it 30s, on a decent size codebase, just to tell you that there is "Nothing to do." There is order of magnitude difference between it and make. It's a complete travesty!
Lack of NVIDIA support is a deal-breaker. The AMD ecosystem is just so far behind when it comes to frameworks like CUDA, OptiX, cuDNN, etc.. Why can't Apple open up kernel-level support by cooperating more with NVIDIA? This state of things seems completely bizarre to me.
I know this is an Apple/Mac discussion, but from a Linux/OSS perspective, it looks very different. If it weren't for CUDA, Tensorflow and so on, I wouldn't even consider buying NVIDIA (anymore).
Currently, I have a laptop with NVIDIA graphics and a desktop PC with AMD graphics. The AMD stuff just works; out of the box (OSS, good performance, happy user). But NVIDIA either comes with nouveau (OSS, poor performance) or the nvidia proprietary binary driver which has all sorts of weird issues (e.g. always-on fans, animations running at different speeds, etc.).
Sure, this doesn't say anything about how a Mac would run with NVIDIA drivers, but it gives a hint that NVIDIA has its own weak spots. But the reason why Apple decided to skip on NVIDIA in this case, is probably rooted deeper within their strategies.
There was an anecdote on a semi-recent (sometime this year) episode of ATP where an inside source related to them that nVidia were a terrible organization to try and work with. They apparently screwed Apple over, and Apple has a long memory for that sort of thing.
The AMD ecosystem will always remain behind if application developers continue to choose single-vendor proprietary frameworks rather than standard APIs.
There was some sleight of hand in the benchmarking for me too, they compared GPU compute rendering engine speeds to "Nvidia Quadros" when no one doing CUDA rendering on Windows uses Quadro cards, they all use GTX because the price to performance is ridiculous on Octane render/Redshift/etc.
The material database provides broad coverage of common and more specialized materials, including isotropic and anisotropic BRDFs of metals, paper, car paints, organic samples, fabrics, etc. Each material is available in a spectral version that covers the 360(UV) - 1000nm (NIR) range with ~4nm sample spacing, as well as a RGB version for compatibility with renderers that do not support color spectra. The representation is extremely compact, requiring approximately 16KiB and 544KiB per channel for isotropic and anisotropic materials, respectively. Furthermore, it provides a natural importance sampling operation that does not require numerical fits or complex additional data structures. Technical details on the underlying parameterization and measurement methodology are available in the paper
nanobind/pybind11 (co-)author here. The space of python bindings is extremely diverse and on the whole probably looks very different from your use case. nanobind/pybind11 target the 'really fancy' case you mention specifically for codebases that are "at home" in C++, but which want natural Pythonic bindings. There is near-zero overlap with Cython.