CrossCode was written in JS/HTML5 and was ported over to consoles just fine, though.
They mention stuff like "an interpreter which translates the [JavaScript] code but locks it up in a cage"¹, their presentation² mention JS interpreters and a JS AoT compiler, so I'm not really sure how they did it
> it is allocating 2 IC nodes for each numeric operation, while Python is not
While that's true, Python would be using big integers (PyLongObject) for most of the computations, meaning every number gets allocated on the heap.
If we use a Python implementation that would avoid this, like PyPy or Cython, the results change significantly:
% cat sum.py
def sum(depth, x):
if depth == 0:
return x
else:
fst = sum(depth-1, x*2+0) # adds the fst half
snd = sum(depth-1, x*2+1) # adds the snd half
return fst + snd
if __name__ == '__main__':
print(sum(30, 0))
% time pypy sum.py
576460751766552576
pypy sum.py 4.26s user 0.06s system 96% cpu 4.464 total
That's on an M2 Pro. I also imagine the result in Bend would not be correct since it only supports 24 bit integers, meaning it'd overflow quite quickly when summing up to 2^30, is that right?
[Edit: just noticed the previous comment had already mentioned pypy]
> I'm aware it is 2x slower on non-Apple CPUs.
Do you know why? As far as I can tell, HVM has no aarch64/Apple-specific code. Could it be because Apple Silicon has wider decode blocks?
> can be underwhelming, and I understand if you don't believe on my words
I don't think anyone wants to rain on your parade, but extraordinary claims require extraordinary evidence.
The work you've done in Bend and HVM sounds impressive, but I feel the benchmarks need more evaluation/scrutiny. Since your main competitor would be Mojo and not Python, comparisons to Mojo would be nice as well.
The only claim I made is that it scales linearly with cores. Nothing else!
I'm personally putting a LOT of effort to make our claims as accurate and truthful as possible, in every single place. Documentation, website, demos. I spent hours in meetings to make sure everything is correct. Yet, sometimes it feels that no matter how much effort I put, people will just find ways to misinterpret it.
We published the real benchmarks, checked and double checked. And then you complained some benchmarks are not so good. Which we acknowledged, and provided causes, and how we plan to address them. And then you said the benchmarks need more evaluation? How does that make sense in the context of them being underwhelming?
We're not going to compare to Mojo or other languages, specifically because it generates hate.
Our only claim is:
HVM2 is the first version of our Interaction Combinator evaluator that runs with linear speedup on GPUs. Running closures on GPUs required colossal amount of correctness work, and we're reporting this milestone. Moreover, we finally managed to compile a Python-like language to it. That is all that is being claimed, and nothing else. The codegen is still abysmal and single-core performance is bad - that's our next focus. If anything else was claimed, it wasn't us!
> I spent hours in meetings to make sure everything is correct. Yet, sometimes it feels that no matter how much effort I put, people will just find ways to misinterpret it.
from reply below:
> I apologize if I got defensive, it is just that I put so much effort on being truthful, double-checking, putting disclaimers everywhere about every possible misinterpretation.
I just want to say: don't stop. There will always be some people who don't notice or acknowledge the effort to be precise and truthful. But others will. For me, this attitude elevates the project to something I will be watching.
That's true, you never mentioned Python or alternatives in your README, I guess I got Mandela'ed from the comments in Hacker News, so my bad on that.
People are naturally going to compare the timings and function you cite to what's available to the community right now, though, that's the only way we can picture its performance in real-life tasks.
> Mojo or other languages, specifically because it generates hate
Mojo launched comparing itself to Python and didn't generate much hate, it seems, but I digress
In any case, I hope Bend and HVM can continue to improve even further, it's always nice to see projects like those, specially from another Brazilian
Thanks, and I apologize if I got defensive, it is just that I put so much effort on being truthful, double-checking, putting disclaimers everywhere about every possible misinterpretation. Hell this is behind install instructions:
> our code gen is still on its infancy, and is nowhere as mature as SOTA compilers like GCC and GHC
Yet people still misinterpret. It is frustrating because I don't know what I could've done better
Don't worry about it. Keep at it, this is a very cool project.
FWIW on HN people are inherently going to try to actually use your project and so if it's meant to be (long term) a faster way to run X people evaluate it against that implicit benchmark.
Don't optimize for minimum hate, optimize for actionable feedback and ignore the haters. Easier said than done, though.
Remember you don't need comment trolls on your team, and you'll go insane taking them seriously. Focus on piquing the interest of motivated language nerds. I personally would have really appreciated a "look, were still 10x (or whatever) slower than Python, so now I need all the help I can get working on the codegen, etc." This would have given me quick perspective on why this milestone is meaningful.
Just a note: we are NOT 10x slower than Python. I think a lot of people got the wrong message from this thread. HVM is actually quite fast already. It is just that, on this specific program, Python was doing no allocations, while HVM was allocating a lot.
If you compare programs that do the same allocation, HVM already outperforms not just Python but even compiled languages like Haskell/GHC, due to using all cores. See the Bitonic Sort example. And that's extremely relevant, because real world programs in high-level languages allocate a lot!
I think I made a huge mistake of using a "sum" example on the GUIDE, since it is actually one of the few specific programs where HVM will do poorly today.
I think the (hidden) reasoning is that it is really easy to have speedups with slow interpreters. However, getting speedups in high-performance level programs is quite hard, mainly due to micro-optimisations.
That's where the comparison to Python comes from: getting speedup on slow interpreters is not very _relevant_. Now if your interpreter has the same optimisations as Python (or v8 or JVM), even a small fraction of what you show would be impressive.
Having said this, the work your team did is a really challenging engineering feat (and with lot more potential). But I do not believe the current speedups will hold if the interpreter/compilers have the level of optimisation that exist in other languages. And while you do not claim it, people expect that.
Perhaps consider moving the warning in the NOTE at the bottom of the README.md to a DISCLAIMER section near the top.
I read the whole thing first, then commented, but people often read half of such a document, assume they've got all the important bits, and dive straight in.
(we used to have that problem at $work with new team members and our onboarding doc; I added a section at the bottom that was pure silliness, and then started asking people who claimed to have read it a question that would only make sense if they'd seen the joke ... generally followed by telling them to go back and finish reading and not to try that with me again ;)
Perhaps you can add: "The codegen is still abysmal and single-core performance is bad - that's our next focus." as a disclaimer on the main page/videos/etc. This provides more context about what you claim and also very important what you don't (yet) claim.
> It is very important to reinforce that, while Bend does what it was built to (i.e., scale in performance with cores, up to 10000+ concurrent threads), its single-core performance is still extremely sub-par. This is the first version of the system, and we haven't put much effort into a proper compiler yet. You can expect the raw performance to substantially improve on every release, as we work towards a proper codegen (including a constellation of missing optimizations).
which seems to be pretty much exactly that?
It's at the bottom, though, so I can imagine people just skimming for "how do I get started" missing it, and making it more obvious would almost certainly be a Good Thing.
I still feel like reading the whole (not particularly long) README before commenting being angry about it (not you) is something one could reasonably think the HN commentariat would be capable of (if you want to comment -without- reading the fine article, there's slashdot for that ;), but I'm also the sort of person who reads a whole man page when encountering a new command so perhaps I'm typical minding there.
> I'm personally putting a LOT of effort to make our claims as accurate and truthful as possible, in every single place.
I'm not informed enough to comment on the performance but I really like this attitude of not overselling your product but still claiming that you reached a milestone. That's a fine balance to strike and some people will misunderstand because we just do not assume that much nuance – and especially not truth – from marketing statements.
Identifying what's parallelizable is valuable in the world of language theory, but pure functional languages are as trivial as it gets, so that research isn't exactly ground-breaking.
And you're just not fast enough for anyone doing HPC, where the problem is not identifying what can be parallelized, but figuring out to make the most of the hardware, i.e. the codegen.
This approach is valuable because it abstracts away certain complexities for the user, allowing them to focus on the code itself. I found it especially beneficial for users who are not willing to learn functional languages or parallelize code in imperative languages. HPC specialists might not be the current target audience, and code generation can always be improved over time, and I trust based on the dev comments that it will be.
Naive question: do you expect the linear scaling to hold with those optimisations to single core performance, or would performance diverge from linear there pending further research advancements?
I think you were being absolutely precise, but I want to give a tiny bit of constructive feedback anyway:
In my experience, to not be misunderstood it is more important to understand the state of mind/frame of reference of your audience, than to be utterly precise.
The problem is, if you have been working on something for a while, it is extremely hard to understand how the world looks to someone who has never seen it (1).
The second problem is that when you hit a site like hacker News your audience is impossibly diverse, and there isn't any one state of mind.
When I present research, it always takes many iterations of reflecting on both points to get to a good place.
I've always taken 'Welcome to the Future' as the thing being presented is futuristic and exists now in the present. Not 'in the future we will welcome you to the future' - while that is a nice sentiment it's utterly useless. To point out the obvious - of course futuristic things exist in the future and of course I have to wait for the future to happen.
I think people might interpret something claiming to be the "Future of Parallel Computing" as something that is just waiting on adoption. Perhaps "Towards the Future of Parallel Computing"...
Do you think calling your project parallel is what people have an issue with or do you think it's that you're calling your project the future of parallel computation when it doesn't perform anywhere close to what already exists?
I think the issue is that there is the implicit claim that this is faster than some alternative. Otherwise what's the point?
If you add some disclaimer like "Note: Bend is currently focused on correctness and scaling. On an absolute scale it may still be slower than single threaded Python. We plan to improve the absolute performance soon." then you won't see these comments.
Also this defensive tone does not come off well:
> We published the real benchmarks, checked and double checked. And then you complained some benchmarks are not so good. Which we acknowledged, and provided causes, and how we plan to address them. And then you said the benchmarks need more evaluation? How does that make sense in the context of them being underwhelming?
Right below install instructions, on Bend's README.md:
> But keep in mind our code gen is still on its infancy, and is nowhere as mature as SOTA compilers like GCC and GHC.
Second paragraph of Bend's GUIDE.md:
> While cool, Bend is far from perfect. In absolute terms it is still not so fast. Compared to SOTA compilers like GCC or GHC, our code gen is still embarrassingly bad, and there is a lot to improve. That said, it does what it promises: scaling horizontally with cores.
Limitations session on HVM2's paper:
> While HVM2 achieves near-linear speedup, its compiler is still extremely immature, and not nearly as fast as state-of-art alternatives like GCC of GHC. In single-thread CPU evaluation, HVM2, is still about 5x slower than GHC, and this number can grow to 100x on programs that involve loops and mutable arrays, since HVM2 doesn’t feature these yet.
Yeah exactly. I read most of the readme and watched the demo, but I'm not interested in installing it so I missed this. I would recommend moving this to the first section in its own paragraph.
I understand you might not want to focus on this but it's important information and not a bad thing at all.
Relatedly, the homepage itself doesnt make it obvious it’s still alpha, or not ready, or not actually going to speed up your code this moment - claims like “automatically achieves near-ideal speedup, up to 1000+ threads” - the point is that it parallelizes code, but the word speedup makes it sound like my code will get 1000x faster.
I think you can avoid this kind of criticism by setting expectations better - just plastering a banner at the top saying that it’s in early stage development and not optimized, but that the future is bright, for example. The current headline saying it’s the “parallel future of computation” isn’t really enough to make people understand that the future isn’t here yet.
Same goes for the README, the fact that it’s not production ready per-se really ought to be at the top to set people’s expectations properly IMO, since a lot of people will not read the whole wall of text and just jump straight into trying it out once they’re on your GitHub page.
They’re critical since they are led to have much higher expectations than what actually exists today.
That said, this is a cool project and wish you the best in making it really good!
It is not in alpha, nor not ready. You can use it in production today, if you want to. It is just not fast. That is different. CPython is still 100x slower than C, and is widely deployed in practice.
Seems like these are major problems for software whose whole purpose appears to make parallelizable programs go faster... Maybe I just don't understand the point then. To me it appears like a cool tech demo that fails to achieve the actual goal of delivering performance increases (by better utilizing the hardware), but it sounds like from your reply that being a cool tech demo that is probably not actually practical for truly leveraging your hardware... is the goal? So this is more of a research project than an actual worthwhile tool?
Based on how you've made a nice marketing page and README that sounds like you want people to actually use this tool in practice, within that context correctness is a minimum requirement/table stakes for a language to be usable at all, but that alone doesn't make it "production ready" if it fails to practically achieve anything you'd realistically want to do with it better than old-school languages that people already know how to use.
I am not a Python dev, but it seems that CPython's goal is not to be as fast as C, but just that it is a default runtime for Python [1] and the fact that C is in its name is just an implementation detail. Very different.
So the criticism leveled at the project appears to be valid.
While it is not fast in a single-thread, it is still 5x-7x faster than Node.js today for programs that are allocate a lot. If all you want is to run a program faster, and doesn't mind a bit more energy, Bend could be useful for you today.
And that's comparing a first-version interpreter against a SOTA runtime deployed in all browsers around the world and optimized by all major companies over 20+ years. If that's not useful to you, that's useful to me, which is why I wanted to share so it can be useful to more people.
Unsurprising this would be the case since LibreOffice strives for compatibility while MS would want clients to upgrade versions to newer, more lucrative editions
Back in the days, Microsoft took backwards compatibility very seriously, maybe more than anyone else besides mainframes. Going as far as making special cases so that applications that relied on some bugs or undocumented features continued working.
IIRC not for Word file formats. Usually only the previous version would definitely convert, any older and things started breaking. And it would always save in the newest version by default so everyone else has to update to edit your file (there were free readers, but they were separate programs and I don't know how popular they were).
Yeah I remember as a kid saving a file with Word 97, then opening it on a different computer with Word 95, and there was a box between every pair of characters. Looking back, it's clear that the internal representation had changed from an 8-bit encoding to UTF-16 but I learned to be careful to "save as Word 95" every time from then on.
Yup, exactly this. And there would usually be an official plugin you could optionally install to read files from ~2 versions ago, presumably because businesses demanded it, but still wouldn't go back farther.
I can't think of any other program that removed the ability to open older versions of its own files. Kinda crazy.
There are two file extensions, but with every new word version there was new features, and thus new stuff that had to be represented in the saved file. If you only went forwards and didn't skip to many versions, and remembered to save the file again in the newer version you'd be fine. but as the mastodon thread says, anything from before a certain date won't open at all, and more complex documents from the 90s and early 00s might have surprising issues in modern word.
Somebody more knowledgeable than me can correct me, but iirc some of the earlier versions of Rollercoaster Tycoon 2 were incompatible with the latest version of Windows at the time that was unreleased at the time of the game's launch (because RCT2 used a memory management bug? I don't remember) so the Windows team coded up a special exception for RCT2 because they knew that users would blame Windows rather than the RCT2 team if the game failed to launch). This patched version of Windows ended up shipping.
I have no source to back me up. This is something I read about sometime back and it's probably lost in some deeply nested bookmark folder I'll never open.
Or you know, standard open formats if you prefer them. But then there is always that MS trolls in comments.
I converted, 100gb of xls/doc files to upload to SharePoint and netted something like 5x space savings just by resaving in an open format.
I am not surprised at all that Libre is better at opening old files, there are files that Microsoft wont let you open without going back into security settings and allowing them, we were converting a library that goes back nearly 30 years, lots of stuff dated 20+ years ago, why would you carry compat for this trash today?
Standard formats? I don't think so. The "standard format" they use for OOXML is not compliant with the strict version - so loose LibreOffice couldn't go to the standard to fix the compatibility issues.
Countries were refusing to use MS office because it didn't support open document standards. MS basically showed up to the spec committee with a bunch of MS Word documentation and said it was a new spec. It was flatly rejected, but then at the last second, a whole bunch of the committee changed their minds for no logical reason (I suspect they were simply paid off) and the "standard" was adopted.
Of course, it's impossible to write something compliant with their garbage and Office didn't comply with the "spec" in the slightest.
At the same time, MS refused to keep up with ODF standards in a massive case of malicious compliance.
In the end, it worked out for MS. They faked being open just enough for European governments to continue giving them huge piles of money for systems that are as locked down and proprietary as ever and consumers (like always) were the big losers.
There is the "standard" [ISO-29500], there are patch notes [MS-OI29500] for the standard and there are Excel only extensions [MS-XLSX].
As someone who develops a library for xlsx, you have to check all three, as well as spend some time experimenting with various changes to xlsx. Patch notes are generally regulary updated with valuable information (when MS is asked a question it often is incorporated there). Standard is full of inaccuracies and is mostly primer. Some essential info misses from all and it's described as "implementation spexific", e.g. text to number conversation in formulas.
Excel has gotten better over the years, so it is mostly in line with standard.
Is it not a lot better on that front than .doc was, though? I realize a lot of stuff can read .doc today but it's due to a lot of reverse engineering that happened over like 20 years because it was entirely proprietary.
One point that a lot of those "actually, just use Go" articles fail to recognize is that there is a sizable amount of developers who simply dislike programming in Go.
Despite its impressive compilation speed and bundled tooling and whatnot, some (myself included) find it tedious and uninteresting to program in Go, with tons of repetition and verboseness. Some people are more utilitarian and don't mind, but not all think the same.
Yep. For me Concurrency and CSP are why I learned Go and why I still use it today. I love thinking concurrently about problems and how data and control flow through those systems. It was a natural fit for me in those aspects and the other language warts don't get in the way enough to matter.
No, let's actually have a standard with some thoughtful design instead of just bunging a bunch of ancient POSIXisms along into another platform. Also, as far as I can tell, it's being pushed exclusively by Wasmer which looks to be a bit shady as a company.
They mention stuff like "an interpreter which translates the [JavaScript] code but locks it up in a cage"¹, their presentation² mention JS interpreters and a JS AoT compiler, so I'm not really sure how they did it
¹: https://www.radicalfishgames.com/?p=6892 ²: https://www.youtube.com/watch?v=KfBzlzvt8RU
reply