Hacker News new | past | comments | ask | show | jobs | submit login
Soul – A language and IDE for audio coding (soul.dev)
323 points by ttoinou 48 days ago | hide | past | web | favorite | 95 comments

I'm excited that audio-programming-in-the-browser is becoming a thing. It means more people are likely to fall down this exquisite rabbit hole. I'm totally in this field so I'll list a few websites that let you program music in the browser, and my rough opinion on them:

- http://wavepot.com/ is just built on top of Javascript. The code tends to be long since the library is pretty bare-bones right now, but it's really well executed.

- https://gibber.cc/ is also JS-based. The library and functions provided make for some pretty terse yet readable code, though focused on somewhat traditional synthesis methods (and I still don't think JS is ideal for this...)

- http://ctford.github.io/klangmeister/ is based on Clojurescript, which is a safe choice since Overtone has long demonstrated that Lisp is great for music programming. Right now there's not much of a library to go with it, though.

- http://faust.grame.fr/editor/ - Faust is one of the greatest music programming languages, and it also runs in the browser, but I think they haven't sufficiently polished the browser experience in a way that lets a novice learn the language in the browser.

- https://audiomasher.org/browse - I'm biased on that one since I made it. An obvious weakness is that the language is a little frightening at first glance, but there's a tutorial. It's terse yet open-ended, and the built-in library is pretty extensive.

For SOUL, it seems running in the browser is just a little demo, and the language is meant to be used in other situations. In fact, most of the above projects have grander aspirations than running in the browser - it's generally just a way of getting people to try out the language or system. But I still wish more projects would take the extra step and create a community-style website where anyone can upload and play with each other's code, otherwise the web experience just seems like a proof-of-concept.

But is a browser-based audio environment ever going to have the low latency and robustness requirements for something like live performance?

WASM will probably start catching up with natively-compiled C++ eventually, but we're working on the hypothesis that even natively-compiled C++ can't quite get the best performance out of audio hardware, because running it on the CPU in a normal OS is the wrong place for a realtime task like that.

Something we do want to create will be network-attached SOUL audio i/o devices which can run the code using native JIT engines. Then, Javascript in a browser can send its SOUL code to be compiled + executed directly on these bare-metal devices and get super-low latency.

Low-Latency is one thing (now much better with AudioWorklet model), real-time rendering is another thing. With AudioWorklet the audio code now runs in a dedicated thread, but without real-time priority for now.

Only checked out wavepot so far. Love their logo (it's a lil wave pot, and the wave is a tilde because it's computery) and the last half of the tracks like "on the verge".

Impressive what people are making with something so low level. Geez, cool mixture of artistic and technical skill.

And shadertoy.com where you have an Audio buffer generated before the real time rendering :)

Just for the sake of completeness and for those new to the topic, here are some mainstream audio programming languages people might also want to take look at:

- Csound This is one of the first ones, a little bit dated now but there is ton of material and nice books

- Pure Data (max/msp) Graphical audio language, also popular and very nice for beginners

- Supercollider Object oriented platform for audio synthesis (smalltalk)

- Overtone Functional audio synthesis platform using Clojure (totally awesome)

Sam Aaron (original dev behind Overtone) has been focused on building out Sonic Pi ( https://sonic-pi.net/ ) for a few years now and it's an amazing tool aimed at simplicity (literally aimed at ten year olds) and power (used for gigging).

>Sam Aaron (original dev behind Overtone)

Was he actually the original dev? I get that he contributed a lot to the project, but overtone isn't dead, and I don't think he was the original dev.

Very nice, thanks for the great info.

Also Faust (the doc is totally awesome : https://faust.grame.fr/doc/manual/index.html#quick-start) and - shameless plug, a project I work on : https://ossia.io which is a merge of the sequencer & dataflow paradigms

There's also ChucK http://chuck.stanford.edu/

Another one to add to the list is Faust (https://faust.grame.fr/) which has great support for cross compiling to other languages (C, Java, wasm etc)

Hello HN - thanks for giving our website a proper stress-test!

I'm the developer of SOUL - happy to answer any questions you guys have.. :)

Hi. Quick fly-by suggestion for you: look into the language server protocol by Microsoft if you haven't already.

In the golang community for example Google has recently decided to build and officially support a language server (called 'gopls'). How it works: You build a language server once and get features like typecheck errors, auto complete, go to definition, code peaking, documentation on hover, etc... in all editors/IDEs that support the protocol.

LSP Website: https://microsoft.github.io/language-server-protocol/ Recent golang talk: https://youtu.be/bNFl7HcyDao?t=354

Yeah, this is on our to-do-list!

Cool :)

Hi Jules, I posted this here; I'm so eager about your project moving forward !

Is there going to be a marketplace for professional SOUL-based VST plugins ?

A way to browse all open source SOUL scripts and share modules so that the community will be able to build on top of previous scripts (without copy pasting all the time) ?

Will all SOUL scripts be runnable in the browser like Shadertoy.com, if no what will be the separate features between web / desktop / hardware ? How are you going to handle the diversity of situation where one could use SOUL ?

Thanks (-:

Side note : http://soul.dev/ doesn't have https ?

Our long-term goal with plugins is to make them much easier to write in SOUL without needing to actually compile anything natively.. as for a marketplace to deliver them, we'd certainly love to be able to offer one of those, and will see how things go.

Yep, we'd love to make soul.dev into a place where you can browse snippets of code, search for people's implementations of various DSP algorithms and try them out and put them together - that's all definitely in the plan!

And as far as cross-platform-ness goes, I don't currently foresee a situation where any SOUL code wouldn't run on all platforms.. Running it as WASM in a browser may not be particularly quick compared to the same code running via LLVM JIT or on a DSP, but it should still work.

(And yes... soul.dev certainly uses HTTPS.. maybe whoever posted this story typed it without that)

Thanks for the reply. Suggestion for the name of the marketplace or community : "Soulfly" sounds good (and probably need to separate it from the "infrastructure" / dev project ?)

Well, we also have this URL: https://audio.dev :)

so this is gonna seem kinda weird, but I noticed y'all run the audio developers conference.

I own the domain adc.io, for a project that went nowhere years ago, so now I mostly use it for some personal things. would you guys be interested in doing a trade or want to buy it or something? I feel kinda trashy asking that here since it doesn't add to the discussion, but I like what y'all are doing, and didn't even know that conference existed until I clicked that link.

thanks for the offer but I think we're a bit overloaded with URLs right now!

> Our long-term goal with plugins is to make them much easier to write in SOUL without needing to actually compile anything natively

could you offer to compile/package SOUL code into a VST as a service?

Yes, we already have a prototype C++ generator as a command-line tool, so mixing that with some boilerplate to build a plugin will be something we do soon.

And yes, the same thing would also work nicely if hosted as a web-service

To LV2 also would be awesome if possible!

Please consider making dynamic graphs a key feature for the 1.0 release! I like how easy it is to use SOUL to build simple processes, but I'd like it a lot more if I didn't have to specify my graph at compile time.

A few more questions:

- How will the language/API/reference VM be licensed?

- Will the API depend in any way on the JUCE ecosystem?

- Do you expect users to bundle the VM with their plugins/applications or to have one installation on a target machine? Or will plugins need to be supported by a host that embeds the VM?

Not sure that being able to change the graph dynamically is a particularly good way to work with this kind of thing. In almost all synths, and even DAW engines, they only modify the graph when you modify the project, not while things are running, and that's when we expect you'd do a SOUL recompile. There's a much longer discussion to be had over this which I won't dive into now, but we have thought about many possible use-cases and how they'd be done.

Licensing: very permissive for developers; probably commerical deals for companies who want to ship SOUL-compliant hardware or drivers

JUCE dependency: no, we'll want to make this as vanilla as possible, to encourage its use in many ecosystems. There'll be JUCE integration, but also stand-alone C++ and a flat C API so it can be wrapped in other languages like python, C#, Java etc

We'll offer an embeddable JIT VM, but our end-goal with this is for there to be drivers in the OS or external hardware which does the work, and the API would just send them the code (like e.g. openGL shaders)

It would be very helpful to have some way to express graph changes due to events in the language, even if it doesn't happen in real time. Otherwise we're going to have to get really hacky with generating SOUL on the fly and recompiling it, rather than just explicitly stating that graph changes happen at the discretion of the runtime.

RE the licensing, I'm asking more about the language itself, the IR, and whether it is going to be possible to develop independent implementations.

Lastly because I always forget to ask about the IR - why not WASM?

I think we'll probably want hot-reloading of sub-graphs in a larger graph, but that'll be a project for 1-2 years down the line when we're more involved in building DAW engines using it.

Re: building a 3rd party back-end, I guess that might be technically possible but we'd rather keep control of that side of things. We've not fully decided our approach there yet.

Why not WASM? Well.. quite a few complicated reasons, involving things like the ability to generate continuations, and to get good auto-vectorisation, and being portable to weird DSP architectures. Also, the system can't just be a straight program, it had to be proprietary, at least at a high-level. And secure, so LLVM IR also wasn't quite the right shape for it. We sweated over this decision, believe me!

You mention custom DSP hardware licensing ... any reason not to target the GPU directly using CUDA or whatever?

I've looked a tiny bit into this sort of thing (using Metal compute shaders), and the problem is that, even with all of Metal's optimizations around CPU/GPU synchronization, the overhead seems to be too high for real-time audio DSP in practice.

The thing I tried experimentally implementing is basically a 128x128 matrix mixer with a little bit of extra processing. On a three-year-old MacBook Pro, the GPU barely had to lift a finger to crunch the data, but the round-trip latency was still high enough that it would struggle to keep up with anything less than a buffer size of 512 or so at 48kHz (which is on the high end for live mic processing). It would be fantastic for offline processing with larger buffers, though.

I haven't tried CUDA or OpenCL, so I don't know if the situation is the same there—but of course they have the problem of vendor support as well.

Our original thoughts on that was "nah, that'd be silly because GPUs are the wrong shape for audio workloads"

..however, having a few conversations with more knowledgable people has changed our mind and we're definitely going to give it a try on Metal and Vulkan to see what happens. Should be interesting.

I can understand the aversion to GPU (with "Graphics" being a primary point). Most ways of interfacing with the GPU use x,y,z, color, lighting wrappers around what is essentially a super powerful vpu. I also wonder how the traditional pipeline (e.g. vertex shading, fragment shading) can be repurposed for audio.

I don't have any experience with Metal or Vulcan but my intuition is that audio DSP is going to include a healthy dose of linear algebra and multi-variate calculus. That points towards some kind of fit with the GPU. Given that basic GPUs are available on practically every device (including phones) it seems like a fantastic fit. Even audio hardware developers would benefit since it would open up access to commodity priced chips (rather than custom asic/fpga/whatever).


And even for very simple audio loads, there's often unused capacity in compute cores (even when running games) which would could do the job "for free" without bothering the CPU.

I'm curious what you mean by "wrong shape"? Because GPUs are for for very data-parallel workloads? Or because the latency is high?

Yeah, we made both of those assumptions. But it's apparently less true of the latest generation of compute engines. We probably won't bother with CUDA, but Metal is certainly a viable platform to try. It's one of those things where nobody really knows how it'll do until you try it.

Also, we do know a few people who want to use SOUL to write audio code which does need high parallelism. It's not a super-common use-case for audio, but it does exist.

I hope you'll consider writing up your results in a blog post or something! As I just mentioned elsewhere in this thread, I have experimentally found latency with Metal compute to be too high to be feasible, but I would dearly love to be proven wrong, if there's a trick I'm missing.

Yes, we'll certainly be writing it up. Similarly i'd be interested in reading about your experiences attempting the same if that's available.

Hi. Great talk. Basically introduced me to audio development! Given you're experience and the future you're talking about with Soul or a similar language being how audio development is done... what resources and approaches would you recommend someone completely new to the domain study?

Audio dev has always been insanely hard - that's part of the motivation behind creating this!

It's hard because you need some serious mathematical skills to understand the DSP itself (that's the bit I'm lacking in..).

Then if you're building a "real" product, you're going to have to write in C++ and have a rock-solid understanding of concurrency, real-time coding and many other very tricky subjects which take huge amounts of experience to get good at.

I've been doing this for over 20 years so have lost sight of how beginners should learn it.. But most people seem to just dive in with an idea they want to build, and start trying to swim! It can be painful to see people using JUCE/C++ who don't really have any interest in C++ for its own sake, but who are struggling to get anywhere without putting the effort in to learn the language properly.

I wonder if the SOUL team has considered looking at Julia, which aims to solve the two-language problem, in which programming languages are either fast or easy. It's in the context of scientific computing, which is usually almost entirely about throughput and not about latency. But I think there's a lot in common.

At a high level it might seem like that, but there's a big difference between "fast" and "realtime". We need to go fast, but the main problem the language needs to solve for us is to break the work down into a steady, realtime stream. Overall throughput is Julia's strong point, but that's not the same thing at all.

I would argue that Julia's strong point is generating specialized code, and facilitating generic programming. Throughput is definitely the driver, but I think several aspects of Julia's design could benefit real-time programming as well.

Currently, there are many blockers to using Julia in real-time application, such as dynamic memory allocation and lack of thread safety. But I find it promising that a subset of Julia could be used for real-time programming.

Hi julesrms,

Some quick 2-second feedback - I tried running the default example in Brave, it failed, and then as a good user I wanted to report the bug to the community, but that requires finding the ROLI forum and signing up (providing DOB!) etc. I think it would be great if it were a bit easier to submit bug reports without as much hassle.

You could report the issue here: https://github.com/soul-lang/SOUL/issues

(We might open the soul.dev website too at some point, but that's private for now.)

I've just tried and it worked for me on OSX with Brave.

For anyone looking for more context see https://github.com/soul-lang/SOUL/blob/master/docs/SOUL_Over... and https://github.com/soul-lang/SOUL/blob/master/docs/SOUL_FAQ....

To quote the overview: "The SOUL platform is a language and an API. The language is a small, carefully crafted DSL for writing the real-time parts of an audio algorithm. The API is designed to deploy that SOUL code to heterogeneous CPUs and DSPs, both locally and remotely."

Exciting! The team behind it is partly from the JUCE team, it looks serious enough

Heh yeah, I have the pleasure of working with them, it’s pretty serious :)

Lucky :D

I recall seeing Soul pop up on HN before, and at the time I was critical of it for being announced way too early (no working examples or even any indication of what the code might look like). This is much better, and it's good to see that it's actually progressing instead of being yet another case of vaporware.

That said, still disappointing that there doesn't appear to be any info on an actual compiler or runtime or however this is supposed to be used in actual projects; the editor configs and code samples are a start, but kinda useless if the only way to actually put 'em to use is to use a buggy web app :)

And speaking of "buggy web app", mousewheel-scrolling in the playground seems to be entirely broken on Firefox on Linux. Apparently this is true of all instances of Microsoft's web-based "Monaco" editor widget (on which the Soul playground appears to be built). What the hell, Microsoft?

Thanks for the spot of the monaco bug - that's a regression that we'd previously fixed. It's been patched and re-deployed and should work now.

Confirmed; still doesn't scroll smoothly (or at least as smoothly as most other things in Firefox), but it at least does scroll now, so that's good enough for me. Thanks!

There'll actually be many ways of using it.. Javascript in a browser is just one of those ways. The real API is written in C++ and we'll be offering a C binding in many languages, which you'll use in a pretty similar way to how you'd use the openGL API to run a shader. We'll publish that C API soon-ish, hopefully within a few months.

Okay, but what about the stuff behind that API? If OpenGL is the model, then is the expectation that audio device/driver vendors implement their own Soul backends? Will there be a reference implementation? Is that reference implementation going to be released under a free software license?

Any info on the motivation for making an entire language instead of a library for an existing language (e.g. for C, Python, or Go)?

Especially if the syntax mimics C-like languages, what important advantages do we get?

I had the same reaction to SOUL at first (why do we need yet another language?!!), but I've come around to it.

The precedent to consider is shader languages. A language like GLSL constrains what the programmer can do, with very good reason: the probability a newcomer will write a performant-enough and correct-enough shader in a general purpose programming language is low.

Similarly with audio programming. Not blocking the audio thread is hard! Heck, just dealing with multithreading itself is hard. JUCE tries to make audio programming much simpler, but at the end of the day it's still C++, and footguns abound.

So I like to think of SOUL as a sort of shader language for audio. We built https://soul.dev/playground/ to be a sort of "Shadertoy for Audio."

oh.. I should also mention that we don't expect it to replace all the other audio systems and languages that are already established and have large user-bases. (At least not overnight ;) )

..but we are attempting to do something that nobody else is, which is to provide a platform they can use as a target.

If we can do the hard work and graft of turning this into a super-portable runtime that can target whatever low-latency hardware the user has (and create a market for new audio accelerators that don't yet exist), then we expect it to be attractive for all those existing frameworks to use SOUL (or our IR) as the thing they emit, rather than using LLVM's JIT or WASM or just interpreters as they do now.

We pre-emptively answered this inevitable question in the overview doc: https://github.com/soul-lang/SOUL/blob/master/docs/SOUL_Over...

The main TL;DRs are:

- this needs to get JITed to compete with C++ performance, so dynamic and interpreted languages are out.

- it needs to stop people doing anything which is real-time unsafe, so any language which involves a heap or GC is out

- it needs to be secure enough to not pose a security risk if deployed to an embedded bare-metal device, so C is out.

- it needs to be super-easy to learn for all programmers, and especially for grizzled old C/C++/Javascript/C# people who aren't into any of that fancy functional nonsense

- it needs to strongly enforce and represent a graph structure at a syntactic level, so.. pretty much all existing languages are out.

> - it needs to strongly enforce and represent a graph structure at a syntactic level, so.. pretty much all existing languages are out.

... except languages designed for real-time signal processing based on dataflow graphs such as Kronos (https://www.mitpressjournals.org/doi/pdfplus/10.1162/COMJ_a_...), Céu (http://www.ceu-lang.org/chico/ceumedia_webmedia16_pre.pdf), Antescofo (https://hal.inria.fr/hal-01585489) and the oldies such as SIGNAL, Lustre, Esterel, etc... :p

OK, fair point! Maybe a better way for me to phrase what I meant there would have been "pretty much all mainstream languages are out"

The overview at https://github.com/soul-lang/SOUL/blob/master/docs/SOUL_Over... should help explain the motivation. I’m sure Jules can elaborate further :)

Concerning Faust-to-SOUL support, it is still reserved for people not afraid of installing and compiling Faust from github: https://github.com/grame-cncm/faust

Then a faust2soul script can be used to directly compile Faust DSP to SOUL source code: https://github.com/grame-cncm/faust/tree/master-dev/architec...

To be tested, the generated SOUL code can then simply be copy/pasted in the SOUL playground.

I hope there's some benefit to other domains that rely on real-time signal processing, like control systems. I would like to have something like this for robotics.

We come from an audio background, but all the way through developing the project, we've been saying "...I bet this is useful for some other domain that we know nothing about.." so have been careful to make it about signal processing in general, not for only audio.

Yes, could be that it works really well for machine control, robotics, etc. Hopefully as it gets more established, people from those worlds will get involved and see what happens.

This has been a great thread, thanks for all the comments!

One final note I'd like to add is that we've also been running a closed early-adopter group for industry insiders, where we're sharing some other non-web tools including our LLVM JIT engine and C++ generator, and discussing partnerships etc.

If you represent a company that think should join this group, ping me and let me know..

There's also ChucK: https://chuck.cs.princeton.edu

Thanks. Just clicked through to the ChucK site.

For anyone interested apparently there's a free MOOC in may: https://www.kadenze.com/courses/introduction-to-programming-...

And a TED talk: https://www.ted.com/talks/ge_wang_the_diy_orchestra_of_the_f...

chuck was really interesting but has been pretty much abandoned, as far as i know...

https://github.com/ccrma/chuck doesn't look super abandoned to me...

How are the relationships with Ableton who are also in the Web Audio space and users of Juce?

They've built a Music Learning platform with Elm (afaik): https://learningmusic.ableton.com/

Elm wants to target WebAssembly, wouldn't it be able to compile to Soul too in theory?

I've not seen the Ableton platform, but the Faust guys have already released Faust-to-SOUL support, and we're hoping it'll make a really good back-end for that kind of thing (of which there are many examples)

This soup is great but need to understand a bit more. Does it allow for manipulation of audio using code? Use case examples?

Indeed, that's exactly what it's for. The examples include some audio manipulation processors (like gain, reverb, compression etc) and some generators, from a cheeky little beep through to an analog style polysynth.

The target use is to unify DSP development across different platforms, whether you are developing web based audio, or gaming, or pro audio, you should be able to write your algorithm in SOUL and get it up and running.

Yes it does; it's only the beginning of this project / infrastructure / ecosystem. It will allow to easily build VST plugins with the interface coded with web technologies à-la-electron / nwjs and the core audio coded in SOUL

*This is great

The soup is also great.

You can edit the comment still, if not the soup.

Dumb question -- but what exactly is an audio programming language? Google seems to be failing me.

I can make some guesses by looking at the example in OPs link. But what is the primary goal of an audio programming language? Who's using it? How? Alternatives?

Genuinely curious.

A programming language specializing into generating audio. Where environments like Processing or the languages used for GPU shaders are designed to generate pixels (the former at a high level, the latter at a very low level, targeting specialized hardware), these languages are for making music and sounds.

> what exactly is an audio programming language

Simply put, it's a language designed to make creating and processing high-quality audio quickly more straightforward than a general purpose language would make it.

I think csound is mature system for programmatic generation of sound/music. How is this different from what csound is offering?

This is about building a cross-platform infrastructure that would sit below things like csound, Faust, and many other similar front-ends, and allow them to emit code that'd run optimally on hardware platforms other than just a desktop CPU, to get better latency and performance.

...and without wanting to "throw shade" at csound or anyone else, we think the SOUL syntax is much nicer to write of course :)

I'm not sure if this is just related to my machine, but I couldn't get it to compile.

    POST https://media.noise.fm/soul net::ERR_CERT_AUTHORITY_INVALID
(this looks very cool and I want to try it out)

Strange, our media.noise.fm ssl cert is issued by AWS, I'd assume you would have the necessary roots for that. I'll look into it further!

What browser and OS are you using?

(Editied with resolution at the bottom)

Yeah AWS should work just fine.

I'm on macOS 10.14.4, and tried with the current version of Firefox, Safari, and Chrome.

Different reasons, but same result for all. That error came from Chrome and was the most descriptive. Firefox threw a CORS request blocking error.

Edit: Yeah and Safari is also throwing a cert error:

    [Error] Failed to load resource: The certificate for this server is invalid. You might be connecting to a server that is pretending to be “media.noise.fm” which could put your confidential information at risk. (out.wasm, line 0)
Edit again!: RESOLVED

I simply won't be able to access this from work. I can't see the noise.fm root path as the corporate routers here are blocking it due to "Domain Parking". (eyeroll) I'll check this out at home later on!

Thanks for checking in.

Aha! Thanks for investigating.

We think this is maybe a weird local CA? We use a CA issued by AWS so should be pretty universal.

Try http. Also try Chrome; I think it doesn't work on my Firefox

Hm. I was in Chrome, and thought maybe it was one of my extensions (HTTPS Everywhere, Adblock), but no.

Tried in FF as well and got CORS errors.

But works on my FF though :P

This is cool.

Can I have it against CoreAudio or Pulse or anything other than WebAudio?

Have a read of the FAQ and overview docs, or watch the talk: https://github.com/soul-lang/SOUL

The browser-based stuff is just a demo to let people explore the language and learn about what we're trying to do. The actual goal is to create the fastest, lowest-latency infrastructure for audio on all platforms.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact