The last two videos on the page are particularly impressive, which demonstrate video mixing using two channels of an audio mixer, and video blurring from multi-path audio reflections.
Cool thanks for sharing the name. I've been thinking a lot about functions that operate across multiple modalities (modalities as in visual, auditory, etc) lately. Good to know someone else has thought about it too, and looks like went much further.
Sunvox really is a stunning piece of software, for anyone who hasn't seen it or tinkered with it. It explores UI and synthesis in ways that haven't been really explored much (Jeskola Buzz started down the path many years ago, and there have been others in that space, but Sunvox is a beast in a category of its own). And, it runs (well) on very small devices...I was able to run it effectively on an old Nexus One, and it does very well on my Nexus 7 table and modern phone. It's hard to actually compose with such a small interface, but the synthesis and UI are efficient as hell. And, of course, it runs fine on a laptop of any sort.
So, yeah, I agree. Alexander Zolotov is brilliant and prolific. The demo videos for Sunvox are worth a watch.
The --with-lame is important if you want to process mp4s.
Might want to consider putting some kind of progress indicator (simple polling on tmp_audio_in.u8 vs tmp_audio_out.u8). I ended up looking at the output directory with
watch -n .5 ls -la /tmp/audio_shop-DIRECTORY
I think the cleanup isn't working quite right, getting a rm unlink failure.
Oh, I love things like that. I wrote a Photoshop file format plugin to load/save wave files a long time ago to manipulate audio files with image filters. Haven't updated it since, so it's still Mac OS 9 only.
I used to play with this as a kid. I couldn't script it, but I could remove the headers from a bmp and add the headers for a wav in a hex editor. Then I tried all the effects in audacity. The resulting images were pretty cool (only grayacale).
I also tried different audio compressions on the picture. Ogg vorbis even looked better than mp3, haha.
I actually played with doing the opposite when I was in high school: photoshop had a "RAW" file format plugin which let you specify the offset past the header and the bitmap data stride, height, width, and pixel format. You could open a WAV file as a Nx1 bitmap and then apply filters to it and get some interesting effects when saved back out and played in an audio player.
Might be interesting to experiment with different mappings of pixel location to audio sample number, rather than just having a row by row linear scan from corner to corner.
I am curious. I do not know much about image encoding and the such. What other types are there besides linear either row by row or column by column? It seems to me that those two are the only logical ways to map pixels.
Any permutation could be used, but I suppose you'd want to use ones that form some sort of visually recognisable pattern. For example, a spiral emerging from the centre of the image, or all the even numbered pixels from a linear scan followed by all those indexed by an odd number.
https://www.fsynth.com is a web synthesizer which is just doing that, a canvas content is generated by the GPU and converted to sounds in real-time, there will be a feature to import sounds directly soon.
http://www.warmplace.ru/soft/pixivisor/
The last two videos on the page are particularly impressive, which demonstrate video mixing using two channels of an audio mixer, and video blurring from multi-path audio reflections.