Couldn't get it to work in FF (Mac/22.0) - it asked for camera permission, camera activity light went on, but no images and constant re-prompts for permission. But, worked quite well in Chrome (Mac/27.0.1453.93).
Nice work. I'm curious about where you got the video, was that something you recorded, or got from somewhere? I don't really work with video, but I'm sure I can find something useful for it in the future.
I kinda wish it was at a higher bit-rate. There are some jaggies at the edges of the hair and stuff that I had to cover up by tinting the color of the video.
This is awesome! I can't wait to see the mashups that spring forth from this (and hopefully make one or two myself). There will definitely be something really creative soon. Thanks for doing so much legwork on this.
I've clocked many hours into After Effects, but seeing this in a browser and _interactive_ is amazing. Never would I have thought that JS and browsers would progress so much in such a short period of time.
OTOH, it's mainly possible because computers are faster so Javascript isn't a slow language anymore, and the document model is just getting out of the way and letting you manipulate pixels directly.
This isn't manipulating the pixels directly. Javascript is still way too slow to do that, and I don't think that will ever change. The real work is done by the graphics card. The Javascript only uploads a frame to the graphics card, makes it run a shader and then displays the adapted frame.
The technology used here to do the image processing is GLSL, in particular fragment shaders (aka pixel shaders). GLSL is a very small C-like language that's become a standard for GPUs. GLSL code gets sent (as a string) to the GPU by javascript via the WebGL API.
Seriously is a JS library for handling the boilerplate of WebGL, composing and compositing multiple shaders in a pipeline/graph, and adjusting their parameters. In addition it comes with a bunch of pre-written shaders.
Shaders themselves are a lot of fun to write, IMO. A pixel shader is just a function that computes a color given an input pixel location. (A shader can also take in additional input such as "textures" to sample from, e.g. a video frame.)
The shader is run with massive parallelization in the GPU. In theory every pixel can be processed simultaneously. This is how these effects can run in real time.
Here are some more examples of what you can do with pixel shaders (including sampling from the webcam in the browser -- should work in Chrome and FF),
http://pixelshaders.com/examples/
No one said it would be MORE complicated than the examples.
"Would also be interesting" and "fun to compare" give no implication of relative difficulty. I would like to see the algorithm though so please post the code if you get it working in JS! That would be awesome!
It is actually slightly more complicated. The existing effects work in GLSL and are just transforming single frames of video. For that effect to work, you need the previous buffer states; this means that seeking will cause the image to be different, and you need to feed the previous render state back as a texture.
You still need to remove the green-screen. This either means an expensive process on the CPU side, or pushing the frame to the GPU to remove it with a shader and then using RTT and rendering that or reading the framebuffer and drawing that to a canvas. Note, it's really not difficult to do this, it's just slightly more complex than the other effects, is all.
Well no, because each frame builds on all the frames before it. Either way, seeking is still broken unless you run through all the frames up to that point.
Each frame has the greenscreen turned into a fully transparent pixel, then gets blitted onto a buffer containing all of the existing buffer's data. That means that for each frame, you're building on the data from every frame that came before. This is fine if you're playing from beginning to end, but when you seek, you're going to break that continuity.
Yes, you do need to process from beginning to end. If you look at the original video, you see that each frame builds on the one before it, which builds on the one before that. Remember, the buffer is never cleared -- it's just overwritten with a few new pixels (the dude moving) each room.
> A real-time, node-based video effects compositor for the web built with HTML5, Javascript and WebGL
I for one am really impressed. While I'm not up to date with node.js, WebGL (or even HTML5 tbh), the boundaries that people keep pushing with web technologies is pretty wild.
EDIT: Reading more, I'm not sure if I am proud or embarrassed about my narrow focus over the last few years. I'm proud that I'm making a good living due to it, but disappointed that I haven't been able play with these things. Reading about WebGL [1], I feel like there is a new world to explore (for web programming).
"Sadly, we are unable to get Seriously.js to work on your computer. Sometimes WebGL gets a bit weird with certain graphics hardware and drivers. Please have a look here for more information."
"Sadly, we are unable to get Seriously.js to work on your computer. Sometimes WebGL gets a bit weird with certain graphics hardware and drivers. Please have a look here for more information."
Yeah, same here, running Ubuntu on a Dell Precision laptop. Guess WebGL doesn't like my setup.
I coded WebGL a little over a year ago on a Precision "laptop" under Suse Linux. What I needed: (A) up-to-date Nvidia proprietary drivers (B) "de-blacklist myself" somewhere deep in the browser's hidden WebGL config page (I think it was about:gpu or some such back then in Chromium, but I'm not really up to date).
Yeah, I run Chrome on Ubuntu on a pretty powerful Lenovo T-something with an NVIDIA whatever optimus pain in the neck. The only way I can get it to work is to add --ignore-gpu-blacklist to the Chrome command line.
Check:
/usr/share/applications/google-chrome.desktop
(or something like that)
You should see some lines that start with "Exec" where you can add that parameter to the end.
I had the same deal with Chrome on my OSX Mountain Lion. Here's what fixed it for me:
1. Visiting chrome://flags
2. Scrolling down to Disable WebGL, hitting "Enable", and then "Relaunch now"
3. Repeating steps 1 and 2, but hitting "Disable" rather than "Enable".
Impressive! It's really amazing to see how OpenGL can enrich the web if it's used right. A few years ago I'd never have guessed shaders could be used in a browser.
I have little experience with Javascript projects, so can I ask why the name Seriously? Wouldn't a more descriptive name cover the contents? Also, why is the main source file 4K lines long? That's a little off-putting.
It's called Seriously for two reasons:
1) At some point, I had this idea that I would make all my Javascript libraries references to Homestar Runner. http://www.hrwiki.org/wiki/Seriously
2) "You can do that in a browser? Seriously?!"
As for the file size, yeah you're right it's too big. I'm working on a grunt.js build script, and when that's done, I'll break the main file into smaller pieces so you can pick and choose which utilities and things you need to load. Right now, it has the color lookup table and noise shaders, among other things, which are not necessary for everybody.
Intel announced a year or so ago they could JIT compile javascript and run it on their GPU. Maybe that's why?! If so, welcome to the bold new age of GPGPU and AMD's HSA... :)
LOL, my old core2 duo has both cpus running over 80% on Ubuntu with a GeForce GTX260. But when I click on another tab, with the video still playing on a non-visible tab in Chrome, the cpus drop to around 20%. When I click on the video tab again, the cpu ramps up to the 80% mark again. Neat trick.
I'm running an i5 processor, and when the video was playing my cores jumped from the 5-10% range (background stuff I've got running) to the 25-40% range.
Interestingly, just switching from the demo tab to this HN tab (FF 21 on Ubuntu) drops the cores to the 10% range. They jump again when I switch back.
This is the sort of thing we're working on at Vidpresso. If this impresses you, and you want to build stuff like this... we should be friends. Contact info is in my profile.
I'm working on porting this over to Seriously.js so it should run in the GPU, hopefully pretty fast. The first crack at it was not so good, and now I'm experimenting with some new features that are only in Chrome nightly builds at the moment (and buggy at that). So look for this in the next 3-6 months.
This is pretty awesome, but it seems like OK Go has cornered the market on "cool web tech" demo music videos. Doesn't anybody remember their collaboration with Google?
Running the latest Chrome on Mac OS X. (btw, realized my initial comment probably came off a little snarky. My apologies. From the sounds of the other comments, it's something incredibly cool!)
I know next to nothing about graphics programming but how does this work? I don't suppose there is a Javascript loop that goes through all the pixels on each frame...
Nope. That would be too slow. I tried that a couple years ago before writing Seriously.js, and the big problem was copying around the memory. You have to first draw the video to a canvas, then you have to extract the pixel array from the canvas, loop through all the pixels and then draw the pixels back to the canvas. Now, for a high-definition video at 4 bytes per pixel (RGBA), you're looking at about 3.5MB of memory for each copy. And on top of that, it has to allocate and garbage collect that 3.5M pixel array for every frame, because it doesn't let you copy into an existing array. If you look at the first video-processing canvas demos, they tend to be pretty small, and this is why.
Instead, this uses WebGL/OpenGL. So the pixels are copied once from the video directly into the GPU memory, and they are processed in parallel there before going straight to your screen. Much, MUCH faster.
The GPU doesn't execute Javascript. It runs small programs called "shaders", written in GLSL (the OpenGL shading language), which are compiled on the fly by the graphics driver to something that can be actually executed on the GPU.
Do you mind sharing your video card info and operating system? It will be hard to debug remotely, but I'll see what I can dig up.
Please post it here or on github: https://github.com/brianchirls/Seriously.js/issues
Doesn't work at all for me in Firefox 21.0 on amd64 Linux. Works in Chromium 25 on the same box. This is particularly disappointing because there's a Mozilla logo on the page.
i did something like this (ok, not as smooth but it goes in the same direction) in 2010 with canvas/video/css http://www.barbafan.de/html5video?video=tron (was covered by boingboing at that time, lots of traffic, some partnership opportunities (mostly from p_rn sites), some job offers, non any interesting))
cool that somebody now packs a similar idea into a lib, will definitely check it out.
Whoa, this caused the second BSOD I've seen since Windows 98. When I started messing with the options, the windows just flashed wildly for a while, some popup about ATI Radeon driver errors was struggling to spring to existence, and then everything froze.
Please keep the bug reports and questions coming. I'll try to fix/answer what I can.