I'm currently trying to get realtime audio streaming over WebSockets to work. I'm encoding audio using libspeex on the server side (an iOS device) and decoding it again on the client using speex.js.
Transmitting Speex "frames" (where each frame is a 20ms chunk of audio) over WebSockets and decoding in JS works beautifully. However, I have a really hard time to queue up these 20ms AudioBuffer nodes perfectly, without producing any glitches or pops. I'm not sure its even possible with the current API.
What I would like to have is an AudioBuffer node where I can dynamically append chunks while its playing. Since all(?) source nodes are for one time only use, the browser could free the data that was already played again.
An AudioBuffer that allows modification of the data while it's playing would work too - you could just loop it and use it as a ring buffer. However, my understanding of the spec is, that modifying an AudioBuffer's data is explicitly prohibited once its playing.
IE doesn't support Web Audio either. The project I linked above does contain a Flash shim for IE, which you may find helpful.
Any proper realtime audio stack (which ogv.js is not: it targets moderate latency streaming) is going to require a jitter buffer to deal with network/processing latency variation. Writing one is a non-trivial exercise (though not impossibly so).
In any case, I'd recommend using Opus (the audio codec used by WebRTC) over Speex.
A few years ago, I tried to convince my colleagues (they're more designers than programmers) to use IRC for informal communication. Using IRC was (and still is) a no-brainer for me. My desktop PC runs all day, so I'm in IRC all day. But for them, joining IRC meant opening their laptops, starting their client and then pinging out 15 minutes later when their laptop went to standby.
When they were in IRC, they had this urge to really be there in IRC. I.e. monitoring it, reading everything, responding in seconds, excusing themselves when they had to go to the bathroom, and finally, finding excuses to close their IRC client again.
I have a totally different mode of operation with IRC, where it's OK to have the client open and not respond for hours or at all if you don't fell like it. I guess it's something you have to learn.
Installing BNCs solved some of the issues, but not all. Also, installing a BNC is not something your average designer-type nerd today wants to deal with.
So we ended up with KickOff for a while, but it never really worked reliably. We're now on Slack and quite happy, but I still want to try some of the various IRC "cloud" services; mainly those for self hosting.
Well said. I have been trying out ZNC for a few days. It wasn't really a fun to setup ZNC. There are things that I just don't understand. ZNC comes with a bunch of modules that you could use to extend its functionality. It is really hard for me to understand those when I am new to both Bouncers and IRC. Then there are no standards. ZNC uses a username but IRC doesn't really have a concept of username. It is usually Channel, Nick, and password. IRC also has concepts like Ident which are hard for general users to understand. I have setup ZNC but the setup is not complete yet.
I am not well aware of history of XMPP and IRC. I wonder if lack of interest in the organizations that control these protocols have lead to numerous versions of them that there are today. Slack has an IRC gateway and Facebook uses XMPP. Both of these protocols are widely used but have becomes too different from the core concept.
You should try using Quassel — it's the perfect solution for an IRC Bouncer/Client-combo, it feels like any modern solution (aka infinite scroll to read previous messages, clients for the bouncer on every platform, etc).
It's an amazing solution. So amazing, that I actually started commuting to the Quassel Android client, just because I want to give something back to this community.
Actually chat secure on android makes facebook chat almost plesant (and those "in the know" can enable otr). Such a shame fb doesn't do federation and google pretty much killed xmpp altogether. I was this close to having a chat platform other than sms that I enjoyed and could share with non-technical friends without hassle. I loje IRC - but so far not on mobile. Anyone have a favorite (preferably Free as in freedom) cliebt for Android?
Also, lack of addressability. IRC is primarily targeted towards channels while all the needs of current IM systems are targeted towards one-to-one messaging. Possible with IRC, but you have to jump through hoops.
We have the Piratenpartei in Germany, full of nerds and people who take a strong stance for privacy, net neutrality and many other things tech related. They get laughed at and ridiculed (often for good reason) and are considered "unelectable" by many, even in the tech field.
BUT they force other parties to think more about these issues and take position themselfs. I think the Piratenpartei is extremely beneficial for the political landscape here.
I have often wondered why the US seemed so slow to adopt these ideas. From my limited understanding, a good deal of the problem is probably the 2 party system in the US, but that doesn't mean a "HN party" couldn't apply a lot of pressure to democrats and republicans alike.
We've been using phash for an image board for a while now and are quite happy with it. We only use it to detect reposts when someone uploads an image. It gives some false positives quite often, but that's totally okay for our use case. We specifically set it up to err on the safe side. Users are only presented with a "Are you sure your upload is not a duplicate?" message.
Currently we're just doing a `WHERE BIT_COUNT(images.phash ^ inputHash) < 12` in MySQL over 400k rows, which still works reasonably well (~200ms) given that it can't use an index for the XOR/BIT_COUNT operation. To my knowledge there's no way to speed up this query in MySQL, so if we continue to grow we probably have to write a small daemon that is able to search hashes more efficiently.
Hmmm...400K rows seems a bit small for this, but you might be able to build a (FLANN) based vocabulary tree that turns phash into a more stable signature (that makes use of database indexes). Then your SQL query would be more like: "where (images.phashsig=inputsig)". Phash might need a tweak, to output 64 floats instead of bits. But it would be more robust to the "random bits that flip" problem.
It's similar to what an engine like TinEye does, but instead of using a bunch of SIFT/SURF/etc features to do numerous visual word lookups, you'd just put in (a modified?) phash and get one word out.
The "video" quality is pretty bad though. From what I can see, the app streams base64 encoded JPEGs via WebSockets and draws them in a Canvas. The overhead of sending complete images costs a lot of bandwidth.
We've actually experimented with jsmpeg for this type of thing. The problem we ran into was that jsmpeg seemed to require every frame from ffmpeg. If the browser or network slowed, the feed would fall behind the simulation, leading to several seconds of latency when interacting.
We'd love to use it and appreciate your hard work though. Do you have any ideas on how that problem can be solved?
There are two sides to this problem. For jsmpeg (or even in the server), it would be possible to simply wait for the next intra frame (a frame that doesn't require previous frames to decode) on certain conditions and throw away all other frames in between. This will give the decoder and network time to catch up. By default ffmpeg emits one intra frame per second, iirc.
The more difficult part is probably detecting this lag. I'm not sure how this could be done. Maybe the node websocket server (ws) can detect congestion somehow.
Thanks. Maybe our ffmpeg settings weren't creating intra-frames somehow (or we happened to drop every single one) because the stream would never recover. I'll probably try it again at some point soon. We really want jsmpeg to work.
we did some testing using compressed video, but ended up finding single frames to be more performant. also, the setup can handle much higher framerate and quality but we've kept things fairly lean for the demo :) can easily handle 15fps w/ high quality images. thanks again for the questions and comments.
I recently made a Wolfenstein like 3D shooter with HTML5/WebGL, called Xibalba. It's based on a 3D Plugin for the 2D Game Engine Impact that I released as open source. With this, it should be really simple to make a similar game.
Also, in case you're interested, I created a making-of video for Xibalba that explains the tech and level editor a bit.