A few years ago I did some image work in the browser and made a little site that converts an image into a cross stitch pattern.
A few things that I learned from that which might interest you:
1. using web workers is how you keep the UI responsive while still handling some processor-intensive work. But until "OffscreenCanvas" becomes more well supported, you are limited in what you can use. Most of your methods look like they could be adjusted slightly to not reference any DOM stuff and could work entirely in a worker without much extra work!
2. along with #1, if you are going to send image data to a web worker, use the "transfer" property of the Worker.postMessage() function to zero-copy transfer the image data over to the worker. If some transformations are easily parallelizable, you can even pretty easily split the image into chunks and send a different chunk to each worker, and then splice it all together when done. I really enjoyed writing this part, something about everything just falling right into place and all the workers running completely separately from one another only to put it all together all nice and neat felt really cool to accomplish!
3. for some extra perf, take a good look at the whole spec for TypedArrays in javascript. Specifically how you can create a "view" into an ImageData array as different sizes. Instead of an array of uInt8's you can have an array of uint32's where each "element" is a whole pixel (4 uint8 treated as one big number). It doesn't "change" the underlying buffer, but just gives you another "window" into it. You can easily have both a `uint32array` and a `uint8array` to the same data, and do operations on both of them. This suprisingly gave me a nice speed boost (IIRC something like 10-20% faster), even if I just iterated over the uint32 array and then broke it up into the 4 pixels first thing in the loop. Some of your functions like `transpose`, `reverseRow` and `reverseColumn` could probably see pretty significant speedups if you wrote them to work on a uint32 "pixel" rather than needing to call it 4 times (once for each uint8 color channel).
4. along with #3, look into using the bitwise operators in javascript with the uint32 numbers. By (ab)using the bitwise operators on the typedArray numbers directly, you can get pretty close to "true" integers in javascript (IIRC they get a bit weird when you try to store them in normal variables! It could have just been me not knowing enough, but just use caution here if you run into 'strange' things happening). It's pretty cool to be able to kind of work with the "whole" pixel at once. I have a hunch that the `contrast` function could be rewritten to work on the uint32 representation of a pixel, again giving you a nice speed up and in some ways reducing the complexity of the code.
5. I never went very far into this, but maybe look into WebGL if you are inclined at some point. It's going to feel like cheating! Operations which would take a few seconds on a CPU can take literally milliseconds on a GPU, after all they are literally purpose-built to do this! It's a whole other world, and basically a whole other language, but it could be a cool place to take this, especially if you are interested in the WebGL world!
If you want, the source code for my stuff is at [1]. It was a silly little project I did with polymer (pre 1.0!) that I can't get to compile any more (mostly because a lot of the build system was SUPER scraped together. At the time I hadn't worked with webpack much and didn't want to muck with it for this project, so all the scripts are just concatenated with gulp, and there are no imports or requires...). Don't judge me too much for most of the code! I was trying a LOT of different things in this project since it had no real risk, and most of it turned out to be a bad idea!
That being said, the real meat of the image processing stuff is located at [2]. Specifically the BitPacker.js and sizor.js will probably be the most interesting to you.
excellent information ^^^. i also have been diving deep into canvas-based image fiddling lately and you covered most of the good bits. the only thing i'd add is maybe look into the `ndarray` library: https://github.com/scijs/ndarray for efficient manipulation of multidimensional arrays. this could help slice and dice by channel, or crop, flip, rotate, et cetera. that plus convolution and other fun stuff.
Oh man, thanks a lot for the great response. I have consider WebGL and WebAssembly, but never had implemented nothing on this project. But thanks again for the material, I'll take a look at it as soon as I'm done implementing a function someone requested me.
A few years ago I did some image work in the browser and made a little site that converts an image into a cross stitch pattern.
A few things that I learned from that which might interest you:
1. using web workers is how you keep the UI responsive while still handling some processor-intensive work. But until "OffscreenCanvas" becomes more well supported, you are limited in what you can use. Most of your methods look like they could be adjusted slightly to not reference any DOM stuff and could work entirely in a worker without much extra work!
2. along with #1, if you are going to send image data to a web worker, use the "transfer" property of the Worker.postMessage() function to zero-copy transfer the image data over to the worker. If some transformations are easily parallelizable, you can even pretty easily split the image into chunks and send a different chunk to each worker, and then splice it all together when done. I really enjoyed writing this part, something about everything just falling right into place and all the workers running completely separately from one another only to put it all together all nice and neat felt really cool to accomplish!
3. for some extra perf, take a good look at the whole spec for TypedArrays in javascript. Specifically how you can create a "view" into an ImageData array as different sizes. Instead of an array of uInt8's you can have an array of uint32's where each "element" is a whole pixel (4 uint8 treated as one big number). It doesn't "change" the underlying buffer, but just gives you another "window" into it. You can easily have both a `uint32array` and a `uint8array` to the same data, and do operations on both of them. This suprisingly gave me a nice speed boost (IIRC something like 10-20% faster), even if I just iterated over the uint32 array and then broke it up into the 4 pixels first thing in the loop. Some of your functions like `transpose`, `reverseRow` and `reverseColumn` could probably see pretty significant speedups if you wrote them to work on a uint32 "pixel" rather than needing to call it 4 times (once for each uint8 color channel).
4. along with #3, look into using the bitwise operators in javascript with the uint32 numbers. By (ab)using the bitwise operators on the typedArray numbers directly, you can get pretty close to "true" integers in javascript (IIRC they get a bit weird when you try to store them in normal variables! It could have just been me not knowing enough, but just use caution here if you run into 'strange' things happening). It's pretty cool to be able to kind of work with the "whole" pixel at once. I have a hunch that the `contrast` function could be rewritten to work on the uint32 representation of a pixel, again giving you a nice speed up and in some ways reducing the complexity of the code.
5. I never went very far into this, but maybe look into WebGL if you are inclined at some point. It's going to feel like cheating! Operations which would take a few seconds on a CPU can take literally milliseconds on a GPU, after all they are literally purpose-built to do this! It's a whole other world, and basically a whole other language, but it could be a cool place to take this, especially if you are interested in the WebGL world!
If you want, the source code for my stuff is at [1]. It was a silly little project I did with polymer (pre 1.0!) that I can't get to compile any more (mostly because a lot of the build system was SUPER scraped together. At the time I hadn't worked with webpack much and didn't want to muck with it for this project, so all the scripts are just concatenated with gulp, and there are no imports or requires...). Don't judge me too much for most of the code! I was trying a LOT of different things in this project since it had no real risk, and most of it turned out to be a bad idea!
That being said, the real meat of the image processing stuff is located at [2]. Specifically the BitPacker.js and sizor.js will probably be the most interesting to you.
[1] https://github.com/Klathmon/stitchpics
[2] https://github.com/Klathmon/stitchpics/tree/master/app/eleme...