Years ago I once wrote a little tool in Java called bookbuilder, where you could turn the pages manually, make a photo and then run an automatic process on all images to build a searchable pdf.
I used https://boofcv.org/, an impressive Computer Vision library in pure Java, still exists and it is pretty fast, too.
It was able to detect the page contour, deskew it, flatten the image and remove finger contours by matching the skin tone, then build a PDF with integrated invisible OCR Layer without any user interaction. I remember that I was working on line slope detection with some kind of watershed algorithm to improve the flattening part.
Fun project, I wonder if I have the source code laying around somewhere... even the download page is gone today. This was long before I went open source with all of my little side projects, because I never thought it could be interesting for someone else :-)
RE ....There is a whole community about this topic at: https://diybookscanner.org/..... The actual f(book scanner ) forum is down at present. It has been down for a few weeks . We are working to get it back up.
We need a little help bringing the forum back online. If anyone wants to help and has some solid PHPBB experience, send me an email - danreetz@gmail.com
The store seems to be down as well. Also some of this hardware is quite old such as the model of digital cameras used. Do we still really buy old point and shoot cameras? Would something like pixel phones and their excellent cameras do better in 2023?
I've been interested in book scanning for a while. I think I remember seeing your software! I built one of the early versions that I saw on the site, and then happened to meet Jonathon Duerig and do some waterjet cutting for him a few years ago.
When I saw the linear bookscanner the first time, I realized it was upside down. If it were suspended, and able to move itself, then all the issues about not knowing the mass of the book, and dealing with friction could be avoided. Counterweights could keep the force constant, and it would be moving a known mass when scanning any book.
BoofCV is incredibly cool, thanks for sharing! It's unfortunate I didn't know about it before today, I would've definitely invested time to learn and hopefully contribute back (there's still a chance, but at this point it seems to cover everything I've ever heard of, and more).
I just spent 30 minutes clicking through and inspecting every example.
There‘s a somewhat similar commercial scanner [1] [2], with a V-design as well but inverted to scan from the top. Much gentler on the books as it‘s the scanner that moves, not the books themselves. Super happy to see someone develop an open-source alternative!
It's probably expensive even to lease/rent, or to use their digitization service. Why? Because they have no pricing info available for those two options either. Just an inquiry form to fill out.
Um,I'm not sure this is a great alternative, as this scanner chops out a page for every scan. After that, I don't think it really matters how gentle it is...
Unless you're concerned with binding the pages to form a new book. I think that would be possible with the leftovers.
You're right! My mistake. It looked like there was a sort of deli-slicer blade and suction to remove pages, but, looking again, it's not what's happening. That's what I get for posting pre morning coffee...
I love the idea, although the risk of torn pages is mildly concerning for archival purposes or valuable books. Though if that were the case, I'm sure scanning by hand would be preferred anyway. I've often wanted a device like this for the purpose of digitizing my excessively large collection of books.
Regarding frequency of torn pages in the FAQ:
> Prototype 1 could scan the majority of books without damage, but may tear one or two pages in some books. Out of 50 books tested, 45% had one or two of their pages either torn or folded. This is a very early prototype and there are many areas for improvement in the design.
In my opinion, this is mostly acceptable. Especially if a future revision reduces the 45% to somewhere around the ~10-20% range. If I had the space for a device like this, I would definitely consider building one.
For a while, the Internet Archive built a book scanner that rests the book in a V-shaped cradle. A volunteer turns pages by hand and lowers/raises a pair of glass panes that gently press the pages for imaging by a pair of DSLR cameras on an angled mount. The whole assembly isn’t automatic, but can be easily operated by hand.
Interestingly the Internet Archive copied the open source design (of their scanner) from the site https://diybookscanner.org/ ( as they are allowed to by its open source licence ) The internet Archive then effectively refused to release any details back to the community. After a lot of “pushing”, the Internet Archive did acknowledge the source their design was based on came from the site bookscanner.org. This would have been very disappointing, in my opinion for the designer of the scanner – who released the info open source. At one time Internet Archive sold this scanner to organizations for $10K I think the price has dropped now ( I think ) to a few thousand.
I'm not sure if one of those machines was the cause, but I've seen far too many old books on archive.org which have pages that appear to have been torn by the scanner; thus I doubt they're manually doing it.
I made the cardboard scanner many years ago both to scan whole books as well as sections of books from the library. It worked pretty well but nowadays in practice I usually look into the open library at the Internet archive first. Hope it sticks around.
Ah yes I should have said something more like “in my usecase”, which is just to digitize ordinary texts that are still in print but that occupy too much space in my home to justify keeping!
This is a problem domain where software hasn't caught up with what is possible, so people do in hardware what could be done in software.
With two or more photos or a stereo image (new iPhone?) one could triangulate to infer a flattened page, and produce images that look like they came from cut pages in a flatbed scanner. Now just pay someone well in Ethiopia to carefully turn pages without damage.
As any researcher can attest, our digital libraries now hold a century of scanned work of questionable quality. AI could infer scans indistinguishable from an outline font format original on an 8K monitor.
I once helped consult on the 1980's font wars, turning old formats and digital scans into Postscript and TrueType fonts. This was hard then, but will soon be understood as the "correct" way to scan text, when software catches up.
For the scientific literature, we need a ChatGPT equivalent to reconstruct LaTeX source that can reproduce each page. (We really need a successor to LaTeX that isn't such an arcane language, and can author fixed and flowable text with equal ease.)
This is very much a physical problem and there aren't too many shortcuts you can take.
I've been part of a preservation project and scanned a LOT of magazines. Generally, it doesn't matter much if you place it on a flat bed or not. Flipping the page manually takes a while either way.
There's already a known way for scanning magazines and books very fast: cut the spine and feed the pages to an automatic scanner. This is of course not applicable to anything you'd like to keep around after scanning, because your copy is destroyed.
All in all, the best way to automate scanning without destroying the item, will have to combine a top level camera with a machine to turn pages. I believe this is what was going on in Google's massive scanning project.
Maybe using x-ray could work for "scanning" some books without having to turn the pages. But I suppose there'll be a new set of problems to solve there.
I recently saw some work from the University of Kentucky on reading the Herculaneum scrolls. These scrolls were carbonized by volcanic activity (Pompeii?) and obviously can't be unrolled without disintegrating. They used some interesting CT (xray) scanning plus machine learning to distinguish the carbon-based ink from the mostly carbon substrate and retrieve legible text.
Of course, that only gets you the printed text. You might lose notes and doodles in the margins, or other physical evidence. But, it's certainly promising for works that are too delicate to physically open and inspect
RE "....There's already a known way for scanning magazines and books very fast: cut the spine and feed the pages to an automatic scanner. This is of course not applicable to anything you'd like to keep around after scanning, because your copy is destroyed......" I've always thought the pages could be rebound. Not perfect but a halfway solution.
Most signatures (https://en.wikipedia.org/wiki/Section_(bookbinding)) are glue bounded, not just sewn. Some use cases prefer/require the pages to be unbound because the printing goes all the way into the gutter and cutting the spine can also leave out some data. It's highly inneficient as you have to heat the spine carefully and then remove the glue residues. A tiny glue leftover can smear your autofeed scanner if not completely jam and tear the page. For a unique item, makes sense using a non-destructive scanning method, but for anything else, a carefully cut spine (or better yet, a bookbinding plow https://duckduckgo.com/?t=palemoon&q=bookbinding+plow&iax=im... ) leave a perfect cut and the loose pages can be kept in a ziplog bag for any future reference.
I guess you could, but everything where you would even consider cutting the spine is probably not worth fixing afterwards. E.g. it's a contemporary magazine, where you could just buy 2 if you really need to keep a physical copy.
How is setting up infrastructure to exploit third world labor a "software problem" exactly?
I think the problem isn't that software can't do de-warping well, it's that by the time you set up everything for book scanning you might as well use a setup that doesn't need it.
"Exploitation" is an arguable word, when without outside intervention there would be no employment above a few cents a day. Any employment by those who can pay even a dollar a day could instantly ratchet an entire family out of abject poverty.
You've also got to wonder about idea behind it that it's supposed to be trivial to send a large number of books to Ethiopia and back without damaging them.
There's also a feature where it tells you to turn the page, detects that it has been turned, takes a photo, etc. And in the background it flattens, splits into pages and OCRs the photos. With a little practice you can scan and OCR a whole book at 1-5 seconds per page.
Aren‘t you losing information in the parts that aren‘t perfectly straight? Yes, you can stretch those to recreate the original layout, but that would come at the cost of resolution in the interpolated sections of the page. Granted, not a problem for most books, but probably a reason prople are still looking for mechanical solutions to the problem.
I think the suggestion is that with AI you can interpolate to the actual letterforms, not to pixels.
Working a typical volume the letter “e” will appear hundreds of times and be identical, so there should be lots of data to help resolve ambiguities in the poorer parts of images.
Not to mention data that can be used across volumes.
If the goal is to ultimately OCR then it's moot. But yes, of course information is lost.
That being said, modern phone cameras are going to produce "scans" above 300 DPI, and while 600 DPI or higher might be tricky they're stills possible if you take partial shots of a document, assuming you can focus that close.
What you lose in quality you make up with convenience, I suppose.
> For the scientific literature, we need a ChatGPT equivalent to reconstruct LaTeX source that can reproduce each page. (We really need a successor to LaTeX that isn't such an arcane language, and can author fixed and flowable text with equal ease.)
Check out Nougat: OCRing scientific papers with a deep net trained end to end. It was released by Meta a few days ago.
“PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents.”
What about very old books, or handwritten books, or books with images in it?
The only valid archival approach has to be taking a good photo / scan while the page is as flat as possible.
Every further processing can be done later based on that, as a separate step... as technology advances.
Also carefully FLIPPING THE PAGES is literally the whole problem. Everything else can be solved by lowering a glass pane on the book and automatically taking a photo from above.
> As any researcher can attest, our digital libraries now hold a century of scanned work of questionable quality
I keep thinking so much collaborative potential is not being utilized. Imagine each (unique) Google Book is basically an editable wiki where people can directly correct OCR errors as they come across them (with an associated Talk page where they can give explanations, etc)
The insurmountable problem is that Google doesn't have, and can't get the rights to distribute all these books. Even regular Google employees can't see them (and I tried when in Legal there).
So this is neither a hardware problem nor a software problem. Unfortunately.
You are right! I was really thinking more about books published before the 20th century (which constitute the majority of my own usage of Google Books), and I had in mind something like a collaborative WikiSource integrated into Google Books.
While they are certainly safer they aren't automatic and require someone to turn the pages, which this project attempts to solve, although it certainly should be done in a more gently manner as to not damage the books.
Just curious - Once we scan, we have all contents in digitized format. So, why unbinding a book to pages before scanning is not a scalable model? Is this to avoid additional work of unbinding?
Archivists typically don't like destroying a work when preserving it. Scanning a book also only gives you an optical image. In some cases, like medieval manuscripts, the pages may have been erased and written over. If we simply scan the book and destroy the physical copy, we've lost that evidence.
But again, for cheap mass-market works, most archivists probably won't care about destroying one copy out of a million to preserve the work. It's really only a problem for very old and very rare works
Unless the book is super valuable then it is often easier to just use a guillotine to slice off the binding completely and feed the pages through a sheet-feed scanner.
I also cringed at the look of this. I have a bunch of old books I would like to digitize, and most bookscanners I come across require you to at least spread open the book completely, which would, as you say shred them... And they don't even involve vacuum, movement across sharp edges, and stepper motors...
It definitely would. And probably some that are not damaged when something gets jammed or the environmental conditions change and react with the paper. But if you’ve got a mountain of books to scan and you can do the bulk of them with a machine like this and then use a more careful approach for the special cases, that’s a win.
Sure, but saying "Google did all the books" seems to imply there's no reason to make a scanner. But Google made this scanner specifically because they were interested in "doing all the books", even if they didn't end up using it for that purpose.
I am allergic to dust mites. After about six months on a shelf, I can't touch a book without getting an allergic reaction.
Simply dusting or even vacuuming the exterior doesn't work. This gizmo looks like something I can use to rid a book of dust mites so I can read it.
Of course this would be possible with DRMd books. One could, hypothetically of course, use e.g. some kind of script to automatically turn the page of the e-reader, screenshot it, then use image recognition to convert to epub, etc. to archive them.
tl;dr a diy system for scanning books. Basically, build a triangular prism with a special zig zag slit for a single page to snake through. Put a vacuum on each side to pull the paper through the slit. Put two optical scanners in the middle of the slit, to scan both sides of the sheet as it hangs down. Attach a motor to move a sled pushes just the top, then bottom, of the book. The site features six designs. Some include videos of operation [0].
In the fullness of time, maybe I would make one of these, given that I live in an apartment and not a house with space to construct/store this scanner. I check etsy every couple of years and haven't seen someone offer a kit. I use 1dollarscan, though they've had to restrict their offering as Pearson, et al notice their existence.
Years ago I once wrote a little tool in Java called bookbuilder, where you could turn the pages manually, make a photo and then run an automatic process on all images to build a searchable pdf.
I used https://boofcv.org/, an impressive Computer Vision library in pure Java, still exists and it is pretty fast, too.
It was able to detect the page contour, deskew it, flatten the image and remove finger contours by matching the skin tone, then build a PDF with integrated invisible OCR Layer without any user interaction. I remember that I was working on line slope detection with some kind of watershed algorithm to improve the flattening part.
Fun project, I wonder if I have the source code laying around somewhere... even the download page is gone today. This was long before I went open source with all of my little side projects, because I never thought it could be interesting for someone else :-)