We were frustrated by the experience of reading machine learning papers on screens (particularly phones/tablets). There are lots of good tools for authoring HTML papers (Distill, Authorea, etc) but nothing that deals with the vast number of PDF papers that already exist.
So, we built Arxiv Vanity: a site that renders Arxiv papers as web pages. It’s still pretty janky, but for the papers that do render correctly, the experience is so much better than reading a PDF. For example:
One of the things that I came across when writing my own janky pdf/latex->html converter for lecture notes[0] is that Pandoc doesn't handle references and subfigures correctly, even with pandoc-crossref and pandoc-citeproc enabled. I had to write a little python module[1] that used regex to extract those and then handle them on my own separately... This is definitely something you should look at.
I was looking for a way to turn my (soon-to-be-defended) PhD thesis into an epub, and investigated the various LaTeX2Html converters.
I was pretty disappointed when I realized that all of them are terrible and have no hope of handling my manuscript. My current solution is to create a rendering of my thesis in a5 format. :/
This look quite a bit better, so here is the question: what do you not support at the moment?
One final thing, and wildly off topic, is that when you do your defense, remember that you probably know more about the specifics of the subject than anyone else in the room. Many folks stress over it, but you're almost certainly going to be the actual expert in the room. Good luck!
A lot of things. LaTeX and its packages have so much surface area. Our approach so far is to just make the papers that we read readable. That probably covers the 20% of LaTeX features that 80% of people use.
I'm not sure I understand. Well, I understand what you're doing but I'm not sure why you'd dislike PDF.
PDF has the great benefit of rendering the same on every system. With very few exceptions, PDF will look exactly the same on every system and will print the same on every system.
HTML doesn't really have that same benefit.
Don't get me wrong, I think your service is a great idea for those who would like HTML formatted results, but I'm not understanding the complaint about PDF.
Pdf pages are usually based on A4 size which is 210mm wide. Even at full size the writing is often tiny. Once rendered on a 10cm wide screen (landscape) it's pretty darn hard to read.
Also in general the mobile pdf reading experience sucks.
For example you have to download a file (rather than browse to) on Android and the hunt it down to open it.
The pdf readers I've used easily accidentally scroll you to a random page if you make a mistake in where you touch the screen. Kindles probably the best but then you have to email yourself the pdf which is a hassle.
IMO, Reading two-column papers on an iPhone (through PDF) is a real pain -- IMO the format relies on you using your eyes to jump from bottom left to top right, rather than having to scroll from the very bottom to the very top (diagonally). Same problem even exists for single-column styles -- you need to zoom in so much that you have to scroll horizontally as well as vertically.
The need to scroll doesn't exist on a large screen or on a piece of A4, but on smaller devices like mobile phones or even tablets, it's annoying. Having a responsive page means you can scroll vertically as you read, rather than having to make a big jumps (or constant horizontal scrolls) that can really break the flow.
I wonder if that's a personal thing? Over the past year, I've been trying to join the mobile revolution - sort of. The majority of my browsing is now done on a tablet.
I read quite a few PDFs and don't actually have any complaints. I am not personally seeing any readability issues and don't mind consuming PDFs at all.
That said, I think I now understand your complaint. Thanks! I just don't personally have any trouble with it. I use multiple tablets, of varied sizes, and I've had good experiences with all of the devices. While some PDFs are horribly formatted, I find that the device choice doesn't help that and it's a design choice from the author.
> The majority of my browsing is now done on a tablet.
Reading PDFs on a tablet isn't too bad because of large screen real estate.
Reading PDFs on a small mobile phone requires me to zoom in to make the font big enough for me to read, and then I have to scroll right to read, and left and down to move to a new section of the column.
Try reading a PDF on a smaller device than a tablet. I'm sure you'll be able to see what we mean.
I consume almost all my media on phone. The problem with pdf is precisely that it renders the same on every screen - this makes most PDFs virtually unusable on the phone as you have to scroll down one column in a page, then up for the second column etc.
I think part of the issue is that PDF isn't responsive to the size of the device. A PDF is not much more than an image from the perspective of layout. I'd love to be able to reflow text from a PDF such that a single column fills my screen edge-to-edge and scrolling allows me to advance through the paper, as opposed to requiring me to reposition the viewport every time I reach the end of a column. I know this isn't the purpose of PDFs, and I love them in different contexts where layout (including typography) does matter to me. But I also really want to be able to easily consume papers in a way that isn't constrained by the PDF layout.
Fwiw, this conversation greatly varies depending on who is doing the reading -- the rather banal fact is that the average 25-year-old student has much different ability to screen-read than the average 60-year-old professor (or a 60-year-old student, for that matter) :)
so no need to search tablet specs for the culprit. PEBCAK :)
The screen of a tablet is large enough to display a PDF. But PDFs are split into pages. That's perfect with paper, where we flip pages. It's very unnatural on screens, especially touchscreens, where we use vertical scrolling to move around.
Then there are minor issues of margins, possibly zooming to make text readable, etc.
That's why PDFs are so bad on mobile. The ideal format is one column text, figures and tables between paragraphs of that column, no page breaks, bidirectional links to notes. That's HTML, I guess.
Yeah fair point, I use an iPad mini. But I have heard similar complaints from older folks (40+) who have full-size iPads. I think much of it stems from dual-column printing, which is just kind of antiquated/annoying on digital.
Yup. I agree. I even find the experience fantastic on the original iPad, as well as a brand new one. I find it just fine on my phone, which isn't nearly as large a screen.
I am guessing it is an individual taste thing. That makes some sense.
Not really, look at the myriad rendering issues between the most popular browsers. PDF should result in pixel-for-pixel reproducibility, browsers don't do that in practice.
That's why we still test pages in different browsers and end up using browser specific code to ensure proper rendering - which often only reaches the 'close enough' format.
I understand the problem with a phone, but PDFs on an ipad/tablet are beautiful and a joy to read. Much better to read the text as originally typeset than to put it through a process such as this which risks corrupting minor but important details in the mathematical content.
On my phone I put it in landscape mode and that allows me to read a PDF OK, but I don't really get why one would read academic papers on a phone, why not use a tablet?
However I'm very interested in engrafo. It sounds like it will allow me to automatically publish blog style content from my LaTeX sources without having to fork the LaTeX content into a markdown / HTML version.
I just don't understand why you don't like reading academic papers as PDFs on tablets!
I actually made one quickly, published here: https://chrome.google.com/webstore/detail/arxiv-vanity-plugi... . It injects the arxiv vanity link on abstract pages and if you click the button when viewing an online arxiv pdf it opens the respective arxiv vanity link.
bioRxiv doesn't expose LateX files, they explicitly only use PDFs to make things easier. Which means you're going to need to reflow PDFs (a la https://docushow.com/), and I would guess there are a lot more edge cases there
Thanks for linking to https://docushow.com
Also a work in progress, but PDF reflow is a hard problem so you never ship if you want to solve all cases :)
Your solution using the LaTex source generates really nice HTML, congrats!
In all three cases I find the original PDFs more pleasant to read. HTML typography is not up to snuff. I read them on a laptop, however, and I can see that this would be useful if one is forced to read on a phone.
(One thing that is very ugly in the PDFs, and most scholarly papers, is the use of different-colored boxes for hyperlinks. Authors, please consider putting
For me the PDF was fuzzier https://imgur.com/a/WJ5y3 and the HTML version was more convenient to read in a single column. The two-column format is nice if I'm skimming to see if a paper is going to be interesting, but when I sit down to read it the HTML version definitely wins.
This is really cool and has a lot of potential. Academic papers are dense and heavily cross-referenced, so experimenting with new display formats that do more to help the reader could make researchers a lot more productive. For example, citation tooltips are a big time saver compared to cross-referencing the bibliography. However, it's also beneficial for every paper to look the same because this makes skimming easier. To get both innovation and consistency is to develop tools, like Arxiv Vanity, that automatically transform the source document. This example makes me hopeful that we'll someday have similar tools for the commercial publishers' papers.
As for immediate tweaks, I tentatively suggest making the text 100% black (like the original PDF) instead of rgba(0, 0, 0, 0.8). The higher contrast will help those of us with less-than-great eyes.
This is amazing. I hope you'll keep working on it. There's always a long tail of details that need taking care of when trying to cover a large corpus, and ploughing through successive 80%'s is (as you are no doubt acutely aware) serious grunt work. But you've made a fabulous start, so I hope you find the stamina to do it!
Yeah, even building upon Pandoc's LaTeX parsing, 3 months of grunt work got us this 20% working. Over the next 12 months we'll get the other 80% working. :)
A big challenge is to get references working correctly.
LaTeXML is quite good at converting latex documents to html [1], including references such as Theorem 2.1, equation (8.1) etc.
For instance, the paper [2] appears to be quite readable on mobile, and clicking/tapping on a reference such as (8.1) leads you to equation (8.1) as you would expect.
The auto-generation of Arxiv-Vanity is really nice, maybe it would be easy to add the LatexML output too?
Only issue I've run into so far is that cross-references to theorem numbers don't seem to always work correctly, e.g. you'll see a lot of "Theorem ?" in https://www.arxiv-vanity.com/papers/1607.06711/.
That said, on cursory look, this is pretty impressive. latex->web converters have existed for a long time, and this appears to have navigated some aspects quite well!
This looks really cool. (The program had some issues with the bibliography and with custom layout, but other than that, was great.)
It would be nice if an option to output MathML existed.
(Why MathML?
In brief, it allows treating Maths as a first-class citizen on the web.
For instance, with MathML the reader can choose what font the equations will be rendered in — if you prefer STIX or Latin Modern Math, then you can specify it with CSS, and the browser will correctly render it. With the mash of spans within spans that arXiv-vanity uses, you couldn't change the font, as then the pre-calculated spacings would be wrong. (Alternatively, the publisher could easily offer several styles, without having to re-render everything, just by changing the CSS.)
Arguably, client-side MathJax offers the same flexibility as MathML, but it's much, much slower, while rendering MathML in firefox is as fast as rendering standard, static HTML.
Another application of MathML is embedding it in SVGs for beautiful graphs.
MathML can also be pasted into other applications that support it, such as Thunderbird and Mathematica.
)
Looks like it's failing to process some standard tex commands (e.g. \textup) as well as some user defined macros. See the many display errors in https://www.arxiv-vanity.com/papers/1710.07406/
Of course, it goes without saying that I want this.
When the render fails, why are you redirecting to the pdf file intead of redirecting to the abstract? E.g. here (link stolen from another comment in this page) https://www.arxiv-vanity.com/papers/1608.04012/
Noob question but how far does Calibre take pdf to epub conversion? I've been really interested in learning more about the epub file format and was greatly intrigued to discover it extends xhtml and is essentially a zip folder if I've gathered that much correctly.
Those problems are normally Pandoc parsing errors. Considering it's open source, perhaps we should print the error message so people can actually help fix it...
The MathJax failures are either things that MathJax doesn't support, or use of \DeclareMathOperator which we haven't added support for yet.
Thank you thank you thank you! I detest the reader-hostile PDF (and WTF? why would you write something and then make it inconvenient to read???)
Unfortunately, among its sins, PDF discards a lot of the presentation semantics (headers, footnotes etc). Congrats on doing a credible job trying to reconstruct some of that! It's a tough, thankless job.
I was horrified when Adobe introduced PDF and indeed it has turned out at least as badly as I had feared.
I believe it's reconstructed from the Latex source, which is how every paper is submitted to the ArXiv. Not to diminish this site but I'm guessing that generating HTML from Latex is a lot easier than doing it from PDF format.
This is awesome ! Going on my home screen now. Love the design. Maybe you could ask Arxiv to have a button on their site that would direct it so that it opens on your site .
PDF is the only format that will preserve the typographical details that are important in many technical papers; it also avoids the relatively bad rendering created by the browsers.
PDF is usually bad, of course, on small screens, unless the publisher makes special versions.
I think this is a dramatic difference of opinion between coders and everyone else in science, if you excuse the generalization. I say that because of the large number of tools to try to one-up the PDF for academic scholarship (e.g. Readcube). Publishers looking to invent ways to define their own value have been slowly trying to force these things onto readers, but almost everyone in my field hates them and wishes they would die. Formatting and typography are critical in many fields, and a PDF is the canonical way to maintain these aspects of a paper.
Definitely useful in certain situations. Can't comment on how well the conversion works (yet) but I can see how this might be useful to a lot of people.
Me? I still mostly prefer reading physical academic papers because of needing to flip back and forth for re-reading (clarification) and adding personal notes/graphs/calculations.
Tried three papers: "Could not find Arxiv ID in that URL. Are you sure it's an arxiv.org URL?"
Why, yes, I copied-and-pasted directly from the browser tab showing the Arxiv URL.
Tried a couple other papers: "This paper failed to render. Take a look at the original PDF instead."
So...with what probability does this actually work?
LaTeX is really tricky to parse, which is why you're seeing those "failed to render" errors. Judging from our logs, it works about ~80% of the time. That's up a lot from plain Pandoc though - it could render hardly anything from Arxiv.
For bitmap based PDFs it would be possible to segment the document into words and images (just bounding boxes, not OCR), then "reflow" them to a different page size, by allocating less words per row.
Does anyone know if this kind of PDF reader exists? Such a PDF reflow reader would work on scanned old books.
Hallejuhah, we need to force academics and people who go around touting "whitepapers" to be ripped out of the proprietary PDF era of terrible UX/readability and into modern readable web documents. This is definitely the way to do it, this is brilliant!
I would pay at least 10$ for an app that made an aesthetically-pleasing HTML flow of any pdf document I’m looking at. At least. And I can’t imagine I’d be the only one. So much of my reading is on-screen now, and PDFs do kind of suck for two column reading.
It definitely feels like a one time payment, since my mental model for it would be “pdf reader.” Also, since I’m accustomed to SaaS in my business dealings, but detest it in the private consumer space, and this would appeal to me as a consumer; my business documents are already mostly PowerPoints, with the occasional word doc.
You might add a contact link so people can easily get in touch. The first-listed personal homepage is kind of insane with all the flashing colors, and it contains dead links (e.g., your Twitter).
Shameless plug: I made an Android app for arXiv if anyone wants something simple to search articles on mobile. Graduating soon so if you try and enjoy it, any positive (but honest) views help the looming job search ;)
I have to admit I am not impressed. My first paper, which I tried to render, does not work properly; references are removed and rendered poorly, figures are misplaced and tables incomplete. Given that not all arXiv papers are under a permissive license and you do not have permission to do this, I would much prefer if you at least made sure that arxiv-vanity rendered papers do not show up in search results, e.g. by offering a suitable robots.txt and with a bigger link to the author-endorsed version of the paper.
Edit to clarify: If people want to use or develop a broken sort-of-PDF viewer, that’s fine. However, if someone searches for a paper of mine, I would like them to only find the version where I at least had a chance to see that it renders correctly and is complete. In particular, I do not want to be "responsible" for broken rendering on random third-party websites. This website actually operating illegally does not make me more inclined to support it.
"Some papers do not render correctly, for example figures and tables in mine [1][2]. Beside that, some articles may not be posted under a permissive license, so you want to double check that you're not running into copyright violation troubles by modifying or publishing them.
This is fair. If somebody stumbles across this thinking it is how you intended it to be displayed, I can understand you'd be unhappy. We should make it clearer that we're just a conversion tool, not a source.
If you want us to remove your paper and just point at the PDF, we're happy to do so. My email's in my profile if you don't want to post the broken render here!
Thank you for your reply. Ideally I’d prefer for you to respect the license associated to each paper and only re-compile and re-host if the license actually allows you to do that (i.e. CC0, CC-BY, CC-BY-SA and maybe CC-BY-NC-SA, depending on whether you think you act commercially).
I also don’t want to keep tabs on every arXiv rehoster and inform them manually by e-mail every time a new paper goes up.
May I ask why this was not done together with the arXiv itself? I.e. have the infrastructure run there, let authors check the HTML render at the same time as the PDF render and then, if the author thinks they look ok, have them show directly on the abstracts page? This would even avoid all your license problems, as the arXiv already has the corresponding license!
I believe in most countries only a court can decide if a site is illegal or not. Not you. And as far as I know this is true in both France [0] and Germany[1].
Each arXiv paper has a well-defined license linked-to from the upper right hand corner "(license)" link below the PDF and source downloads. If they only re-hosted and re-compiled papers for which they have a license to do so, I wouldn’t complain at all, but re-hosting and modifying content without a valid license is clearly illegal, no?
> re-hosting and modifying content without a valid license is clearly illegal, no?
No.
Longer version: it's illegal only if a license is required, which is a matter of the copyright law of the jurisdiction relevant to the act. In the US, that question may turn on things like fair use analysis, which can be tricky.
Fair use applies to citations (not of the whole work), parodies and similar creative processes. Similar requirements in Germany include some creative input by the person claiming fair use which usually should exceed the creative content taken from the original work. Simply re-compiling the LaTeX source is certainly not creative work sufficient for a fair use exception. Checking the other limits of copyright law in e.g. Germany ( https://de.wikipedia.org/wiki/Schranken_des_Urheberrechts ) nothing remotely applies to this site.
Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
> Fair use applies to citations (not of the whole work)
Time-shifting is one of many examples of where copying a whole work was found to be fair use; the idea that fair use applies only to citations is very, very wrong.
Fair use is extremely precedent dependent (and very hard to predict without clear applicable precedent) because the statute law gives only factors to weigh in the analysis.
> Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
I didn't state an opinion on that; I said that, because it skips the question of whether license is required, the blanket statement that rehosting without a valid license is “clearly illegal” is inaccurate and overbroad.
Why the hate for PDFs, I can see why you would not want to communicate dynamic content with them, sure. But for academic papers, which are usually static text and equations and maybe some plots, PDFs do exactly what you want: At least in principle they look the same on every system.
If I want to email them to somebody, store a copy, read them offline on a reader etc. I have to deal with one single file, in a format that just works on many readers. What am I gonna do when I typeset a paper with webtechnology ... fire up a local server to display some inline math in my html page .. no thank you.
I agree that maybe layouting based on physical paper is maybe not ultimately necessary, but it gives the reader a familiar structure. The way the advertised web site is transferring the papers into a long scrolling list of text ... I find it rather disorienting and unstructured. Text that is split up into "pages" (whatever size they are in the end) somehow helps break up the reading flow.
In the end it remains to be shown that the gain from having academic papers not typeset in PDF outweighs the hassle of having to deal with non-standardized ways of rendering properly formatted text on websites (thinks like MathJax etc. do not support everything that is available in full LaTeX etc.).
But anyway, in the context of the discussion about this webpage/project, it's not relevant to ask why these PDFs exist. They do, and the scientific community is nowhere near a transition away from them. So bfirsh is trying to find a solution to consume those existing PDFs.
So think as not getting downvoted for expressing your opinion, but more for not contributing to the discussion about this particular project.
So, we built Arxiv Vanity: a site that renders Arxiv papers as web pages. It’s still pretty janky, but for the papers that do render correctly, the experience is so much better than reading a PDF. For example:
https://www.arxiv-vanity.com/papers/1705.04085v3/
https://www.arxiv-vanity.com/papers/1708.00884/
https://www.arxiv-vanity.com/papers/1705.06031v2/
The source for the LaTeX to HTML renderer is on GitHub[0]. It’s built on Pandoc[1] and Distill.pub’s template[2].
[0] https://github.com/arxiv-vanity/engrafo
[1] https://pandoc.org
[2] https://github.com/distillpub/template