Great tables has done some really nice work on python/jupyter tables. It looks like they are almost building a "grammar of tables" similar to a grammar of graphics. More projects should write about their philosophy and aims like this.
I have built a different table library for jupyter called buckaroo. My approach has been different. Buckaroo aims to allow you to interactively cycle through different formats and post-processing functions to quickly glean important insights from a table while working interactively. I took the view that I type the same commands over and over to perform rudimentary exploratory data analysis, those commands and insights should be built into a table.
Great tables seems built so that you can manually format a table for presentation.
Thanks for your work on Buckaroo! Jupyter print() and IPython display() have limitations given their dead static output and feels like printf debugging of yore, which I know Buckaroo was written to address.
What are your thoughts on Visidata's hotkeys and controls? I used Visidata in the past and always wondered why it couldn't be added into Jupyter (eventually) for dataframe explorations.
>It looks like they are almost building a "grammar of tables" similar to a grammar of graphics.
Agreed that Great Tables seems to be taking annother crack at formalizing a "grammar of tables", and I welcome this approach too given the power of tabular formats and wider adoption of the dataframe concept via the R/pandas/Arrows/polars ecosystem, although I believe the term was initially referred to in the 90s[1] from the statistical S language.
Buckaroo started as a lowcode UI with an accompanying table. The low code UI lets you click on columns and perform actions (drop, fillNA, groupby). The dataframe is then modified, AND python code to perform the same action is emitted. Controlling the lowcode UI through keyboard shortcuts should be fairly straightforward.
The other feature I have played with in this area is auto-cleaning. Auto-cleaning looks at individual columns and emits cleaning commands to the low-code UI. Different cleaning strategies can be implemented and toggled through.
Buckaroo takes the view that being opinionated is good, so long as you can toggle through opinions to get the right combination of cleaning, display, or post-processing that you are looking for quickly. All of the features of buckaroo are also built to be easily extendable by users.
This feature saw very little use, so I haven't developed it much (I had to disable it after some refactorings). The lowcode UI is demonstrated at the end of the youtube video linked above.
The example they show of a Great Table is, to my taste, way too busy. Here is my unsolicited opinion:
The top and bottom horizontal rules on the Title appear to be superfluous, and I dislike how it is aligned with the first column (row labels) rather than the second. I feel like a little space to breath at the bottom, along with a bold font would add visual hierarchy w/o the clutter.
The row label backgrounds are far too dark and the font weight makes it hard to read. I'd prefer a very light blue here instead. I don't like the row group label ("Name") being italicized.
The spanner labels floating in the centre make the table hard to scan. Would be much nicer aligned left.
Finally, I really dislike the font (maybe this is just my browser, though).
I mocked-up some of the changes here, I think this is a much easier to read table:
You might want to read Edward Tufte's Beautiful Evidence.[1] He discusses stuff like what you brought up about readability and distracting from the message / point of the data.
If you've seen sparklines, [2] Tufte coined the term.
Whenever I do a UI review I end up paging through it just to see if there's something we're not thinking about, and its an interesting book to just open to a random page and read.
Plus he has an entire treatise on why PowerPoint is terrible.
According to everyone that has ever told me how to do powerpoint presentations, they need an intro slide that tells the audience what they are about to be told in the presentation they are about to be told, and a conclusion slide that tells the audience what they were told in the presentation they were told.
> According to everyone that has ever told me how to do powerpoint presentations, they need an intro slide that tells the audience what they are about to be told in the presentation they are about to be told, and a conclusion slide that tells the audience what they were told in the presentation they were told.
Sorta? I guess that might be like driving. Hands on the wheel, gas on the right, etc. But what about driving in the rain (presenting to a cold prospect)? What about driving uphill (addressing a conflict)? What about driving offroad (explaining to an irate customer what happened?
There's storytelling and communication in each of those, but they're different.
See here[1]. It does have an outline, and it does have takeaways, but there's a a structure to the story in the presentation.
It opens with "3 parts" but immediately grabs your attention with item 0. I do that because I'm presenting to engineering students, and there's some CS/ECE folks in the audience and we count from zero. Plus its fun to shake people out of "here we go, another presentation" mode.
Each part has some interesting stories and plot points. The first explains moore's law, and advances in ML / DL. You can't tell from the charts, but there's a narrative that says 2011 to 2016, we saw this advance. There's a relentless march of technology. Isn't that interesting. Part two talks through real projects, and real outcomes, and the shift in value and complexity of delivery at scale vs "we got it working on a laptop." Part three looks at the future - and calls back to pt 1, and proposes a question - what rate do you think stuff'll advance? Here's my guess. Here's some resources. Keep the convo going. Thank you. etc.
It's an incredibly fun presentation to give, and people enjoy it as well. This doesn't have the basic structure you shared, but -- it kind of does. We come back to the "agenda" pages when transitioning between the 3 sections. It has a clear punch line. And it contextualizes what we're going to talk about.
It does this, though, without having a boatload of bullet points and outlines, and, looking at it with fresh eyes, probably makes no sense without a talk track, but people seem to like it and find it useful.
Table titles should be either centered above, or captioned below. Left-aligning them above any column instantly conveys a generally false/unintended impression of the title being a top level in the information hierarchy of the table. In the modernist makeover above I was immediately uneasy that the title stipulated “names, addresses, characteristics” whilst apparently aligned to exclude the names.
In contrast the census manual chooses to center almost all labels within their box, and when not it is almost always due to indentation, and moreover is unafraid to set column widths to fit the data not the labels, with indent and hyphenation to match. The result is both horizontally compact and intuitively comprehensible.
edit: on further reflection I also think it’s a crappy title. Titles and captions should convey context, scope, purpose - and may otherwise be omitted entirely for the editorial sin of failing to justify their own existence. As given, this one could be retitled “Table 1” with no loss of information or generality. For an article that’s trying to discuss and reformulate tabular presentation from first principles, that’s a tad disappointing. Since table titles form a crucial layer of their information catalogue, it is hardly surprising that the census manual devotes an entire chapter to the matter of title construction, and even though somewhat domain specific and archaically worded it is well worth the visit
Keep going IMO: shorten the title to remote correspondents since the rest is redundant with the column names. The blue highlight is now redundant with the title so ditch all of it. Personal characteristics vs location don’t meaningfully improve the organization so ditch those as well.
the white text on a dark background really was a glaring misfeature in the original example, to the extent that i wonder if the colours looked different on the author's monitor
This a good article with some fascinating history.
More recent history involves the production of CALS tables https://en.wikipedia.org/wiki/CALS_Table_Model. The company Datalogics https://en.wikipedia.org/wiki/Datalogics was heavily involved in the CALS table initiative. Datalogics staff was part of the ISO committee forming SGML, and trained many people on SGML, including DoD staff and their contractors involved with documentation.
I was involved with the team that produced an editor for SGML-based documents. It had as one of its features the ability to specify the formatting of an element based on the SGML context of that element. This was before XSLT and its kin.
Alumni of Datalogics helped Microsoft learn about XML ("No, you can't arbitrarily switch case on XML element tags").
Also TeX practitioners have pretty well-formed opinions about how tables should be formatted.
Odd side-note: I learned that the documentation for a fighter airplane of the time, if printed out, would weigh more than the aircraft and would fill a football-field sized collection of filing cabinets.
And as much as many today don't like XML, coming from the SGML world it is a boon.
> This a good article with some fascinating history.
Indeed. I don’t think it’s all correct, though. On Visicalc, it says “The grid cells couldn’t be styled with borders for presentation purposes, the values couldn’t be formatted, and the tables couldn’t even be printed”
I think I even the first version had (limited, of course) formatting support.
Something that always annoyed me about numeric data like dollar amounts in tables is that visually the comparison between quantities is logarithmic instead of linear.
E.g.:
Cost
$1500
$130
$110
$210
The text in the last three rows look 4/5ths the size of the text in the first row. However, even if summed, the last three costs add up to only 1/3rd of the top row! People visually see the number digits, which is roughly the same as Log 10.
I’ve so often had this issue that I started putting in-cell bar charts into every finance-related spreadsheet.
Otherwise meetings will get derailed debating the cost of something trivial that is totally irrelevant compared to the biggest absolute costs.
As a real example, I had many meetings spent debating a $15 monthly cost for server log collection in the cloud for a VM running a database engine that costs $15K monthly for the license alone.
Hey one of the co-maintainers of Great Tables, along with Rich Iannone, here!
I just wanted to say that Rich is the only software developer I know, who when asked to lay out the philosophy of his package, would give you 5,000 years of history on the display of tables. :)
It makes me wonder how we've gone this long with increasingly poor data table presentations (the mid-century modern tables are astutely pointed to as shining examples).
This makes me excited to get back into data analysis with python. Moreover, I see some possible API improvements and extensions I'd like to make.
I love this package and have been using it for a few years in R. It's great [for making] tables in html but the pdf and docx output is a little less polished. I do worry that the recent shift to bringing the python version up to speed with the R version has slowed down the R development. Though it's well worth checking out whatever your language.
Wonderful! In the 90s a colleague and I wrote a book (EBRI Datebook on Employee Benefits) which was mostly tables. In addition to SAS, our other primary tool was an ancient language called Table Producing Language ("TPL"). Despite dating back to the 1970s, TPL was incredibly flexible, expressive, and efficient - once you figured out the syntax.
The designers of Great Tables might want to check out TPL. It covers everything Great Tables aims to do, and I think may have a few more tricks up its sleeves:
This guy deserves some prize for 1) great work, 2) attention to detail, 3) in-depth research, 4) excellent presentation of his work (sparing the usual questions like "but what is it, really?", "how do I start?", "can you provide some examples?").
Interesting aside: AI models trained on spreadsheets need "good tables" such as column names, headers, etc. to understand context. Like Fortap: https://arxiv.org/abs/2109.07323
I'm interested in the midcentury modern ones because they have lots of vertical rules. I'm active on the subreddit for LaTeX and there is a religion common there that even one vertical rule is an unforgivable abomination.
I just made a table this morning for Calc II notes. The first column says something like $f'(x)$ in the first row and $f'(0)$ on the second. The table body lists values for different functions, one per column. I put in a column rule separator because the leftmost column seems separate from the others.
In any event, I'm suspicious of rules (pun noted).
Yes. They are sparklines. I actually asked Rich the author if they should just be called that in great_tables but he had some reasonable thoughts on why a distinct name made sense.
Tables are underutilized for how concise and descriptive they can be when making comparisons. It's a shame most text editors start with a blank table instead of inserting one pre-configured with some good design choices.
This looks great. I so wish that the HTML table element would get some progress - it’s so limited.
I don’t want to have to use some JS library component just to show tabular data especially given how badly they perform one big - but a server side rendered HTML table can be enormous and render fine. But again, so limited.
Past a certain table size, the JS libraries will use less memory. DOM elements take a lot of memory. Libraries like ag-grid only render a small portion of the total table at a time.
The next performance gain web tables comes from using a binary encoding instead of JSON, particularly arrow. Perspective uses arrow (in addition to rendering to canvas).
IME building buckaroo on top of ag-grid, I can render the table with up to about 300k elements very performantly with just JSON. Rendering speed is a non factor because only 50 rows are rendered at a time. Moving to arrow-js should be about 3 times faster for the entire system (python serialize, js deserialize, js render). Beyond 900k elements, you really want to lazily load from the server as the user scrolls. The memory usage for just the data in the browser tends to slow things down. (I am working on a library and benchmark for different serialization techniques).
Would be great if the example tables were great for such a post. Like their remote correspondent's table: they've actually made it partially worse in the "great tables" style:
- now the dates are not vertically aligned, and
- the weights have repetition of units (lbs) (although losing the decimal is an improvement)
- the name column is way too visually "heavy", that's a style you'd reserve for a header, a simple bold would suffice
They've also retained other issues from the original like city not being a city, but a combo of city and region, or similarly no separation of the first name of a person, which is especially important for a diverse group of people
Not mentioned yet are DocBook tables, of which there are several types. The kind we used starts here: https://tdg.docbook.org/tdg/5.1/cals.table. You have to drill down to get inside the tables. They have some--but I think not all-- the structure of GT.
There's also of course LaTeX (mentioned in a couple other comments here), which has "ordinary" tables and long tables (tables that span more than one page).
I use plotnine whenever I need to make (static) plots in Python. It's really quite well done, a close match to R's ggplot2, and more feature complete than any of the other Python grammar of graphics packages I've tried.
I really like great tables and its cousin in R, gt. You know they're taking a long view when there are photos of CLAY TABLETS and VISI-CALC in the article. Bravo!
Tables, when you really care, are so very difficult to get right. Sometimes you really want to densely compact the data to communicate all details to others who are already deeply invested in the dataset, sometimes you need to remove all but the most essential information to get across a single idea with clarity, and then there are a continuum of variations between those extremes. The problem expands into more dimensions when the media becomes a consideration-- you simply can't (and should not) use the same approach when dealing with pdf's vs html vs a slide-deck. On top of all that, you often have a personal style that you want to get across, or have a style that you need to comply with in some inflexible way.
I like how gt just NAMES the parts of table in their docs (and in the schematic in the article). This is a problem where agreeing on what things are called makes a ton of difference in usability.
Here is a very short guide on how to make nice tables (which, IMHO, look a lot better than the visually cluttered "Great Table" example from the article) for scientific papers:
I Use SAS reports in my job pretty heavily I'm keen to find alternatives - this looks pretty promising.
One thing in particular I'm interested in but could not see an example for is if this will let you insert "break lines" i.e. for displaying sub totals and similar.
For example based on the demo, which shows names and addresses from census data it might be nice to be able to break at each change in postcode and display some summary data like a count of people found at that postcode or an average age (based on the DOB) living at that postcode or similar.
Otherwise conditional formatting is another pain point either using rules i.e. if value in column B is greater than a specified threshold make the entire row bold. Or automatically creating a color gradient to highlight the cells ala Excel.
For bonus points management types like things like red and green traffic lights (or down/up arrows) you can display next to kpi data in a table It's a gimmick but wins you points.
I want to praise the Great Tables people for having created an excellent marketing piece. It is a lesson in how to engage people in a product announcement. I started reading the article and by the time they started explaining the product, I was sincerely interested in knowing about it.
While I'm waiting for the packages to download, can you explain how I get tabular output when I run a python application using your package at the command line? Does it produce HTML output? PDF? Your "getting started" docs doesn't explain.
You use it indeed to generate HTML output. For Python+Jupyter folk, that's most directly applicable to Jupyter Lab or Jupyter Notebook settings. You can use it with Jupyter book, nbconvert, or any other tools that convert .ipynb to HTML output...
(Disclosure: Quarto dev here) ..., like Quarto. You can use `great_tables` in code cells in Quarto to get great tables in your RevealJS presentation or website, https://quarto.org/docs/output-formats/html-code.html.
This article is mostly blither, whether or not it is AI generated. It is about a Python library for generating nicely formatted HTML tables, though they don't tell you much about it til the near the end. The library seems to use an OOP approach. An alternative approach might be more declarative. The product name "Great Tables" appears in boldface over and over (no idea if the font helps SEO) and the name itself is awfully pretentious imho. Overall, the library itself sounds ok,, but the blog post is the annoying market-speak that frequently makes me cringe here on HN.
It would be nice to add some interactivity features to the tables, like ActiveAdmin in Rails.
It's not AI generated. I have tracked the PR as they have worked on it. From what I have seen, the library has declarative elements that are similar to the grammar of graphics.
Actually, it's a near-perfect article from my point of view. I don't (regularly) work in tables, I'm not immersed in data visualization culture, and I wouldn't have thought twice about reading a simple "here is our product" page aimed only at people who already know what they're looking for.
However, this article gave me some really interesting and valuable background material and then concluded with solid, crystal clear (even to me, who's never written python) examples. I actually came away thinking, "well, this look fun - I should spin up some toy project to play around with this and learn how to use it."
I don't think I learned anything about tables from the pictures of rectangular grids found on cave walls. Most of us don't "work in tables", but we have all still seen some, made some, and know what they are.
There was some design blog post several years ago, maybe not even surfaced on HN:
Creating beautiful tables both UI and UX wise, with some features being e.g dropping separator between columns (row?), doing some visual accents, etc.
And yes, the most distinguished feature was that the tables weren't looking like your busy PowerPoint non-tech organisation stuff, they were very modern yet simple.
I don't remember the specifics, but I was really impressed and regret not bookmarking the article.
Does anyone know of the article in question, and maybe could share the link?
The best thing anyone can do with tables is make it so people use tables
Not "divs" - tables, can be parsed, and exported to other data formats easily, and are easy to recognize, without visualization.
Many websites build tables out of div's, and they may look like tables, but they are hard to manipulate/export data from.
If a table is on a website, I feel it should be easy to export it as csv, at the least, to "free the data" :D
How does this compare to https://github.com/jieter/django-tables2? That one makes it really easy to display database models as HTML tables with column sorting and pagination, and search/filtering can be added on top with django-filter.
If anyone is looking for a great open source table, I recommend slick grid: https://github.com/6pac/SlickGrid it has all the features shown in the article and has proven itself capable of everything I've needed for the last 2+ years.
The simple trick is sort the respective columns one after the other, starting with the least significant one. E.g. if you want to sort by columns A, B, and C, first sort by C, then sort by B, then sort by A. If the sort is stable (which it should be), this will result in the desired multi-column-sorted state.
Of course, this isn’t intuitive in the sense of being self-evident, but once you know this method, it basically works almost everywhere.
This article is about a Python library called “Great Tables” that is focused on the display of tables for publication and presentation (not for interactive browsing).
The article does not specify which output format it supports.
Also you get some bonus historical context on tables.
I'm one of the maintainers of Evidence (open source tool based on markdown + SQL) and working on a similar approach to creating presentation tables configurable in code.
I overall like the approach for complex scenarios but their example is not the best one. The original version is much more readable and their final version adds mostly noise.
Also, in documents all images and tables should have descriptive captions. So their header with title and subtitle would be redundant.
This is an excellent blog post - I'd never heard of Great Tables before, and I'm a newly minted fan!
> confronted with an all-too-familiar dilemma: copy your data into a tool like Excel to make the table, or, display an otherwise unpolished table.
One add-on (coming from the past 4 years of working on a tabular-data from Pythons startup [1]) is that users aren't just copying data into Excel because if it's good formatting capability: very often, there are organizational constraints that mean that Excel _needs_ to be where this data ends up.
The most common reasons I've seen for data ending up in Excel:
1. Other parts of the report rely on Excel features - you want to build pivot tables or graphs in Excel (often, these are much easier to build in Excel than in Python for anyone who isn't a real Pythonista)
2. The report you're sending out for display is _expected_ in an Excel format. The two main reasons for this are just organizational momentum, or that you want to let the receiver conduct additional ad-hoc analysis (Excel is best for this in almost every org).
The way we've sliced this problem space is by improving the interfaces that users can use to export formatting to Excel. You can see some of our (open-core) code here [2]. TL;DR: Mito gives you an interface in Jupyter that looks like a spreadsheet, where you can apply formatting like Excel (number formatting, conditional formatting, color formatting) - and then Mito automatically generates code that exports this formatting to an Excel. This is one of our more compelling enterprise features, for decision makers that work with non-expert Python programmers - getting formatting into Excel is a big hassle.
Of course, for folks who can ditch Excel entirely, this is entirely unnecessary. Great Tables seems excellent in this case (and anyone writing blog posts this good is probably writing good code too... :) )
Playing nice with Excel (and PowerPoint) is an underrated feature. The next step I see from business users is taking the formatted Excel table and pasting it into a PowerPoint slide. The hacker mindset often says the Microsoft Office suite is the wrong tool for the job, so we should use X tool and Y process instead. That may be true, but there's so much institutional inertia at established organizations that it's hard to completely abandon the Office suite. Anything that lets a technical user do something programmatically, but allows the output to be easily manipulated by a non-expert is invaluable.
I've had success generating svg visuals and placing them in slides, which PPT treats as a "shape" (the Graphics Format ribbon appears), and business users like that they can modify the shapes (for example, change the color). Great Tables supports pdf export, but not svg. I just tested a pdf vector in the current version of PPT, and while it maintains the vector, PPT won't let me convert it to a shape (only the Picture Format ribbon is available). Great Tables doesn't seem to support svg export directly, so there needs to be an additional pdf -> svg conversion.
The historical background about tabular displays of quantitative information is very interesting. I imagine it must have been fun think deeply about this problem.
Unfortunately, the API design in the example is just not very good:
(
GT(simple_table, rowname_col='Name')
.tab_header(title='Names, Addresses, and Characteristics of Remote Correspondents')
.tab_stubhead(label=md('*Name*'))
...
)
I'm uncertain if it's trying to mimic something in another language like R (or some grammar of graphics thing or D3.js.) Hopefully, it's not trying to mimic the look of long, chained `pandas.DataFrame` operations (because it misses the point of why those look the way it does.)
Of course, for ad hoc, in-a-notebook, cut-and-paste/written-from-scratch use, the API design doesn't really matter that match. Usually, users will readily memorise the required incantations then fiddle with the result until they get what they want or they give up.
It's probably the case that for most tools that produce visual outputs, a majority of users are creating things in this style. (There are, e.g., millions of casual Matplotlib users out there.) But programmatic use is not too far off. Tools that produce visual outputs (even those as formally rigidly at display tables,) are often subject to consistency requirements, which directly implies programmatic use.
So, when I discover that my colleagues and I have six tables across three notebooks that need a consistent look, and I decide to interact with this tool programmatically, am I expected to write…?
Imagine the web if every site was exclusively tabular. No UIs just a table of figures and a CRUD for modifying it. Something like hypercard meets excel
I'm less interested about web, but I think tables would work well for shell/terminal. If you squint, stream-of-objects and tables are sort of similar, so with some sort of PS/nushell style environment having a terminal with native table rendering ability could be great. I think lot of stuff we interact with in shell tends to be somewhat tabular in nature, but is now getting muddied by formatting it for dumb text terminals.
Just look at the first example on nushell frontpage: https://www.nushell.sh/ could that not look better with "Great Tables" or something similar?
“The democratization of computational tables arguably began with VisiCalc in 1979…
I mean, try it out and you’ll see that this is quite limited in more than a few ways.”
Them’s fightin’ words. IMHO VisiCalc’s ability to generate models quickly changed civilization. It freed people to try out ideas at no cost and to view or manipulate data in ways no one could hope to do before.
I have built a different table library for jupyter called buckaroo. My approach has been different. Buckaroo aims to allow you to interactively cycle through different formats and post-processing functions to quickly glean important insights from a table while working interactively. I took the view that I type the same commands over and over to perform rudimentary exploratory data analysis, those commands and insights should be built into a table.
Great tables seems built so that you can manually format a table for presentation.
https://github.com/paddymul/buckaroo
https://youtu.be/GPl6_9n31NE