> Writing Polynote’s code interpretation from scratch allowed us to do away with this global, mutable state. By keeping track of the variables defined in each cell, Polynote constructs the input state for a given cell based on the cells that have run above it. Making the position of a cell important in its execution semantics enforces the principle of least surprise, allowing users to read the notebook from top to bottom. It ensures reproducibility by making it far more likely that running the notebook sequentially will work.
a = [1,2,3]
b = (a,True)
b = 5
# Cell 2'
b = 4
Currently it takes a shallow copy of the state output by each cell, meaning every value is going to be a primitive value or a reference. If it's a reference to mutable state, you're kind of on your own with respect to keeping reproducibility. I felt like this was a good compromise between strictly enforced reproducibility and practicality; if it turns out to be confusing we could consider deep copying the state, or having an option to do that (I could imagine it being pretty bad for efficiency in a lot of ML use cases, though).
Personally, I'm looking forward to trying out the SQL support. I haven't seen an elegant solution for SQL notebooks in Jupyter, it was always second-class via Python or some such. Or have I missed something?
Interesting. Judging by that it seems to be implemented with a JVM language and a screenshot shows "Scala" as a supported language, I'm guessing at least all the JVM languages are supported (personally hope for Clojure) but can't seem to find a list of supported languages anywhere in the post or on the website.
What languages are supported by Polynote?
Closest thing that comes to mind is something like Apache Drill, which coincidentally also uses Apache Calcite as the SQL interpreter.
Also wondering why I would use this over Zeppelin which can support other interpreters like Flink?
As ever, the best answer is the Notebooks Are Bad, Actually
We've had some better options contributed in the past couple of weeks, but as long as we're going to change it I didn't want to rush that. So we stuck with my questionable typographic treatment for the blog post.
(Edit: autocorrect typo)
(this looks fantastic btw, I'm definitely going to explore it more in depth. thank you a TON for releasing it!)
Also not a physicist FWIW <3
That project experimented with a lot of interesting themes I see echoed here.
It's nice know that your report actually worked, and is not showing something odd because of a hidden state, but sometimes you just want to print the darn thing now!
The server use case is real, though. Users typically like to run it on a beefy cloud machine with access to a Spark cluster.
For you they are not meant that way. But other people started using them that way. Netflix is a well known place for this workflow.
I mean something like these:
My vision is separating financial analysis from formatting, or bringing CSS / markup languages into the world of Excel and sidestepping PowerPoint entirely if possible
PowerPoint is a slide presentation tool that is sadly used for authoring so-called "books" with lots of formatting in Investment Banking, Management Consulting and Corporate America at large... and its shortcomings become immediately obvious
As we speak (!!!), I'm having to reformat a book from our standard template into the client's template by manually resizing charts, recoloring series, etc... literally wasting hours of my time because we use Excel + PPT
And then the task would just be mapping the data generated to the templates, mad lib style?
A lot of people just don't have an intuitive sense for design or plain don't care, so you're left with a high variance in the quality of the work that is produced. Speaking from my banking experience, the result is senior bankers have to spend time marking up documents on things like formatting, and junior bankers feel frustrated because they waste their productive hours on non-productive work. And it's generally a frustrating experience indeed as the tools we use aren't built for this purpose.
But formatting is just part of it. "Code" reusability (more like financial model reusability) is basically zero, auditing is a pain and people use many different approaches to do the same things, but again with varying levels of efficiency and accuracy. I can tell you 10 ways to calculate Total Shareholder Returns for a public company using the FactSet add-in in Excel. Also several ways to pull Revenue and EBITDA figures, and these aren't even the more esoteric metrics like Free Cash Flow or Funds from Operations.
These decks that we prepare should work like recipes. Input the ingredients and out comes the prepared dish, but everything is so god damn manual
Not to mention issues with Excel itself, like how Excel's (non-cascading) styles and defined names (similar to variables) propagate to other workbooks if you copy-paste content from one to another, which in the long-term creates files full of garbage that crash and corrupt frequently
The whole paradigm is a shit show. One day it will be different, I am certain. But nobody has taken a holistic, multi-disciplinary view at the problem because bankers / consultants don't know what is possible with today's technology and the ones trying to address these usability issues are only looking at a couple pieces of the puzzle at once
As I understand it, financial companies often want to gather data from multiple places but consolidated in a digestible form to make financial decisions. (what kind of financial decisions, I'm unclear on, since most trading is done by computers now, but maybe this is for ETFs or OTC trades) And they're use to looking at it on PDFs and Powerpoint, because that's what people email around, and no one trusts having financial slides on the web. (why, btw? In case they leak?) And because you have many clients that want the same or similar sort of analysis on publicly traded companies, you'd ideally be able to change the analysis once, and then generate all the PDF and PPT reports they want to see.
It does seem like a giant waste of time to cut and paste data from excel into powerpoint by hand. However, you should be able to export Excel data to Power Point via the Visual Basic Editor. (https://www.wallstreetmojo.com/vba-powerpoint/) Do people not use VB to prepare this?
I don't get the impression that analysts in finanace would be willing to move from Excel to a notebook. Do you get a different sense? What would a notebook have to offer to get them to switch? Analysts generally seem to love Excel, with the exception of the slow first load and crash.
Yes, this is spot on.
> (what kind of financial decisions, I'm unclear on, since most trading is done by computers now, but maybe this is for ETFs or OTC trades)
Sales & Trading is but one part of banking. Yes, a lot of S&T is automated, but long-term strategy isn't defined by computers and neither is pitching to win new businesses. Besides S&T, there's also Restructuring (advisory and financing) and Mergers & Acquisitions. My opinion is written from the perspective of an M&A banker. I probably make ~3-4 PPT books every week on average.
> And they're use to looking at it on PDFs and Powerpoint, because that's what people email around, and no one trusts having financial slides on the web. (why, btw? In case they leak?)
Concerns over leaks is certainly a driver, but traceability and auditing also play a role. 3 years from now, I can certainly retrieve info that was attached to an e-mail but a link may have long expired. Also most recipients are over the age of 40 so might not like using links in general
> It does seem like a giant waste of time to cut and paste data from excel into powerpoint by hand. However, you should be able to export Excel data to Power Point via the Visual Basic Editor. (https://www.wallstreetmojo.com/vba-powerpoint/) Do people not use VB to prepare this?
Certain tools like the add-ins provided by FactSet, S&P's Capital IQ and, less commonly, Bloomberg, export data into PPT with some metadata attached to it that allows you to refresh content quickly (in theory). It's all built as plugins on top of MS Office apps so your experience is not always smooth. Plus they don't solve the bigger issue of reusability
> I don't get the impression that analysts in finanace would be willing to move from Excel to a notebook. Do you get a different sense? What would a notebook have to offer to get them to switch? Analysts generally seem to love Excel, with the exception of the slow first load and crash.
I think Analysts like spreadsheets with very responsive UIs and countless hotkeys. Muscle memory is a major thing in this business.
My vision of a solution would be one that still implements spreadsheet-like functionality but does so in pieces that connect together all the way through publishing via an integrated paradigm
Alternatively build something more tightly integrated to MS Office and sell it to Microsoft