Hi HN, I developed Starboard over the past months.
Cell-by-cell notebooks like Jupyter are great for prototyping, explaining and exploration, but their dependence on a Python server (with often undocumented dependencies) limits their ability to be shared and remixed. Now that browsers support dynamic imports, it has become possible to create a similar workflow entirely in the browser.
That motivated me to build Starboard Notebook, a tool I wished existed. It's:
* Run entirely in the browser, there is no server or setup, it's all static files.
* Web-native, so no widget system is necessary. There is nearly no magic, it's all web tech (HTML, CSS, JS).
* Stores as a plaintext file, it will play nicely with version control systems.
* Hackable: the sandbox that your code gets run in contains the editor itself, so you can metaprogram the editor itself (e.g. adding support for other languages such as Python through WASM).
I see you're using lit-html, LitElement/web components, and you've exposed lit-html to users. That's awesome! I'm one of the authors.
Observable's html templates were inspired by lit-html, but they ended up implementing their own library. I'm glad the lit-* stuff is useful to you, and really excited to see it available in cells. I'll have to try this out!
This is the first I'm hearing of lit-html, but it looks great! Is it still an active project? I'd be curious to hear your thoughts about lit-html in comparison to newer tools like svelte, solidjs, etc., which, like lit-html, abandon VDOM in favor of "surgical updates" to the DOM.
It's very, very much active. It's used heavily at Google in apps, Chrome, YouTube, Waymo, etc., and by companies like Microsoft, GitHub, Netflix, IBM, Adobe, Maersk, SAP, Apple...
One big difference to the projects you listed is that lit-html doesn't require a compiler. It's plain JS, and is fast because it uses template literals and built-in HTML parsing/cloning to associate expressions with DOM.
A late response, but I can add a bit more motivation.
Without lit-html Starboard would be nowhere near as flexible or elegant. Starboard has to be completely customizable at runtime, that would probably be difficult to achieve with anything that has a build step, with lit-html it's "just Javascript" all the way down.
If you usually use a framework like React, Vue or Svelte it is still worth trying out lit-html, they are not mutually exclusive, it's simple to learn and deceptively powerful. It's great to have it on your toolbelt.
It's half abandoned now. one issue of js based notebook is the missing of a file system. data scientists load huge data into notebooks, this is hard to do on the web. There is the new web filesystem api, but that only helps load local files.
My vision is for this to be handled by things like remoteStorage, Solid, or (if you must) Google Drive. I find having control over your data and delegating access to various apps to be a nice flow when implemented well (see for example https://app.diagrams.net/).
It's not a silver bullet. For larger data you can't afford to download it every time. That tends to balance itself out though, because you don't really want to store huge data on your laptop anyway, so you're going to have to find a different solution for that anyway (such as using Jupyter). You can accomplish quite a bit with streaming + sampling, which is the crux of how all our apps work in iobio.
I guess the web first way of dealing with this would be to have the data online also, maybe even on the same platform. Then you could have things like: Look at visualizations of this dataset, at what people have done with this dataset etc.. And you could immidiately verify the results, assuming you trust the data.
- did I understand correctly that notebooks would be hosted (at least stored and served) on the Starboard website? (you mention it's kind of a github, in the About section) What does this imply for private notebooks? Are they possible?
- any example on how to run python there with pyodide? I didn't understand by quickly looking if that integration is part of future plans or it's already a possibility.
- I read that not all features are ready at the moment, for example exporting to a proper website. Any forecast on when will features like this one ready?
- If you log in and create a public notebook, it's stored on Starboard's server, then you can just share it by link. If you create an offline notebook it's stored in your browser's LocalStorage. Right now you can't create a notebook yet that only you can see or can be shared with a private link, to do!
But of course you don't have to use the Starboard website, you can just save it as a text file on your disk. In the future I want there to be a CLI tool that serves that notebook for you on localhost.
- No examples yet, it's future plans for now (but I hope to land that in a week or two!)
- No forecast.. It's just myself working on it, so you'll have to bear with me. I am working full-time on this.
> Cell-by-cell notebooks like Jupyter are great for prototyping, explaining and exploration, but their dependence on a Python server (with often undocumented dependencies) limits their ability to be shared and remixed. Now that browsers support dynamic imports, it has become possible to create a similar workflow entirely in the browser.
Pretty cool. As someone involved in building a machine learning platform which also allows users to leverage Jupyter notebooks, I feel your pain. We had to add many features to make working on non trivial projects more feasible. For example near-real-time collaboration so people could pair-program on one notebook.
One other problem you mention is the kernel in the backend. The problem with that is often found in issues where a user launches a long-running notebook, closes their tab/browser. The computation happens but the results don't find their way, which means users must have code for side effects and artifacts in the notebook.
We've added a functionality to run notebooks asynchronously to address that problem, which also addresses the problem of multiple users competing for limited resources (tragedy of the commons) and kernels dying. Now users can run their notebooks asynchronously: they're scheduled, run, and saved seamlessly.
Jupyter is a useful tool which has these problems that aren't trivial to solve. For example, for the near-real-time collaboration, the Jupyter team is revamping their API with Lumino (https://github.com/jupyterlab/lumino), which was PhosphorJS but the development stopped when the main contributor had retired.
Hey! I'm using jupyter notebooks on a daily basis, and I would be really interested in the 2 features that you mentioned: enabling real time pair programming (a la google doc) and resuming running notebooks whose frontend was closed.
Are you building a product that you plan to release? Or are these contributions that are available as open source components?
In any case, I'd like to know more, or to have some information on relevant topics that might be useful to understand and find workarounds for these issues!
>and resuming running notebooks whose frontend was closed.
The platform schedules the notebooks, runs them, and saves the results with the output[0]. You can still check the progress, status, and position in the queue of the notebook if you wish. You still get the results after it's done even if you had closed the browser.
>Are you building a product that you plan to release? Or are these contributions that are available as open source components?
Well... We're a machine learning shop, present in Algiers, Algeria and Paris, France, that's been building custom turn-key products for large organizations for quite some time. As you know, non-trivial projects with real stakes are a bit more "challenging" than portfolio projects. Tracking rationale of projects, tracking metrics/params/models, shared access to data, deploying and managing models, challenges in collaboration, setting an environment and keeping things from breaking, or needing a beefy machine.
We started building the platform to allow our teammates to remotely work on projects involving machine learning, as commuting is hard and having contributors in different countries can be challenging. We also wanted to simplify access to a larger talent pool from other countries, which means we had to have a way to work together. We also wanted self-service: our data scientists had to sollicit the help of other engineers to either use some language specific feature, or to deploy their notebooks. We wanted to allow them to do so themselves in one click so they don't wait, and other engineers can work on other things. We wanted the results to get as soon as possible to the domain expert, and allo that domain expert to be able to train models themselves with no code experience, by exposing parameters that are relevant to some sector, and allow them too to deploy a model. In many cases, a domain expert would chime in to tell us that some variable that was dismissed was actually critical at some point in a decision-making process. So we wanted that, too.
We also wanted developers to interact with the models with HTTP requests period. We kind of suffered from having too many simultaneous projects with different stacks.
We needed a way to execute projects efficiently and effectively, but we didn't like other products. We also wanted extensibility. As I said, we built whole products, not just the models, so we needed a way to plug functionality in.
We're actively developing this.
>In any case, I'd like to know more, or to have some information on relevant topics that might be useful to understand and find workarounds for these issues!
Sure, shoot us an email at the site below or using the info in my bio.
Neat! There's a project called Jyve that compiles Jupyter Lab to WASM (using iodide). https://github.com/deathbeds/jyve There are kernels for JS, CoffeeScript, Brython, TypeScript, and P5. FWIU, the kernels are marked as unsafe because, unfortunately, there seems to be no good way to sandbox user-supplied notebook code from the application instance. The README describes some of the vulnerabilities that this entails.
The jyve project issues discuss various ideas for repacking Python packages beyond the set already included with Pyodide and supporting loading modules from remote sources.
https://developer.mozilla.org/en-US/docs/Web/Security/Subres... : "Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match."
There's a new Native Filesystem API: "The new Native File System API allows web apps to read or save changes directly to files and folders on the user's device." https://web.dev/native-file-system/
We'll need a way to grant specific URLs specific, limited amounts of storage.
> The Python scientific stack, compiled to WebAssembly
> [...] Pyodide brings the Python 3.8 runtime to the browser via WebAssembly, along with the Python scientific stack including NumPy, Pandas, Matplotlib, parts of SciPy, and NetworkX. The packages directory lists over 35 packages which are currently available.
> Pyodide provides transparent conversion of objects between Javascript and Python. When used inside a browser, Python has full access to the Web APIs.
I've had a look and wasn't able to get it to work yet, imports can be bit iffy: usually anything that exposes ES modules or puts variables on the window object work fine.
It looks like the reason it doesn't work here is because it relies on "this" being the window object. To make top-level await possible your code gets wrapped into a function, and in there "this" no longer points to the window. I am sure a workaround is possible, I'll make a ticket on github.
This is really cool, and I've wanted something like this to hack on for ages. I'm a big fan of being able to present concepts along with code that you can run, and I'm also a huge fan of not having to rely on a server backend. I have a static blog where I try to talk about various math concepts which I illustrate with code. I hacked together a pseudo-literate html sort of presentation, and it works just fine, but it's inflexible.
It is indeed a good work. The barebone demo is an empty notebook. How could I loaded a locally saved notebook ? Should I loaded it through javascript API or embed it in the index.html ? Thanks.
I've been working on something similar on and off for a while, and I've found the space to be fascinating and somewhat puzzling. Most of the first questions I try to ask about these in browser notebooks don't have clean answers. In no particular order:
- Do you have any ideas about solving logic errors? As it stands right now a while(true) loop crashes the page and consecutive re-openings of the page for a short while (in chrome). What about things like document.body.innerHTML = "" (you actually seem to handle this pretty gracefully, I'll see if I can poke more holes in this another time). For what it's worth none of the online notebooks I've seen have a satisfactory solution to infinite loops, loop timeouts are too blunt an instrument and crashing the tab can take the browser minutes to recover.
- Is there a particular reason you use eval rather than the new Function constructor? From what I've read using new Function is much more performant, and other than different scoping a better choice than eval. Can't find the link at the moment, but it was wrapped up in the mdn[1] design docs for their codebox examples.
- I see you're using the first codemirror 6[2] beta release, how are you liking it? I really enjoy the interface so far.
- Do you have any favorite resources or inspirations about why you went about building a hackable offline local first notebook environment? I particularly like the experiments at Ink and Switch[3]. (many related hn submissions). As well as webstrates[4].
My attempt at this game is to break out of the notebook style single column layout and embrace an art board style canvas, which is a rather radical idea in that it is not obvious what that should look like or how basic interactions like hierarchies or execution order should look, but fun to explore. I also desperately want to prevent crashing as a result of logic errors and workflow footguns(like deleting DOM elements or overwriting storage), to that end I have a separate storage of scripts to rebuild a "safe boot" interface, but there is more thinking to be done here.
I look forward to poking around your code some more soon. Thanks for posting.
Edited out a question about your motivations. I reread your comment and the about page and I realized my motivations are similar to yours. While observable doesn't have a few of your key points, it is a fantastic product. The reason I don't settle on it is that I'm interested in experimenting with the environment outside of a notebook-with-cells interface.
> As it stands right now a while(true) loop crashes the page
> [...] I also desperately want to prevent crashing as a result of logic errors
I could be wrong here, but I think the new Site Isolation stuff in Chromium[0] may mean that you can sandbox an iframe to a different origin, so that the main threads of the top frame and the iframe aren't synced. So the iframe where the code is running would freeze, but the main frame would still be responsive. Again, I may have misinterpreted this because I only read about it in passing, and it was a while ago.
A brief skim through that doc looks like it is about a kind of security that's not relevant to my project. I'm trying to protect the user from themselves while simultaneously giving the user full power over the document.
Web workers are interesting and relevant to this use case, however the almost total isolation doesn't lend itself to manipulating the DOM without the user explicitly writing a custom API to interpret the worker's messages. So, back to the same problem.
I’ve been thinking a little of doing something in this space. But doing so as part of a larger project involving a new programming language so perhaps not applicable to you situation.
I also want to leave the notebook style, focusing more on graph approach. Think higher order spread sheet without the grid.
One, somewhat tangential, thought I’ve had to handle the infinite loop issue though. Is to fragment the language somewhat like they suggest in the out the tarpit paper. Trying to avoid infinite loops by construction, only used at the highest orchestration layer to schedule execution blocks, where the runtime can keep track of things going out of control.
Not sure what it would take to extract a “total” subset of javascript though. But one idea would be to insert a trampoline breaking out of infinite loops often enough for error handling.
To answer some of your questions (let me know if I didn't answer all of them!)
- I think logic errors can't be solved or prevented when exposing vanilla HTML, CSS and JS. You can completely break the notebook if you want to. Like the sibling comment to this stated: the notebook is run in an iframe of a different origin so you (should!) not be able to break out of that iframe. But of course if you introduce some endless loop that crashes the browser (or maybe just the iframe depending on the browser), I can't stop that.
- No particular reason, but it was always a bit iffy getting the return value correctly. To support top level await your code actually gets wrapped into an async function, I don't know how nicely that plays with new Function.
In the future I want to support ES modules with import instead. Perhaps no magic will be required at all anymore and performance will probably be even better (https://2ality.com/2019/10/eval-via-import.html)
- Codemirror is very good and doesn't have a huge bundle size. I decided I wanted to support touchscreens. The monaco editor completely fails on my phone (if I type space it usually copies the word in front of it), whereas codemirror just works. The autocomplete and language support is much worse though, on a desktop I think I will always choose the Monaco editor.
- Probably not the most exiting answer, but for me it was just wanting to use Jupyter for things it was really bad at. There is Project Iodide which is very similar, but they made some different design decisions that move away from the code-output-code-output structure.
By putting the editor inside the sandbox things become much more straightforward and less "special" or "magic", that's another big difference from Project Iodide.
I think more and more fully client-side editors are becoming possible now due to dynamic import and webassembly, do share your project with me if you get the chance to work on it!
Ah, okay this is interesting. The entire notebook is one iframe, which lets you do more of the metaprogramming and sharing variables. Both printing (via console) and non-printing infinite loops crash the iframe until you close the page. And the reason my document.innerHTML = "" test was recoverable is that your notebook-top-button-container is not in the iframe (and it has that refresh button!).
I'll have to look more into async stuff. Great eval article there. I've found these four lines to work for importing content[1]. I dove into realms[2] and the realms shim [3] a while ago after reading about Figma's engineering behind plugins [4] but I never figured out how I might use that for my system.
You might take a look at how the codemirror author built his book [5] and editor [6] in this repo [7]. He implements the loop detection I derided in my first comment. I shouldn't be so harsh, it's a good and simple idea. It's just not an answer, more of a hack. My interests align pretty well with the p5js editor audience[8], so some of the early issues there convey why the simple loop detection isn't ideal.
The only viable way to avoid an endless while loop or overly heavy operations is to have a worker do the eval(), though it would defeat the meta-programming aspect and isolate execution making it pretty much useless. It's still possible to proxy local objects via comlink (or implement your own worker proxy) but it would limit communication to POJO's so no dynamic module imports and access to non-serializable objects.
Right, where you could send the worker a terminate signal if some conditions were met like time or no response or too many responses...
And that drawback obsoletes the strategy it seems, unfortunately.
It's an interesting problem, how to provide full access and control as well as non-corruptible recovery systems.
One of the biggest problems I consistently run into is that printing to the screen is a slow operation, every learns this day one of any graphics work, but things like console.log or Node.appendChild are the primary way to destroy someone's computer for a short period of time because orders of magnitude more operations are queued up before they can complete. I wonder if painting operations could be limited to some max number per second to prevent that orders of magnitude queue problem. And, more importantly, how do you code that limit?
> One of the biggest problems I consistently run into is that printing to the screen is a slow operation, every learns this day one of any graphics work, but things like console.log or Node.appendChild are the primary way to destroy someone's computer for a short period of time because orders of magnitude more operations are queued up before they can complete. I wonder if painting operations could be limited to some max number per second to prevent that orders of magnitude queue problem. And, more importantly, how do you code that limit?
For node.appendChild you have the whole virtual DOM thing leveraged by modern JS frameworks. If you hit issues try using any framework that boasts having a virtual DOM.
The same strategy could be done for the console to some extent.
The built-in console object is over-writable so you can add a layer that would batch console calls (if you do many in a short time). Note that this will break the call address you have on the right-hand side. Also IE11 does strange things with the console object depending if devtools are open or not.
The solution above is doable but it's a hack. If you really have to call console a lot of times in a short time adding a custom log method would be better.
globalThis.__debugLog = [];
globalThis.__printLog = () => globalThis.__debugLog.forEach(i => console.log(i));
const log = (...args) => globalThis.__debugLog.push(args);
// then in code:
log(whatever);
// and once you want to check it, call printLog in the console
__printLog();
Again with this solution you don't know where the call happens but it's a starting point. I recently started using the pattern above to debug performance and memoization issues I had in heavily used code. The console wasn't readable anymore without it. Console.table is helpful for printing out bigger sets of debug data btw.
I meant to dig more into requestAnimationFrame, thanks for the prompt. I should clarify, the problem is that you can crash a web page if you do any animation inside of a runaway loop but especially painting with console. I suppose this indirection might help you realize that before it happens though, and that's probably worth doing. Thanks for the code.
You're welcome. I avoid runaway loops like fire because the crashes tend to be problematic to debug. Consider the timeout thing from SO. Generally working with events in JS is easier than central run loops.
Note that it does not mention promises and async/await but it does include it. Promises are essentially syntactic sugar for a setTimeout calling success and failure methods. This also means that promise code will be executed in parallel but on one thread and one heap, so each promise will fight with all other for priority. This is generally fine for async stuff like fetching resources but it would completely destroy performance if it would do anything heavy, like, say unzipping zips with jszip.
Historically the event-driven model of JS enabled Node.js's API to be callback-driven. This lead to what is called "callback hell" and eventually to a proper introduction of promises and async/await to the language.
A good read, mdn is far and away the best resource I have found for web technologies. I'm excitedly looking into some p2p tech and the webrtc articles are useful.
From the article:
> "Run to completion"...
> A downside of this model is that if a message takes too long to complete, the web application is unable to process user interactions like click or scroll. The browser mitigates this with the "a script is taking too long to run" dialog. A good practice to follow is to make message processing short and if possible cut down one message into several messages.
Good advice. And this is what happens when you `while(1) {console.log("loop")}`. Though I've never see a "taking too long to run" dialog, I'm just eventually able to kill the process myself and sometimes it takes a few minutes while my fans spin.
IMHO As an indie founder you never really have to worry about your brand being the same as another until something is internet owned. Which means that it is undeniably #1 on all search engines for that word. For example you couldn’t really start an indie project called “gap” because it’s just so dominated and there’s no room for you to ever overtake it.
But “wide”, for example, is not internet owned so you could start something scrappy called “wide” with the hope that one day you can internet own it.
Starboard is definitely not internet owned so if you do decide to drop it at least know that there is no reason to do so because of external optics.
Also note, there is Probably no other brand that you could choose that didn’t already have a lot of folks using that brand already.
In the end the market will choose the winner!
All that said, this version of starboard here does look pretty cool. I don’t mean to upset anyone. But I did want to explain this point of view on brands for indie founders.
Elevator pitch: a Pinboard-like bookmarking website with an emphasis on good UX (while preserving the information density and without a sluggish JS SPA).
Cell-by-cell notebooks like Jupyter are great for prototyping, explaining and exploration, but their dependence on a Python server (with often undocumented dependencies) limits their ability to be shared and remixed. Now that browsers support dynamic imports, it has become possible to create a similar workflow entirely in the browser.
That motivated me to build Starboard Notebook, a tool I wished existed. It's:
* Run entirely in the browser, there is no server or setup, it's all static files.
* Web-native, so no widget system is necessary. There is nearly no magic, it's all web tech (HTML, CSS, JS).
* Stores as a plaintext file, it will play nicely with version control systems.
* Hackable: the sandbox that your code gets run in contains the editor itself, so you can metaprogram the editor itself (e.g. adding support for other languages such as Python through WASM).
* Open source (https://github.com/gzuidhof/starboard-notebook).
You can import any code that targets the browser directly (e.g. puts stuff on the window object), or that has exports in ES module format.
I'm happy to answer any questions!