Congrats on the release! I've been using Cursor but somewhat annoyed with the regular IDE affordances not working quite right (absence of pylance), and would love to go back to VSCode.
I'd love it if you lean into pooled model usage, rather than it being an addon. IMO it is the biggest win for Cursor usage - a reasonable num of LLM calls per month, so I never have to do token math or fiddle with api keys. Of course, it is available as a feature already (I'm gonna try Continue) but the difference in response time b/w Cursor and Github copilot (who don't seem to care) is drastic.
Our Models Add-On is intended to give the same flat monthly fee as you’re accustomed to with other products. What did you mean by leaning into pooled, just making it more front-and-center?
Yep, exactly that. IMO agent workflows, MCP and tool usage bits are all promising, but the more common usage of LLMs in coding is still chat. AI extensions in editors just make it simple to supply context, and apply diffs.
An addon makes it seem like an afterthought, which I'm certain you are not going for! But still making is as seamless as possible would be great. For ex, response time for Claude in Cursor is much better than even the Claude web app for me.
Quick question, what do you mean by new pylance in cursor?
It's basically VSCode - I just ported all extensions in Cursor settings, it installed existing VSCode extensions including Pylance and it just works...
I meant Pylance isn't legally available in Cursor (Vscode license restriction, which is justified). It broke very frequently, so I switched to Based pyright, which works but just not as well.
IMO a hindrance to this was lack of built-in fixed-size list array support in the Arrow format, until recently. Some implementations/clients supported it, while others didn't. Else, it could have been used as the default storage format for numpy arrays, torch tensors, too.
(You could always store arrays as variable length list arrays with fixed strides and handle the conversion).
I see it being a trade-off between how explicit the state persisted for a workflow execution is (rows in a database for Temporal and DBOS) vs how natural it is to write such a workflow (like in your PL/compiler). Given workflows are primarily used for business use-cases, with a lot of non-determinacy coming from interaction with third-party services or other deployments, the library implementation feels more appropriate.
Though I am assuming building durability at a language-level means the whole program state must be serializable, which sounds tricky. Curious if you could share more?
There's certainly a tradeoff between the two approaches; a simpler representation (list of tasks or DAG) is easier to query and manipulate, at the cost of being less expressive, lacking features like loops, conditionals, etc.
In the workflow engine I described, state is represented as a graph of objects in memory; this includes values like integers/strings and data structures like dictionaries/lists, as well as closures, environments, and the execution stack. This graph is serialised as JSON and stored in a postgres table. A more compact binary representation could be added in the future if performance requirements demand it, but JSON has been sufficient for our needs so far. A delta between each snapshot is also stored in an execution log, so that the complete execution history is stored for auditing purposes.
The interpreter is written in such a way that all object allocation, object manipulation, and garbage collection is under its control, and all the data needed to represent execution state is stored in a manner that can be easily serialised. In particular, we avoid the use of pointers to memory locations, instead using object ids for all references. So the persistent state, when loaded, can be accessed directly, since any time a reference from one object to another needs to be followed, the interpreter does so by looking up the object in the heap based on its id.
Non-deterministic and blocking operations (including IPC receives) are handled outside of the evaluation cycle. This enables their results to be explicitly captured in the execution log, and allows for retries to be handled by an external mechanism under control of the user (since retrying can be unsafe if the operation is not idempotent).
The biggest win of using a proper language for expressing the workflow is the ability to add arbitrary logic between blocking operations, such as conditional tests or data structure manipulation. Any kind of logic you might want to do can be expressed due to the fact the workflow language is Turing-complete.
That's really interesting! It does seem that this is identically semantically to the library approach (as the logic your interpreter adds around steps could also be added by decorators) but is completely automatic. Which is great if the interpreter always does the right thing, but problematic/overly magical if the interpreter doesn't. For example, if your problem domain has two blocking operations that really form one single step and should be retried together, a library approach lets you express that but an interpreted approach might get it wrong.
I kinda agree with you but I can also see why it isn't that far from "reasoning" in the sense humans do it.
To wit, if I am doing a high school geometry proof, I come up with a sequence of steps. If the proof is correct, each step follows logically from the one before it.
However, when I go from step 2 to step 3, there are multiple options for step-3 I could have chose. Is it so different from a "most-likely-prediction" an LLM makes? I suppose the difference is humans can filter out logically-incorrect steps, or prune chains-of-steps that won't lead to the actual theorem quicker. But an LLM predictor coupled with a verifier doesn't feel that different from it.
> The succeeded operations would skip because workflow has run completion record for same idempotency key. Is that correct?
This sounds about right. But you need to make sure the service being called in that step is indeed idempotent, and will return the same response which it earlier couldn't in time.
PyArrow string arrays store the entries (string values) contiguously in memory, so access is quicker, while object arrays have pointers to scattered memory locations in the heap.
I agree, I couldn't really figure how the new numpy string data type makes it work though.
I think that's a little harsh. Imo the reason for difference in popularity/github-stars is just different user bases- an order of magnitude more people use LLM APIs (and can leverage Dspy) vs those who finetune an LLM.
Agree about the abstractions btw. I found Dspy very convoluted for what it does, couldn't make sense of Textgrad at all.
I think this is a fine opinion, we like tools that do exactly how much we want them to. But I'd suggest setting up python (and virtual envs) was actually a big headache for a lot of newer users, and some of the old ones (me that is).
I also don't see why leaning into python being a wrapper around rust/cpp/c is a bad thing. Each language has its own niche and packaging/bootstrapping is more of a systems level language problem.
I'd love it if you lean into pooled model usage, rather than it being an addon. IMO it is the biggest win for Cursor usage - a reasonable num of LLM calls per month, so I never have to do token math or fiddle with api keys. Of course, it is available as a feature already (I'm gonna try Continue) but the difference in response time b/w Cursor and Github copilot (who don't seem to care) is drastic.