Making software durable and resilient to failures can be a big architectural investment. But what if it wasn't? Would we make everything durable by default?
Xorq is a Python lib https://github.com/xorq-labs/xorq that provides a declarative syntax for defining portable, composite compute stacks for different AI/ML use cases.
In this example, Xorq is used to compose an open source FeatureHouse that runs on DuckLake and interfaces via Apache Arrow Flight.
The post explains how:
- The FeatureHouse is composed with Xorq
- Feature leakage is avoided
- The FeatureHouse can be ported to any underlying query engine (e.g., Iceberg)
- Observability and lineage are handled
- Feast can be integrated with it
Composite data engines such as the one in this Trino-DuckDB example can be created using the xorq framework to simplify multi-engine data pipelines. Useful when a dataset's native query engine does not support a required operation.
(from DBOS) Great question. For better or worse, it seems like discussions about workflows and durable execution often intertwine. Usually ending up in what types of jobs or workflows require durable exec.
But really, any system that runs the risk of failing or committing an error should have something in place to observe it, undo it, resume it. Your point about "big enough scale" is true - you can write your own code to handle that, and manually troubleshoot and repair corrupted data up to a certain point. But that takes time.
By making durable execution more lightweight/seamless (a la DBOS or Restate), the use of durable execution libs become just good programming practice for any application where cost of failure is a concern.
No, they don't. They work just like Temporal and the others, which send the durable state to a separate store. I totally understand that this is good for the tool business model, since many users will end up paying for the separate store, instead of keeping it in the same DB where the application already is, without needing to pay extra for it. After all, the amount of data for keeping that state should be relativly small, so no one would provision a separate DB for it if the SDK didn't force them to.
My bad, I had misunderstood that. You're right, and thank you for sharing it.
That is a great differentiator. I see you have Python and JS/TS libraries, but my team is working with Go right now, so I'll have to pass. However, I'll keep it on my radar. Being able to handle wrap DB commands in the same transactions as the workflow store is awesome! [2]
It depends what part of DBOS you're looking at.
DBOS Transact is the framework (TypeScript) used to develop apps/workflows such as those in the benchamrk.
DBOS Cloud hosts and executes DBOS Transact apps/workflows a la (AWS Lambda+Step Functions). So it is apples:apples. Functionally, DBOS Cloud is like Lambda and Step Functions in one.
I've hired, worked with, and managed Dev Advocates at different open source software companies (working with Apache CouchDB, Apache Flink, and projects). They are really valuable and are an important part of awareness raising, user adoption, user retention, and technical recruiting.
A few thoughts:
1 - I think it's good to focus some of them on developer (user) acquisition and some on developer retention. The former are more evangelical - social, talks, demos, content that attracts new users. The latter focus on the user experience (UX feedback, docs, issue resolution, community support, how-to vids, etc.) to make sure new users become and remain active with the software. Both are critical to adoption and growing word of mouth.
2 - Dev Advocates can own the open source contributor experience! Recruiting, welcoming, recognizing new contributors is a great thing to see happen. It's an easy thing to overlook with companies placing so much emphasis on user acquisition. It's important for community goodwill building and expanding the network of external advocates.
3 - Make sure the executive team understands the breadth of Dev Advocate responsibilities. It can get tense if they only expect talks and blog posts.
OtterTune removed the AWS availability zone limitation in the free tier today. You can use the free version to tune RDS PostgreSQL or MySQL in any availability zone now.
reply