SQL, JS, Excel are really hard to substitute because of how widely used they are by people. Even if something new comes up that it's objectively better, so far has always failed gaining traction because of this reality.
I wonder though, is such a dialect better for agents? Have you tried to measure if an agent performs better expressing queries in such a language instead of SQL?
Claude had no problem translating SQL into Prela, and because you have fine grained control over the query plan (a Prela query is a plan), it was able to optimize queries to be very fast
I'm more curious about going from text to Prela instead of going from text to SQL and measuring any difference in the performance there. On one hand models have been trained on a lot of SQL on the other hand they are really good in mathematical reasoning too so thinking in Perla might be a natural fit for them.
Yes, maybe not the language itself, but the ideas behind it. Tarski's Algebra of Relations is actually a better model for modern columns stores than the standard relational algebra, because a column is a binary relation from the primary key into its value.
It would be pretty easy to put a DuckDB data source into this code.
It might be pretty easy to use overloading to get special case implementations that form SQL queries progressively until the results need to be materialized as something like a dataframe for the function code to work on.
Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here.
maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point.
If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out reports, they will be up for a really bad experience.
Please educate people to do the right thing and realize the scope of the work they are facing, it might feel that it hurts your growth in the short term but it will benefit you greatly in the mid-long term as a vendor.
IDK, AWS Zero ETL from Autora into Redshift really helped us at some point. You right that data transformation is very limited if not possible. But having data in an analytical store, being able to experiment with queries, understand what is wrong with your OLTP schema and then build ETL is way better than doing an upfront design.
Of course it is. What you describe is one of the reasons that ELT became popular, if you couple it with a variant type and schema on read, you have a very powerful and flexible architecture.
But there’s no free lunch, building and maintains data infrastructure that is reliable requires work. Many companies don’t realise that when they start their analytical journey and aggressive marketing doesn’t help. That’s the point I was trying to make.
I don’t disagree, just placing emphasis on a different aspect.
In an ideal world there is a tool that moves your schema into an analytical store “as is”
with a single click. Then the same tool lets you add arbitrary transformations of the data. Surprisingly I have not come across such a tool. It is earthier “one click to move your data” or “any transformation you want” but only after a significant upfront investment :(
I think I didn’t articulate myself very well on my reply. I actually wanted to say that I agree with you and emphasise again the need for educating users for the complexity of these projects.
What you describe has been pitched by many different products for different parts of the data platform. Fivetran for example claims to do that for the extraction and loading part, good old Informatica was offering the ETL in a graphical interface etc.
The problem that many teams ended up having is the explosion of the tooling needed by data teams.
The comments are definitely not worth reading. It’s a very sad thread, you literally had to go through all of them to find one that wasn’t about hate and stating some facts about the issues of the code.
I found them worth reading for the following set of thoughts came up:
- programmers had problems with delivering quality long before LLM’s
- very much research and tools went into that, bringing us {Git, libraries, VSCode, reviews, …,} but the human factor stayed the same (and more pronounced imho than in other fields of engineering)
- LLMs democratized programming, enhancing a few, dropping the bottom to no skill programming
- the tools and practices created for the quality problems from the past turn out to be wholly incapable of maintaining quality in the present
The main problem behind this is that those delivering the QA tools of the past are central in the AI race. Old school engineering would separate these concerns.
This is a good article and hopefully it served its purpose of generating awareness for the platform but I have to disagree with the core claim of the article.
Distribution was always the hardest part in building a business, sure today it’s easier to vibe code something in a day or two but there is a reason that for a long time now the standard advice to technical founders has been to not build before you have conviction on the product market fit.
And product market fit means that you have also figured out how to get to the market, aka distribution.
So maybe now the advice could be to build a passable mvp and then use that to figure out the product-market fit.
It's a lot easier to think about it and discuss it with others when the product is tangible instead of just an idea. Biggest challenges then are being able to still diverge (broaden/pivot the product idea) and killing functionality that doesn't work out afterall.
This is a great post and I really appreciate making the cognitive science terminology clear.
the author is doing a great job telling what is missing from the current memory frameworks for agents but what is missing in my opinion is also an argument about the necessity or not of these missing components.
fair — this post mapped the gaps without making the case for whether filling them changes what an agent can do. the interesting ones are procedural and prospective. both deserve their own post.
Hopefully I didn’t sound too critical of the post because this wasn’t my intention. The post delivered what was needed and thank you for this!
The reason I asked the question is because in the case we don’t need the rest, it would be better to not use this terminology for these systems. We already anthropomorphize LLMs too much and although I get the marketing value of that, it’s not always to the benefit of the people who interact with them.
yeah i agree with you on not using the terminology, although it's intuitive it's also confusing enough. it's tempting to do that, but i share your sentiment
Thanks for writing this, and look forward to the one on procedural memory.
Seems like teams are encoding procedural knowledge in skills repositories, and I wonder if there’s additional utility from an auto created procedural memory layer
* Large language models amplify inconsistent technology and quietly reinforce consistent ones. *
This is another way of saying that the tools you equip the LLM affect their effectiveness, in other words, the harness you build around them matters and matters a lot.
At the end of the day, the language you pick, enriches the harness with the toolchain, libraries etc. it offers. This is most evident with the toolchain as the author mentioned but if you think about it, picking a specific framework that constraints the choices the model can make (e.g. the Ruby on Rails example) is also affecting the behavior it has.
My understanding so far is that that Mythos (and any model in general) can produce candidate reasoning but you really need a system around that reasoning that is capable of producing auditable security findings.
So, success is coming not just from the model but also from the harnesses they built around it. The Cloudflare post was more detailed on that front and I wish the rest would share more about it.
I wonder though, is such a dialect better for agents? Have you tried to measure if an agent performs better expressing queries in such a language instead of SQL?
reply