This seems like something that was evident for AI to disrupt into. I recently went through an entire underwriting process when getting my first mortgage and honestly was shocked with how slow and old-fashioned the process was. Even had to fax a few documents (!?).
Thanks for the comment! Our approach has been very much depth-first rather than breadth-first. We’re laser-focused on the private credit industry for now and plan to stick with B2B for the foreseeable future. The biggest takeaway for us while building Arc Intelligence has been that accuracy is everything—and hitting those high accuracy levels really comes down to staying focused.
We were inspired by datalakes and thought the name of a super deep lake could be a cool domain. Turns out 10 of the deepest spots on Earth are all trenches, and the domain was cheap, so we went with trench.dev
https://www.marineinsight.com/know-more/10-deepest-parts-of-...
In my experience, at scale (~2-3k QPS), you'd run into a bottleneck ingesting so many events without Kafka. If you don't have this level of throughput, you could totally do the above and still get the advantages of ClickHouse's columnar datastore.
The stack is indeed very similar to Posthog. The biggest difference is that we don't come with all the feature bloat (Session Recordings, Feature Flags, Surveys, etc.) and instead provide a very minimal and easy to use backend + API that is applicable to a ton of use cases.
We (Frigade.com) actually use Posthog as well as Trench in production. Posthog powers all our website analytics. Trench powers our own SDK and tracking scripts we ship to our own customers.
I actually tried to spin up Posthog originally before building Trench, but there was just way too much overhead and "junk" we didn't need. I would need to strip out so many features of their Python app, it would eventually be faster to build a clean solution in Typescript ourselves.
Yes for sure. We intentionally designed Trench to be very unopinionated when it comes to the application. So you can use it to stream and query anything from page views, log traces to IoT object events.
Thanks for flagging. Just fixed this.
Grafana is intentionally not included by default -- but it takes a few minutes to set it up. We're still trying to figure out what to bundle by default in terms of UI -- for now it's API only.
No worries. I am going to test it as we are looking for a simple centralized tool for multiple customers to run reporting on events. Most tools have been too complex to setup and yours is promising.
Once you've outgrown a single physical server, you can continue to scale the Trench cluster by spinning up more Trench application servers and switching to dedicated Kafka and ClickHouse (either self-hosted or via cloud offerings). You can also shard Trench itself depending on the structure of your data (e.g. 1 Trench instance per customer, use case, etc.)
Auto-archiving to cloud for Kafka (Confluent, AWS KMS, etc.) / ClickHouse (ClickHouse Cloud, etc.) is definitely high on the roadmap.
I think a major difference is that Jitsu depends on you having a data warehouse whereas Trench can be spun up as a standalone system. The nature of Trench's data is also to enable real-time querying a high scale which will be much slower when depending on ETL'ed data in a data warehouse.
1) All data is partitioned based on the "instanceId" of events (see `instanceId` here: https://docs.trench.dev/api-reference/events-create). Instance IDs are typically a logically meaningful way of separating users (such as by company/team/etc.) that allows for sharding the data across nodes.
2) Yes, this the number 1 thing on our roadmap right now (if anyone is interested in helping build this, please reach out!)
3) We're using the Kafka engine in ClickHouse for throttling the ingestion of events. It's partitioned by instanceId (see #1) for scaling/fast queries over similar events.
4) My benchmarks in production showed a single EC2 instance (16 cores / 32 gb ram) barely working at 1000+ inserts / second with roughly the same amount of queries per second. Load averages 0.91, 0.89 0.9. This was in stark contrast to our AWS Postgres cluster which continued to hit 90%+ CPU and low memory with 80 ACUs, before we finished the migration to Trench.
5) We seemed to solve this by running individual Node processes on every core (16 in parallel). Was the limit you saw caused by ClickHouse's inbound HTTP interface?
6) Right now the system uses just a default MergeTree ordered by instanceId, useId, timestamp. This works really well for doing queries across the same user or instance, especially when generating timeseries graphs.
7) I am still trying to figure out the best Kafka partitioning scheme. userId seems to be the best for avoiding hot partitions. Curious if you have any experience with this?
Let us know how the migration goes and feel free to connect with me (christian@trench.dev).
How do you guarantee ACID with Kafka being responsible for actually INSERT'ing into ClickHouse? Wouldn't it be less error prone to just use ClickHouse directly and their async inserts?
I am thinking about setting this up as as a configuration for the type of traffic that doesn't require Kafka.
That being said, Kafka has in my experience come in super handy again and again, simply because it adds an incredible extra layer of fault tolerance when running at scale, including the ability to replay events, replicate, fail over, etc. I'd be nervous about letting the amount of throughput we receive directly interface to ClickHouse (though I'd be excited to run an experiment with this).
Not sure of the CH Kafka engine but generally I think you should partition by userId.
Because the next step would be trying to run some cron for a user or event based trigger based on the events.
And the only way to avoid multiple machines doing the same work / sending the same comms - would be to push all users events to a partition. This way with multiple workers you don't have the risk of duplicate processing.
check "partial ordering" concept. What is the minimum independent "thing"? Probably user?
example over user+invoices:
i.e. there are things that have to come in exact order (e.g. activity on certain invoice), and there are things that can move around (i.e. processing those, timewise), being independent from one another (different invoices' activities, wholesale). But when same user acts on different invoices, then whole one-user-activity should be in exact order.. not just invoice-activity
I looked around, but all the open source analytics projects I could find were bloated with all kinds of UI and unnecessary code paths. They also all seemed to use row-based RDMS as the data backbone (vs columnar stores like ClickHouse). I was looking for a backend-only solution that we could shape for our product use case that could scale.
So TLDR, if you're at a smaller scale (<1M MAUs), you probably will be fine just using a table in MySQL or Postgres. If you have a lot of traffic and users, you will need something like Trench that uses Kafka and ClickHouse.
You are selling the underlying technologies(Kafka/Clickhouse).
I'm interested in your project can do for me, my project(s), team/company. There is a reason that most of the internet still uses PHP and old technologies. Because they focused not on the latest tech but on solving problems for others.
It seems pretty clearly spelled out. If you have enough traffic that an events table is slowing down your postgres instance, you can easily set this up as a service to offload the events table. The author says <1 million MAUs, and you probably don't need this.
It's built on tech known for handling very large amounts of traffic, which answers the how after the what.
Use case #1: You have a problem table (e.g. a high-volume events table) that grows non-linearly as your business starts to scale up. A queue + columnar store package like Trench moves the problem table out to a system better equipped to deal with it and lets your DB server handle its relational business in relative peace and quiet.
Maybe I wasn't clear enough but my questions have been rhetorical. They were not for me. If one starts stating technologies, it is akin to describing the individual ingredients of a sandwich.
The question remains: Why choose Trench over just using Kafka and Clickhouse or any other message queue and columnar database / big data base?
If the goal of the post and the landing website is to entice people to use the tool, then answering these questions is important. If what is being discussed seems obvious, then who is the target demographic? Because they already know the space, use alternatives or have built their own.
Is your target market consumer or b2b financiers?