At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there.
If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.
I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
The ultimate source of truth about what orders you sent to the exchange is the exact set of bits sent to the exchange. This is very important because your software can have bugs (and so can theirs), so using the packet captures from that wire directly is the only real way to know what really happened.
Among all the software installed in a reputable Linux system, tcpdump and libpcap are some of the most battle tested pieces one can find.
Wireshark has bugs, yes. Mostly in the dissectors and in the UI. But the packet capture itself is through libpcap. Also, to point out the obvious: pcap viewers in turn are auditable if and when necessary.
Cisco switches can mirror ports with a feature called Switch Port Analyzer (SPAN). For a monitored port, one can specify the direction (frames in, out, or both), and the destination port or VLAN.
SPAN ports are great for network troubleshooting. They're also nice for security monitors, such as an intrusion detection system. The IDS logically sees traffic "on-line," but completely transparent to users. If the IDS fails, traffic fails open (which wouldn't be acceptable in some circumstances, but it all depends on your priorities).
No, really, I get where you and your parent are coming from. It is a low probability. But occasionally there is also thoroughly verified application code out there. That is when you are asking yourself where the error really is. It could be any layer.
It’s the closest to truth you can find (the network capture, not the drop copy). If it wasn’t on the network outbound, you didn’t send it, and it’s pretty damn close to an immutable record.
It makes sense. I'm a little surprised that they'd do the day to day reconciliation from it but I suppose if you had to write the code to decode them anyway for some exceptional purpose, you might as well use it day to day as well.
Storage is cheap, and the overall figures are not that outlandish. If we look at a suitable first page search result[0], and round figures up we get to about 700 GB per day.
And how did I get that figure?
I'm going to fold pcap overhead into the per-message size estimate. Let's assume a trading day at an exchange, including after hours activity, is 14 hours. (~50k seconds) If we estimate that during the highest peaks of trading activity the exchange receives about 2M messages per second, then during more serene hours the average could be about 500k messages per second. Let's guess that the average rate applies 95% of the time and the peak rate the remaining 5% of the time. That gives us an average rate of about 575k messages per second. Round that up to 600k.
If we assume that an average FIX message is about 200 bytes of data, and add 50 bytes of IP + pcap framing overhead, we get to ~250 bytes of transmitted data per message. At 600k messages per second, 14 hours a day, the total amount of trading data received by an exchange would then be slightly less than 710GB per day.
Before compression for longer-term storage. Whether you consider the aggregate storage requirements impressive or merely slightly inconvenient is a more personal matter.
And compression and deduplication should be very happy with this. A lot of the message contents and the IP/pcap framing overheads should be pretty low-entropy and have enough patterns to deduplicate.
It could be funny though because you could be able to bump up your archive storage requirements by changing an IP address, or have someone else do that. But that's life.
Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header.
Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.
Typically not a literal pcap. Not just wireshsrk running persistently everywhere.
There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier.
Except it is literal “pcap” as they capture all packets at layer 3. I don’t know the exact specifications of Pico appliances, but it would not surprise me they’re running Linux + libpcap + some sort of timeseries DB
Well, probably, but I meant more like it's not typically someone running tcpdump everywhere and someone analyzing with Wireshark, rather than a systems configured to do this at scale across the desktop.
I don't think that's what anyone was assuming. A "pcap" is a file format for serialized network packets, not a particular application that generates them.
Yeah, FIX or whatever proprietary binary fixed-length protocols (OUCH or BOE for example) the venue uses for order instructions.
Some firms will also capture market data (ITCH, PITCH, Pillar Integrated) at the edge of the network at a few different cross connects to help evaluate performance of the exchange’s edge switches or core network.
Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.
As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem).
The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.
Yeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies.
The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades.
You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you.
The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.
Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense.
Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
> The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements.
If you're just using PG as a convenient abstraction for a write-only event log, I'm not completely opposed; you'd want some strong controls in place around ensuring the tables involved are indeed 'insert only' and have strong auditing around both any changes to that state as well as any attempts to change other state.
> In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
But it -must- be write-only. If you really did have a bug fuck-up somewhere, you need a compensating event in the log to handle the fuck-up, and it better have some sort of explanation to go with it.
If it's a serialization issue, team better be figuring out how they failed to follow whatever schema evolution pattern you've done and have full coverage on. But if that got to PROD without being caught on something like a write-only ledger, you probably have bigger issues with your testing process.
Footnote to QLDB: AWS has deprecated QLDB[1]. They actually recommend using Postgres with pgAudit and a bunch of complexity around it[2]. I'm not sure how I feel about such a misunderstanding of one's own offerings of this level.
Yeah. I'm surprised it didn't get enough uptake to succeed, especially among the regulated/auditable crowds, considering all the purpose built tech put into it.
> Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log;
Isn’t this how crypto coins work under the hood? There’s no actual encryption in crypto, just secure hashing.
Theoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.
Is a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.
"Prevents tampering" lacks specificity. git is a blockchain that prevents tampering in some aspects, but you can still force push if you have that privilege. What is important is understand what the guarantees are.
? If I use something like Blake3 (which is super fast and emits gobs of good bits) and encode a node with say 512 bits of the hash, you are claiming that somehow I am vulnerable to tampering because the hash function is fast? What is the probable number of attempts to forge a document D' that hashes to the very same hash? And if the document in structured per a standard format, you have even less degrees of freedom in forging a fake. So yes, a Merkel tree definitely can provide very strong guarantees against tampering.
Fwiw, increasing the BLAKE3 output size beyond 256 bits doesn't add security, because the internal "chaining values" are still 256 bits regardless of the final output length. But 256 bits of security should be enough for any practical purpose.
Good to know. But does that also mean that e.g. splitting the full output to n 256 chunks would mean there is correlation between the chunks? (I always assumed one could grab any number of bits (from anywhere) in a cryptographic hash.)
You can take as many bytes from the output stream as you want, and they should all be indistinguishable from random to someone who can't guess the input. (Similar to how each of the bytes of a SHA-256 hash should appear independently random. I don't think that's a formal design goal in the SHA-2 spec, but in practice we'd be very surprised and worried if that property didn't hold.) But for example in the catastrophic case where someone found a collision in the default 256-bit BLAKE3 output, they would probably be able to construct colliding outputs of unlimited length with little additional effort.
In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.
> In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.
Yeh, but that's kinda my point: if your primary use case is not "needs to be distributed" then there's almost never a benefit, because there is always a trusted authority and the benefits of centralisation outweigh (massively, IMO) any benefit you get from a blockchain approach.
This doesn't address the question in any way except to note that you also use Merkle Trees. Do you reply to any comment mentioning TypeScript with a link to your Show HN post as well?
What disrespectful marketing. We don’t care that you use Merkle trees because that’s irrelevant. I guess I can add Fireproof to my big list of sketchy products to avoid. It’s embarrassing.
While your intentions may have been around discussion, I don’t want to be marketed to when I’m trying to understand something unrelated. I have a business degree so I intimately understand that HN is technically free and it’s nice to get free eyeballs, but we are people too. I’m so much more than a credit card number, yet you’ve reduced me to a user acquisition in the most insulting way possible.
Perhaps instead of your ideas, it’s worth seeding your own personal make up with a firm statement of ethics??
Are you the kind of person who will hijack conversations to promote your product? Or do you have integrity?
Just purely out of concern for your business, do you have a cofounder who could handle marketing for you? If so, consider letting her have complete control over that function. It’s genuinely sad to see a founder squander goodwill on shitty marketing.
In founder mode, I pretty much only think about these data structures. So I am (admittedly) not that sensitive to how it comes across.
Spam would be raising the topic on unrelated posts. This is a context where I can find people who get it. The biggest single thing we need now is critical feedback on the tech from folks who understand the area. You’re right I probably should have raised the questions about mergability and finality without referencing other discussions.
Because I don’t want to spam, I didn’t link externally, just to conversation on HN. As a reader I often follow links like this because I’m here to learn about new projects and where the people who make them think they’ll be useful.
ps I emailed the address in your profile, I have a feeling you are right about something here and I want to explore.
> Spam would be raising the topic on unrelated posts.
I think you need to reread the conversation, because you did post your marketing comment while ignoring the context, making your comment unrelated.
If you want it distilled down from my perspective, it went something like this:
> Trog: Doubts about the necessity of Merkle trees. Looking for a conversation about the pros and cons of Merkle trees and double ledger accounting.
> You: Look at our product. Incidentally it uses Merkle trees, but I am not going to mention anything about their use. No mention of pros and cons of Merkle trees. No mention of double ledger accounting.
Crypto/Blockchain makes it harder to have an incorrect state. If you fk up, you need to take down the whole operation and reverse everything back to the block in question. This ensures that everything was accounted for. On the other hand, if you fk in a traditional ledger system you might be tempted to keep things running and resolve "only" the affected accounts.
It's mostly a different approach to "editing" a transaction.
With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest. The difference is that you've got a ledger that's clearly a fork because of cryptographic signing.
With a traditional ledger, you fix the wrong transaction in place. You could also cryptographically sign them, and you could make those signatures depend on previous state, where you basically get two "blockchains".
Distributed trust mechanisms, usually used with crypto and blockchain, only matter when you want to keep the entire ledger public and decentralized (as in, allow untrusted parties to modify it).
> With a traditional ledger, you fix the wrong transaction in place.
No you don’t. You reverse out the old transaction by posting journal lines for the negation. And in the same transactions you include the proper booking of the balance movements.
You never edit old transactions. It’s always the addition of new transactions so you can go back and see what was corrected.
> With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest.
You're handwaving away a LOT of complexity there. How are users supposed to trust that you only fixed the transaction at the point of fork, and didn't alter the other transactions in the replay?
My comment was made in a particular context. If you can go back, it's likely a centralized blockchain, and users are pretty much dependent on trusting you to run it fairly anyway.
With a proper distributed blockchain, forks survive only when there is enough trust between participating parties. And you avoid "editing" past transactions, bit instead add "corrective" transactions on top.
It's a question of business case. While ensuring you are always accounted correctly seems like a plus, if errors happen too often potentially due to volume, it makes more business sense sometimes to handle it while running rather than costing the business millions per minute having a pause.
Any time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it.
A good option is “what would happen if we” instead of anything involving the word “just”.
“Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.
Sure. When I ask "why don't we just" I'm suggesting that the engineering solutions on the table sound over-engineered to the task, and I'm asking why we aren't opting for a straightforward, obvious, simple solution. Sometimes the answer is legitimate complexity. Equally as often, especially with less experienced engineers, the answer is that they started running with a shiny and didn't back up and say "why don't we just..." themselves.
Counterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.
Apologies, my wife is not keen on me sharing Strava profile on the web as people can basically see where we live (UK countryside and we are the only house in the area!).
Low 15:00s. I doubt I'll be able to break 15 though as I'm just getting too old now. I used to train seriously when I was a teenager so maybe those adaptations stick with you as you get older.
It's a good question. I find that the most important thing is running at or close to race pace. I follow Canova's training philosophy quite closely. Definitely recommend reading the runningwritings link I posted above.
I do between 80km to 100km a week.
Most of my running is done at a moderate pace. I just do this to get mileage in. Most of it is quite hilly. Looking at about 300-400m elevation for each 10km and runs range from 10km to 20km. Pace is around 4:00/km to 4:20/km.
If I do a workout, I follow it with a recovery run. Pace around 4:40/km. If a super hard workout then I do two days of recovery runs.
I never run doubles because I have young kids and don't have the time.
The most important ingredients are the workouts. I never repeat the same workout. A typical one for 5km training would be: a few km warm up, then 3km @ 97% 5km pace, 2 min moderate, 2km @ 5km pace, 2min moderate, 1km @ 103% 5km pace, then a recovery pace warm down. About 15km in total. That's a hard workout and would be followed with easy running the next day. I would do that workout to improve lactate buffering and improve anearobic capacity.
For lactate threshold I'd do something like 20x400m at LT pace with walking rest for 3 minmutes inbetween. This is where I might use the hyperbolic model to figure out when all the metabolic byproducts are flushed out so you can basically do the next 400m mostly recovered.
Other workouts might be aerobic thresohld runs. So marathon pace running for 15km to 30km. These are the long fast runs that canova talks about. They are hard and you need a lot of recovery after. I really find they improve performance quite alot. They are not presecribed in most training plans and I feel they are a key missing ingredient.
I do a fair bit of hill reps and sprints as well. They are probably best for improving VO2max. Lots of rest inbetween reps, you need to be fully recoved for the next one to get the benefit.
Lastly, you need to keeo touching on all the various metabolic systems. If you don't use it, you lose it. Keep your training varied. Sprints, short interbvals, long intervals, long fast runs, hill reps, ladders, alternates, with varying levels of recovery and at different speeds. Make sure your training is additive. Don't remove any class of session, just add more different types.
Do strength/core workouts. Plyometrics. You need to be strong to bash out miles at MP and not get injured.
I use theragun after any hard workout. Helps recovery loads.
Eat. Alot. On hard workout/long run days I do about 5000kcal.
Most importantly, I think, don't listen to any of the 80/20 polarised nonsense, MAF method, or anything like that. One size fits all training regimes don't work very well because everyone is different. They work for beginners because _any_ running will have them improve.
For more experienced athletes, to run fast you need to practise running fast at the paces you will race at. So doing a training plan which prescribes 80% of easy running as a rule is no good. You only run easy if you need to recover. Every other workout/session needs to target some metabolic system to promote adaptinos which help you run faster. It's the workouts which really make a difference to your race performance.
Simply because I find it gets me much better results. I know that to run a 15 minute 5km I need to be able to run that pace in training. I don't really care what my lactate threshold or VO2max is other than it's just interesting to know. I use a critical speed (pace/duration) model to know what paces are metabolically steady state and which are not. So based upon how I perform in training, I know pretty much what paces should be within the realms of possibility for a maximal effort and I use that to guide my training.
edit: I'll just add that I do use physiology to guide the training paces. But when you've been training for a while you know that 1) Marathon pace is aerobic threshold pace. 2) Half maraton is lactate threshold pace. 3) 5km is critical speed. 4) 10km is between critical speed and lactate threshold. So whilst I do use physiology to guide trianing paces I don't use a zone system or anything like that. I just design workouts which I know touch on the various systems I need to improve.
They stopped the one child policy nine years ago; they had it in the first place because demographic projections had them severely overpopulated if they didn't.
During the period in which it was active, their GDP increased by a factor of about 62. Not percentage, multiple.
They reversed the policy too late. Their rapid growth was them seizing the low hanging fruit upgrading from a country of almost entirely peasants to slightly fewer peasants, all funded by Western greed. Those gains will never happen again.
Entrenched and connected types tend to exit industries before the gavel hits, and enforcement from the CCDI isn't impartial, with plenty of bribery to remain off their radar.
Truly Schumpterian creative destruction is good in a vacuum, but reality isn't a vacuum.
There is no such thing as "best possible model, full stop". Models are always context dependent, have implicit or explicit assumptions about what is signal and what is noise, have different performance characteristics in training or execution. Choosing the "best" model for your task is a form of hyperparameter optimization in itself.
Plenty of places use DL models, even if it's just a component of their stack. I would guess that that gradient-boosted trees are more common in applications, though.
Still mostly NLP and image stuff. Most actual data in the wild is tabular - which GBTs are usually some combination of better and easier. In some circumstances, NN can still work well in tabular problems with the right feature engineering or model stacking.
They are also more attractive for streaming data. Tree-based models can't learn incrementally. They have to be retrained from scratch each time.
ML is very good at figuring out stuff like every day at 22:00 this asset goes up if this another asset is not at a daily maximum and the volatility of the market is low.
You might call this overfitting/noise/.... but if you do it carefully it's profitable.
Real-time parsing of incoming news events and live scanning of internet news sites - coupled with sentiment analysis. Latency is an interesting challenge in that space.
Multiple parts of the iPhone stack run DL models locally on your phone. They even added hardware acceleration to the camera because most of the picture quality upgrades is software rather than hardware.
Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.
This led to an argument with my boss, who assumed things magically worked.
Days later, we received an email announcing an audit one one of these accounting discrepancies.
JPMC proposed using crypto, internally, to consistently manage cash flow.
Not sure if it went anywhere.
reply