This week I've been debugging a performance issue in a Ducklake and knocking out some good improvements, but this precise symptom—read_parquet() making tons of tiny range requests to load a parquet's footer from S3—was still baffling me. How lucky am I that I stumbled on your comment! Amazing, thank you for the pointer.
> read_parquet() making tons of tiny range requests to load a parquet's footer from S3
This is somewhat unavoidable if you have a number of files and a tight filter which rejects most of them. The footer choice was a bad one in hindsight (not for HDFS), but for anything which uses XOR or other kind of erasure coding. The other other file format I worked with used headers only instead of footers, to save a few ms in latency for the initial GET
I don't think Ducklake can maintain a zone-map or ART index over parquet files, but there is no equivalent of iceberg metadata/something which has min/max/HLL compacted across multiple files yet.
The secondary indexes with file identifiers stored in a sequential file is most likely the next useful thing duckdb can make.
Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results)
Is it even worth using apple's? Maybe they have improved it but when they first started using it to transcribe voice memos it was laughably bad (in my experience). I defaulted to whisper.cpp
I used it a lot in my AI client before I finally got around to hacking a Deepgram API server into Handy: It sucks enough to give me a headache worrying about its unreliability in every given transcription.
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
interesting! I only tested against Parakeet v2 because my agent suggested it would be faster for English transcription (v3 is multilingual but somewhat slower, it suggested)
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
As much as I hate vibe-coded "slop". I feel like this is idiotic. I and most good engs with decades of experience do use AI now to type nearly all the actual "code", but we read EVERY SINGLE line, always fix and rework every feature before a commit to make it clean, optimized, and de-slopified.
But still, we wouldn't be allowed to use codeberg because "generative ai" was used significantly for most lines of code?
AFAICT they are dealing with resource exhaustion from slopcoding agents hammering them with copycat garbage. It's existential. If you use AI to help you thoughtfully write code, review it with humans, and respect their service, I don't think they will have a problem with you.
Maybe not... I personally wouldn't use a service that would boot me off if they did find out because I spoke about it online or committed too much code one week or something.
right, but say you give the agent access to github and it can push as you, or make a gist; now it can easily exfiltrate your secret.
And that's just an easy case - really if it has any network access at all it can come up with a clever way to route a request through the network such that the key comes back somewhere in the request. If you scan for it inbound too, the machine can obfuscate it.
Our agents are trained to be so intensely helpful and they have such intricate knowledge of how things work that they will do some incredibly clever tricks to do what you ask them to do.
The agent has no access to the secret. It has a placeholder that is replaced at a higher level. When it makes the network request the secret is substituted but that is outside of the caller's worldview.
microsandbox maintainer here. the custom certificate is installed in the guest's trusted root CA list, so it should work across any program, except where the program opts to explicitly pin certificates for a destination.
Yes, I read it. That means that it doesn’t work in those cases. Btw, as a developer it’s very easy to have something like that. It’s not as trivial as it seems at all. I encountered with similar problems all the time, with similar solutions (mainly for security theater reasons) in the past. There are websites which simply doesn’t work if you replace certificates, regardless of browser or CA for example.
Yes, I've worked with people who have run into issues with "security" solutions like ZScaler. I have tried it with some APIs (like GitHub) and it does work. Not to say it will work in your case.
Operating systems ought to be providing us the utilities we need to safely sandbox processes (agent or otherwise), but they appear to not be interested in the job
This is what I currently do, but my software uses docker and docker mounts act as a bypass for the file system restrictions, plus docker processes started outside the sandbox allow network proxy escape.
Currently, I don't allow the agent access to docker, start docker myself, and then do short-lived sandbox-free sessions when the agent needs to do things that interact directly with docker; but that's annoying.
"Honest assessment: I was wrong to say I was being straight with you. You pointed out that a "smoking gun" is a sign of evidence, and I clearly didn't have any. This is not a bug but a gap that can be fixed like [this]. Give me the word and I'll wire it in."
··You've hit your monthly spend limit · raise it at claude.ai/settings/usage
no, definitely not
reply