Also with the M1 Pro MacBook, my LG USB-C monitor broke during the upgrade. Black screen, same on another connected USB-C laptop. Tried different cables, unplugging power from the monitor and factory reset on the monitor, no luck. Other inputs like HDMI and DisplayPort still work. I don't think the breakage during the upgrade is a coincidence.
I would really advise against having anything connected to the Macbook during upgrades, except the charger...
One approach that can be used is to use the centralized service to answer a broader question like: given this user, what rules can I use to know if a document is accessible for them. And have the service give you a set of rules to apply. Then take the result and embed those restrictions in your query.
An example access service response would be: this user can access data from groups they are part of + documents for which a share exists towards this user + documents for which a share exists to any of the users' groups.
This is not exactly the same as the first option you described, because instead of storing access controls in the index data, you use the available metadata + the rules from the access control service.
Great product! I think with the updates and deletes it will be applicable to many more scenarios. HTAP is really getting important now that many more apps have user-facing analytics. In our case, we use plain PostgreSQL at the moment but I'm always on the lookout for PostgreSQL extensions. With Citus, Timescale and now Hydra this space is exciting!
I noticed by reading the documentation that the fact this is built on top of a Citus columnar fork is a bit hidden (mentioned only in a blog post). Why did you choose to fork rather than contribute to the Citus columnar project directly? AFAIK they also want to add update and deletes and vectorized execution.
There's also Rancher Desktop in the same space, which includes k3s as a local K8s solution.
For personal use I found it great and lighter than Docker Desktop. At work, unfortunately all options but Docker Desktop have issues with either 1) Our Cisco AnyConnect VPN, or 2) Our authenticated http proxy. Couldn't find anything else providing a container runtime + a local k8s on MacOS that works in this environment. So we just got Docker Desktop licenses.
I user Rancher Desktop on an i9 with 32gb of RAM. Starts in less than a minute. I also have Teams and slack. Sometimes I have over 200 browser tabs open (yes, I have a problem). The UI is responsive pretty quickly.
A lot of delays has to do with starting VMs. You need this for Linux on Mac/Windows.
Disclaimer: I started Rancher Desktop. I might be biased.
Last week I switched from Rancher Desktop back to Docker Desktop because I couldn't get VSCode Dev Containers to work properly. I was stumped because it should work out of the box. However it didn't work on a fresh install of Docker Desktop either. Apparently when Rancher Desktop was first released I've installed it and setup a Docker alias in my ~/.zshrc:
alias docker=nerdctl
After removing that alias everything worked first try with Docker Desktop. However after starting up a couple of Dev containers and some debugger my machine crawled to a halt and was memory swapping like there was no tomorrow. I found that this behavior could be normal for Docker Desktop. So I think I'm going to switch back to Rancher Desktop (or perhaps Podman Desktop) sooner than later.
Sorry for using this comment as a way to get ahold of you but there is no DM function on HN and the comment I was wondering about was posted by you 18 days ago and the comments are locked now.
I have been controlling my water heater with HA for a few months and I too am risk averse when it comes to legionella. You have taken it to another level by replacing the sensor inside with a DS18B20. I was also interested in doing this but I don't really want to drill into it. How did you install the sensor? Is the water not under pressure? I've just measured water temperature at the tap with a meat thermometer and under flow into a container to determine that the temperature falls in the range that is safe for legionella growth. Would love to look into a proper sensor again if you have any information about that.
Again, sorry for hijacking your completely unrelated comment.
First of all, I'm also concerned by legionella growth. However all the sources I can find suggests that at a legionella run above 60 degrees celsius once a week should be enough to kill all bacteria. So that's what I'm doing.
My sensor installation was extremely easy. My warm water heater is in fact a barrel within a barrel. And some insulation material between those barrels. If it would only be metal, you lose too much heat. In the outside barrel (excuse me for lack of a better term) there was an analogue thermometer. This thermometer has a metal back so it makes direct contact with the inside barrel. I just pulled this one out and replaced it with a DS18B20 probe. Again metal against metal, so maximum contact.
Finally I have calibrated the sensor by running it with hot water at the tap and measuring the temperature both there and on the warm water heater (with 2 DS18B20). I've done this for several temperatures with intervals of 10 degrees. I've ignored possible sensor deviations. Finally I used Excel's INTERCEPT and SLOPE functions on the range to calculate the value needed for a linear equation. I have used the formula:
boiler_temperature * SLOPE + INTERCEPT
My math is probably far from perfect and I might revisit it one day. But it works for me currently. I also visualized the measurement results with my calculations and they seem to be pretty accurate. Especially when accounting for missing measurements.
I've also added my personal page to the 'About me' on here, and I have the same Reddit username if you want to DM me there.
"Docker Desktop" uses a VM for container execution on all supported OSes - Windows, macOS and Linux - default installs are not running "docker" natively at your local CLI if you are using "Docker Desktop" to run docker, even on Linux.
"Docker Desktop on Linux runs a Virtual Machine (VM) so creates and uses a custom docker context desktop-linux on startup." I had previously assumed it would be native on Linux, but apparently not. To be clear, this only applies when talking about "Docker Desktop" installations - not the same thing as "docker" etc etc.
I also think the performance is pretty bad, and I've used it on all three OSes at one time or another. I simply never install "Docker Desktop" on Linux typically anyway - it adds so little value over a basic native local docker install there.
The only thing I think "Docker Desktop" is really great at is creating FUD regarding using the alternatives or even plain ole free "docker", but that is probably its primary means of generating revenue for the company - I've seen docker desktop licences get deployed everywhere recently, regardless of the merits. So many users I encounter don't even understand the distinction now between docker/docker-desktop, or that there is one.
Running in a VM is there for very good reasons. If you look at where things are going to be installed on the base system, how you can reset the environment, how you deal with variations with other things installed on the host, and more. It's difficult to do this all well outside of a VM.
Also, there are many people who want that VM boundary. We found this when designing for Linux in Rancher Desktop and talking with people about it.
The DD WSL2 backend is also creating a VM in Hyper-V. Actually, it's creating two VMs (docker-desktop and docker-desktop-data). It's also running a proxy in your WSL2 VM so you can access the docker server. It's all a bit convoluted TBH.
I actually decided today to stop using DD on my Windows machine and just run docker native inside the WSL2 VM instance instead. Still not sure what solution I'm going for on my Mac.
This is the way. It's even easier if you just winget install RedHat.Podman - that'll give you a tiny Fedora image where "docker" (podman) just works straight out of the box. No need to worry about getting iptables-legacy packages for your WSL distribution or whatever. It's so simple and lightweight, it feels like a much better solution than anything Docker ever did for Windows.
I do the same for the most part. I have the docker daemon setup both locally in Windows and also in WSL2. I then have multiple contexts setup in Windows so that I can easily switch between Windows/Linux containers from my host terminal. Thus far, I've not experienced any issues.
The initial setup was a little more complex than just running Docker Desktop, but since then, it's running flawlessly.
Talking about WSL2 in general: it creates one VM, with mount namespaces per "distro". That's why everything shares the same network (they didn't set up separate network namespaces). Also, the GUI support in Windows 11 is a separate mount namespace.
Well, yes, but it creates those when you log in. The proxy fixes some corp firewall issues (see other comments here). Start up is faster overall for me versus the normal backend.
Also known as a VM, but maybe you get better warm up times. My main complaints are with the "docker desktop" app experience - not so much "docker" itself VM or otherwise. It adds so little to docker for the license cost too, at least so far.
You can simply install docker in a VM/WSL2 natively yourself and avoid docker desktop/licensing altogether. "Docker Desktop" is/was really a tool to simplify getting a Linux Kernel (the critical dependency for cross-platform container dev) on non-Linux platforms IMO, which seems wild to pay for to me in its current state and when Windows has a great built in VM via WSL2 anyway. That it even exists on Linux now (recent addition) is kinda amusing - on MacOS and Windows there is at least some argument to be made it simplifies getting the kernel...
The WSL2 backend is pretty broken from my experience, it will often lock up using 100% CPU after I put my laptop into sleep/hibernation, and the only reliable way to bring it back seems to be killing the VM in task manager and restarting. Dunno if this is Windows' fault or Docker's though.
Yup. I have a Mac Pro 2019, 12 core Xeon, 192GB, and the Desktop app and VM location reside on a RAID0 M.2 array that can throughput 6GB/s. Still a minute+.
You can use wsl-gvproxy(*), which uses our CRC usermode network stack to allow use with VPN. We are working on making this an option for podman machine. Alternatively, or to test, you can use CRC from our crc-org project and run the podman preset. This uses a dedicated VM using native hypervisors and the gvproxy setup.
I am the teamlead of CRC and work on the Windows enablement of podman machine, with the podman desktop team. Gladly take questions here or by email.
I had similar issues with a different VPN/Proxy at an earlier role. I solved with https://github.com/sakai135/wsl-vpnkit and trusting the root certificate of the proxy on the rancher desktop WSL2 vm (Assuming you're on Windows as I was).
Docker desktop pays for itself by solving these issues though IMO (I wasn't able to get a licence at the old role however)
Podman works with Docker Compose enough to run stuff I've had to deal with at work and home. I prefer to use the podman-compose script usually, since it does offer some small advantages when using Podman. That said, even with the podman-compose script, I ran into an issue where some syntax somewhere needed to be adjusted for Podman; I can't remember exactly what and I don't have access to the repository to check, but it was a security-related flag, and it was fixed in master at some point, I believe.
Getting Podman to run CUDA/Nvidia workloads was a bit more challenging, but that can also be done.
Docker Compose works fine with Rancher Desktop. You can use it with Podman on Linux too, you just need to enable the socket since normally Podman does without - I'd imagine there's some way to enable this on Podman desktop too.
For Rancher Desktop, Docker Compose works with Rancher Desktop when you choose dockerd (moby). If you choose to use straight containerd (with nerdctl as a CLI) than compose isn't going to work.
Fully managed PostgreSQL service, with point in time recovery like in Heroku + ability to take manual snapshots if needed. Daily snapshots are not flexible enough.
Essentially, they observed sizeable performance improvements by using UUID generators that are tweaked to get more sequentia resultsl. It results in better indexes. The articles compares sequences, random UUIDs and 2 kinds of sequentialish UUID generators.
There's another benefit to UUID - You can generate them anywhere including application side.
Doing this on application side would have tremendous batching benefits or inserting objects with relationships at the same time (Vs waiting first insert to return an ID to be used in the FK).
Caveat programmer: this could be problematic, not in the sense it doesn't work, but in the sense that someone working on backend code may have a preconceived expectation that UUIDs are also effectively a keyspace i.e. they're hard to guess. The validity of that is already challenged by variants defining temporal or logical order, and evaporates completely if you let clients declare their own that you accept at face value. Applications may have potentially guessable/gameable object identifiers sloshing around inside as a consequence, which is modestly ironic given that one benefit many folks expect from adopting UUIDs in the first place is hardening up the attack surface of trivially enumerable sequences.
There are a few mitigations but my favourite is the "casino chips" approach: pregenerate them server side, and allocate to clients on demand, including en masse if need be ("here kid, have a few million UUIDs to get you started"). Verify with whatever simple signature scheme comes with your application server framework, or at small scale just toss them in a crude LRU store.
Or, remember where the UUID came from, and apply their organisational scope to any trust you place upon it. This might work particularly for multi-tenanted SaaS. However it requires that all usage is tenant-bounded end-through-end throughout your application. This may be in conflict with a) your framework, b) your zenlike contemplation of simplicity in data management, or c) programmers in a hurry forgetting to scope their queries properly.
Ultimately, relying on UUIDs as intrinsically unguessable security tokens is probably not a great idea, but it's one that remains thoroughly embedded in the programming zeitgeist. As ever, nothing useful comes without a compromise. Keep your eyes open to the systemic consequences of design choices, and don't leave traps for your fellow developers.
There definitely are people out there in this thread proposing clients be able provide UUIDs. I’ve seen it elsewhere too.
I’ve also personally experienced UUID collisions due to badly set up VM environments under Windows. It isn’t a good idea to blindly trust any value - and that includes supposedly ‘never collide’ id’s like UUID.
For what it’s worth, I also had the joy of debugging someone’s distributed hash table that was using md5 as the hash bucket key (this was... 2 decades ago?) and had no way to handle collisions because obviously that is impossible.
I getcha, but these days the ambit reach of "application" extends to Javascript executing client-side in an environment that's basically overrun with lions/tigers/bears, and I'll suggest that's particularly a consideration when the front-end is a SPA participating in a CQRS/event-sourced overall application architecture.
Should you ever use a plain token (where you just check if it exists in some authed_users table) vs, I dunno, some sort of signed/HMAC type thing, where you have to call some function on it? I genuinely don't know but I know enough to generally leave authentication up to those that do know.
Maybe I'm just thinking of OAuth where there are multiple hops involved?
The comparison to OAuth is quite reasonable. Perhaps the most obvious parallel is the use of a state parameter during the three-legged exchange, without which it's exposed to a CSRF clickjacking attack.
Right. Maybe it's paranoid, but it seems like a bearer token has potential avenues for forgery (CSRF or others), replay attacks, add-on jacking, etc. Also harder to coordinate with distributed apps. I think the Captain Tightpants approach would be to initialize some client-side private key/cert, use that to sign each request and verify based on that cert.
That should also make it easier to, say, verify and unwrap the request at the gateway to the server, before sending it to the rest of the application-proper.
It's not a question of whether UUIDs can be generated unguessably. They can be, as you point out.
It's whether the UUIDs in your system can be reliably presumed to be unguessable - including the UUIDs that were generated by code which was written after you wrote your query that assumes unguessability.
Today you might say "Oh, this SecureRandom-based UUID generator is unguessable and meets all of our requirements". Tomorrow you might say "Ah, this SecureRandom-based UUID generator is too slow, let's generate them in our Android app instead of on the server". But now the UUIDs stored in your database aren't reliably unguessable, because you accept whatever your client API tells you without verification. How plausible is it, within the timeframes you actually get, to review every query for whether it assumes the trustworthiness of the UUID generation? Better to assume UUIDs have some convenient properties, than to assume that they're unguessable just because the API is cryptographically secure today.
I think you can probably get the same "batching benefits" if you use a global ID generation service of some sort, with sequential IDs to improve indexing. Using sequential IDs doesn't necessitate using auto-generated sequential IDs.
correct, but global service need local caching. Some "HiLo" type identity generation could work. Sequential-UUID simplifies it as you don't need that service, so it's preferable.
HiLo explanation below:
A client simply gets a range of ids to be used, exclusive to them. Then can use it without any roundtrip to server.
This can be achieved with a "hi" stored on service, and a batchSize that's constant in the system.
Each time a "hi" is requested, it's incremented by 1.
Now the client can generate (hi * batchsize, (hi+1) * batchsize - 1).
The Postgres JDBC driver does not guarantee that batch inserts come back in the same order that you insert them (when you use RETURNING *). So, if you generate UUIDs server-side, you can't conveniently match them up with the records you just inserted. You're better off generating them in the app server first and then sending them to Postgres.
Plus, it's way better to generate a UUID in the app server and send it to the server as a string. The reason is that you can deal with the id as a plain string and don't have to deal with a non-standard datatype in your app.
There's nothing that prevents you from fetching a batch of N IDs from the server. Server just does +N on the sequence and you can do whatever you like with the IDs client side.
You can also use one sequence for everything on the server, and then you can also pre-create ID based relationships client side.
You can just use PostgreSQL's writeable CTEs to get the same batching benefits plus the benefits from using serials. So, no, I do not think batching is a good reason for using UUIDs.
They almost got it right, a better implementation would overflow regularly to make use of the entire key space, and counter untuitively more resistant to overflows.
Clocks aren't reliable enough for timestamps anyways so garbage collection is the only thing you kinda wanna rely on them for.
A good sweet spot seems to be, 32bit milliseconds + 96bit of entropy. This overflows appeoximately every 50 days, allowing for 50 day rolling data retention.
Not the worst idea—50 days is a nice sweet spot between infrequent enough to have some indexing benefit and frequent enough that potential downsides will be discovered early in the product’s life cycle.
Personally I wouldn’t do this. A scenario where for each individual millisecond of elapsed time, 96 bits of entropy is an upgrade over 80 bits of entropy, is fairly extreme. I don't think there are many databases in the world which would ever need more collision mitigation than that.
> I don't think there are many databases in the world which would ever need more collision mitigation than that.
Individual instances? Maybe not. But for those an autoincrement key would also work.
That is not the scenario that ULIDs and GUIDs are advertised for.
The goal is to have an universally/globally unique ID. So whenever you encounter two IDs you can be (resonably, probability wise) sure that they won't collide.
Any such sheme thus must, by definition, serve every single use case now and forever everywhere. That's a tough one.
Also it's not really 80bit vs 96bit (which due to the birthday paradox is already a huge difference) but more 80bits vs. 128bit as the timestamp is recycled with sufficient usage.
I'm actually concerned that 96bit isn't enough, as it relies on the assumption that you'll use this scheme for for data spanning years, in order to properly use the timestamp as entropy.
I was debating between using ULID vs just using the same sequential bigint sharding strategy that Instagram uses[1][2].
I ended up deciding to use sharded bigints because it enables better indexing, even though there are drawbacks when first writing the instances; The benefit of faster search was more important for me.
They seem like what I'm looking for. I was looking at moving to something more random because our current UUIDs (uuid1) are too easy to mistake for one another at a glance.
I was hoping someone would mention these. Its really quite nice. 128bit, sortable, client or server generated, no collision (well almost zero: 80bits are random per millisecond)
To alleviate the issue of having a sequential part, they make it wrap around so that you cannot tell the order between two UUIDs. It's already some protection, and the random part is still large.
Did you use OPA as a sidecar or a separate service? I have a similar setup but with a separate service the "diff pushing" approach adds quite some complexity, due to OPA and the data source having separate lifecycles.
Assuming you're deploying to the same OS/architecture that you're working from, otherwise you'd have to rebuild (which Rust makes easy to do, but still)
Though in all those cases, for me, all alternatives are just as bad or worse.
Getting ansible running on a windows server is... unfortunate. Ensuring that your bash script runs on OSX is easier, but I've had lots of unexpected issues with tools like grep or find working just a tiny bit different. And so on.
Trivial. All of the tooling, dependency management, linking, etc. is neatly bundled together under cargo. Just run "cargo build" for the current system, or specify one or more other architectures as arguments to get multiple binaries. Everything's statically-linked so you should be able to scp those wherever.
Still, when we're talking about casual "scripts" even this extra step seems a little bit burdensome
A JVM library in this space I recently started using seriously and fell in love with: jOOQ. It's not an ORM, rather a query builder but an extremely smart one.
In the codegen mode, it scans your DB schema and generates record classes + a lot of utilities. If the DB is well done (and it should be), it interprets many constructs, including relationships, domain types and various constraints. It can also generate activerecord-like classes if needed.
It allows far better safety and composability than raw strings and a lot of control on the query. Most DSL functions are called the same as in standard SQL, and the docs always shows the DSL next to the SQL version.
Everyone in the Java world seems to reach for JPA directly, but for me working with something closer to the DB is really a breath of fresh air. The DB-firat approach really works wonderfully.
Another jOOQ fanboy here - it's not just the manual that shows DSL->SQL, you can just .toString() most of jOOQ constructs in your code and get the raw SQL out. In fact, I believe you can just take that .toString() and put it straight into your prepared statement if you don't want jOOQ to run your queries and just use it for query composition.
Like any toString() implementation, it is meant for debugging. If the object you're calling toString() on is "attachable" (e.g. a Query), then you get the vendor specific string for additional convenience. If you just call substring(a, b, c).toString(), you'll get a generic rendering.
If you always want the vendor specific SQL string, use DSLContext.render(QueryPart)
I would really advise against having anything connected to the Macbook during upgrades, except the charger...