Skip locked seems to be the way that new queueing packages are going, though they don’t solve fairness or multi-tenancy.
It’s nice to keep things simple, but how does that work at scale? If I have 1,000 users I don’t think that these standard queueing systems can prevent one tenant from impacting another. It doesn’t look like that’s possible in SQS, either.
On the other hand, is that even necessary? At what scale is this important?
I'm thinking about relatively simple systems and not "real" fairness. The case where I've seen this being a bit problematic is with such a queue in an application where jobs are sometimes inserted in large bursts. It doesn't actually matter much if they are truly fair, but in the cases where a large number of jobs is inserted at once the result is that for some tenants the queue doesn't seem to move at all.
It's not a huge problem in that case because the regular priority system still puts the urgent jobs at the front of the queue. But it does sometimes cause confusion for someone looking at this from a single-tenant perspective because the queue looks stuck. There are other ways to fix the confusion, but it would be nice to have a very rudimentary fairness in this simple queue that would ensure that the queue iterates a bit over the tenants.
Seems to me a solution is to have a "generic" queue handler that will take any message, and then have tenant specific queue handlers.
Now, mind, I'm not thinking of a dedicated handler for each tenant, rather one that can round robin, or rotate through the tenants to pull items specific to that tenant off the queue. This can be the "fairness" arbiter.
With the Postgres SKIP LOCKED technique, you can be selective as to which messages you pull (you're not limited to the head of the queue, but can dig deep if you want).
So, you have one general purpose "head of the line" handler, and then one (or several) that acts like the fellow at the Post Office "Anyone here just picking up mail?", and pull them out of the queue.
Of course the generic version can just switch modes (every 1m, or 100 messages, go into "fair mode" for a short time), vs having different handlers.
> On the other hand, is that even necessary? At what scale is this important?
I can see it being helpful in front of a shared resource that doesn't have any way to control priority or fairness. My use case would be Elasticsearch. I don't want to block any searches from running if there's capacity, but also I don't want misbehaving clients to monopolise resources, and I want to make sure that all customers/users get to run their searches eventually, though I may prioritise some customers/users over others. And I may need my own internal jobs to run at a lower priority than a user search, but still within a guaranteed window (to avoid timeouts on the client side).
Honestly for my use case this could be useful with just a handful of users - less than 10, even.
> On the other hand, is that even necessary? At what scale is this important?
That's the critical question. To that I had that there might be a myriad of articles on how someone used postgres to build a message queue instead of using a dedicated message broker, but there isn't a single article showing any form of benchmarking, performance tests, load test results, and more importantly report the correlation between load and performance degradation and eventual overload and resulting failure modes. This information is of critical importance to be able to design systems and pick which tool is suited for the job.
For anyone organizing meetups: listen to asciimov on this one! It's so stressful to park in a city (especially if it's not something you do every day and you don't know the right places/garages/open back-streets/etc.)
I'm one of the people helping to run Michigan TypeScript (we meet in person monthly in addition to doing the YouTube channel) and I also ran a React meetup in Detroit in 2016-2018. This article has lots of great stuff in it! I especially liked the parts about knowing your reasons for organizing a meetup.
There is one thing that I was expecting the article to cover, though, that it didn't: finding speakers. From talking to other organizers around the country, this seems like something that's a consistent point of stress. I think with MiTS we've found a good system, since only one month out of last year did we actually have to really "look" for a speaker, which is really great, but also not something that you can always expect to continue.
So on that note, if others have ideas to share on this topic, I'd really love to hear your thoughts.
I touch on this a bit in the post, but here’s about what I do for speakers:
1. Pay close attention to intros channel in the Cleveland area Slack channel and cherry pick the individuals who are doing cool work. We get a lot of boomerang thirty-something remote workers moving back to Cleveland to be closer to family. Giving a talk is a great way for them to reestablish themselves and meet folks in the area.
2. Straight up cold-contact interesting businesses and ask if they have any interest in participating. Surprisingly high success rate on this one, but it does work. Companies like to brag about what makes them special, I just happen to choose the ones that I really do find to be unique.
If those don’t work, I’d rather host a general social with no speaker than force a bad topic or always host repeat speaker appearances.
re: cold-contacting... any pointers on actually finding those businesses? I'm assuming Michigan and Ohio are roughly similar in this case, but man... so many people have left the state it's feeling kinda sparse some days not gonna lie. We used to have a JavaScript meetup in Ann Arbor that could consistently get 80-100 people (or more!), but many many many of those people have simply left. And so have the employers along with them. Not trying to be a downer, haha, but that really does seem to be the situation. We're trying our best to help things go in a more positive direction, but it feels like a chicken-and-egg sometimes. If it weren't for the YouTube channel's successes (which we're very grateful for), it would be a lot more painful.
Anecdotally, people do like getting together with others IRL, but it feels like it's a bit more selective and between YouTube and virtual events, I see people opting out of things that take a non-trivial drive for a short in-person event.
This is significant because although Bitwig has supported Linux since the beginning, recently ableton live started supporting Linux (although only on their own hardware for now: the Push 3)
in my heart I feel where the author is coming from (a place of pain and experience).. however.. without an easy adoption path to TypeScript that the any type provided... we'd probably not have TypeScript today. TypeScript works because you can migrate to it incrementally. That's huge for many people.
creating a new project by just SSH'ing in seems interesting but.. like.. what's the security model for such a thing? Do other platforms do this? ...asking for a friend :)
GitHub and GitLab both do this. They only give you enough of a shell for Git commands. I'm not sure how GitHub's is implemented, but GitLab's used to be a locked down Unix shell (i.e. in theory only the commands needed for Git would be accessible, with the caveat that there could be bugs) and now [0] it is a Go program [1] that listens for SSH connections and implements an in-process shell that only supports Git commands. Fun fact: you can create a GitLab PAT with an SSH command. (Another useful trick, to verify your SSH keys are configured properly for Git, is to run `ssh -T git@github.com` which will print a welcome message including your username.)
The most important thing to know here is authentication and authorization is based on public-key cryptography (via SSH).
Users create an account by SSH'ing into our server (e.g. `ssh new@pgs.sh`), we record the public-key used to remote into it. Then to use the service you either a) run SSH commands (e.g. `ssh {user}@pgs.sh help`) or copy files to our server (e.g. `scp -r ./public/* {user}@pgs.sh:/my-project`).
Under the hood, we use golang's ssh library to forward files being uploaded to our object storage service, record entries into our postgres database, and then have our web service serve those files.
That's actually a really great idea that we hadn't considered, thanks!
Currently the only way to have multiple public keys is to log into the CMS: `ssh {user}@pgs.sh` and then go to "Manage Keys" and then there's an option to add another public key (press "n").
Great question! That's correct. We do help people recover lost accounts when requested which typically involves some proof they owned the account. At the top of the feature backlog we plan on adding recovery codes so a user can access their account when they lose their private key.
But currently, our recommendation is to keep your private key that you use to access pico services somewhere you won't lose.
