Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: What are some tools / libraries you built yourself?
364 points by graderjs on May 16, 2021 | hide | past | favorite | 602 comments
Many times you just want to plug something in. PostgreSQL, Node.JS Express, Java Spring, numpy, Three.js. There's many examples where the already existing solution fits well.

Sometimes that's not good enough tho. What are some tools, libraries or services you built, are they open-source and why weren't you satisfied using what already existed?




Last year I built Recut[0], a native Mac app that helps speed up video editing by automating the rough-cut process. It finds silence, gaps, and pauses and removes them, creating a cut list that you can import into editors like ScreenFlow, Final Cut, Premiere and Resolve. Here’s the Show HN [1] from a couple months ago.

I built it because I was making screencasts and cutting out silence + mistakes was 90% of my editing time. This makes it a ton faster. There were some command-line scripts to do a similar thing but I wanted something visual with a fast feedback loop, where I could quickly preview how it would sound and tweak the parameters in real time.

I didn’t know anything about video and had never built anything with Swift before so it’s been a fun way to learn a bunch of new stuff.

0: https://getrecut.com

1: https://news.ycombinator.com/item?id=26317265


I genuinely wonder what Taran Van Hemert (LTT video editor) would think of this.

It seems a reasonable part of his workflow (https://www.youtube.com/watch?v=O6ERELse_QY) is doing this exact thing.


Good thought! LTT was a big help when I was building a PC for the first time in years. Might see if he has any interest in trying it out.


This solves a huge problem for me, and is easily worth three times the amount I paid for it. Thank you!


Awesome, thanks! Let me know if you have any feedback/feature ideas.


"Want to check this out back at your computer?"

This is clever.


Thanks. Not many have signed up for it but probably a quarter that many have told me this is a cool idea :)


I don't make videos, but I'd recon that someone who makes long videos, doesn't have time to look at the result of all of the cuts.

Maybe a separate time line with just the cuts that you can fast forward through and see/hear if there's anything you still want.


That's an interesting idea! Right now it lets you preview either the original file, or with the silence cut out (and you can quickly toggle between those). I guess this could be a third state, to only play back the cuts.

For my own workflow I usually play back the recording with the silence cut out, and tweak as I go. (The new version I'm working on has the ability to manually cut/keep parts of the timeline). It's way faster than what I used to do, which was play it back while making the cuts as I went. I'm not sure I'd personally want to listen back to only the cuts because it'd be easy to lose my place, but I could see how someone could want to do that.


Genius. You need to find a way to market to the right audience and up your price to $79 or something. The current price seems too low for all the time it saves (if I'm a serious vlogger).

I'll definitely check it out (and maybe purchase it before your price increase, haha).


Hah, I’ve been thinking about exactly both of those things! $79 is the ballpark I was gonna aim for too.

I’m thinking the YouTuber audience might be a good place to start. People seem to “get it” right away, but nobody knows about it yet.


I'm not a vlogger or anything but this is a great product. Nice job!


Thanks!


Starting to get more serious about video editing for my YouTube channel and I'll definitely be giving this a try, most likely purchasing it!


Great! I'd love to hear what you think, whether it fits your workflow or if there are missing pieces that you'd need. Contact info is in my profile if you're open to a chat!


This is awesome - love the landing page too. Great job!


Thanks!


Thats damm good.


In 2013, I started https://webtorrent.io the first torrent client that runs in a web browser. It’s open source here: https://github.com/webtorrent

I built it to make torrent technology more accessible to the masses. We’re still actively building it and we even have a slick desktop app that uses the same engine for streaming.

WebTorrent also powers https://wormhole.app an end-to-end encrypted file sending service that I built with my friend jhiesey.


Wormhole looks awesome!

I was looking for something just like this a few days ago since Firefox Send was shitcanned by Mozilla. I ended up using send-anywhere, but Wormhole looks so much better. I wish it showed up in my Google results when I needed it.

Is Wormhole p2p? If so, why the 10 GB limit? If it’s not p2p, why are you storing user data on servers (even if it’s E2E encrypted)?


According to the explanation on the website, it appears that the 10GB limit only applies if you make the optional choice to also store a copy of the file on Wormhole's servers, to keep the file available after you close your browser.


Where did you find that? The tooltip I saw says they store files up to 5 GB on their servers, but it doesn’t appear to give you a choice. It also says that for files larger than 5 GB, it will use p2p (that answers one of my previous questions, I guess), but it still says “up to 10 GB.”

Why would there be a 10 GB limitation for p2p transfers if data is not being stored remotely?


I'm looking forward to a npm ads follow up. I feel like opinions have changed around money and open source.


Say what now?


I got WebTorrent running on iOS via play.js (Node, rather than Safari, so it supports native sockets etc.). Might be the first non-jailbroken torrent client for iOS :) Very nice project, and works extremely well!


iTorrent[1] works non-jailbroken, however you need to sideload it with something like AltStore [2].

[1]: https://github.com/XITRIX/iTorrent

[2]: https://altstore.io


kudos to you, I love Webtorrent. Thanks for making such great app!


Wormhole is Killer. Love it.


Early 2017, I built Listen Notes [1] , so I can search podcast episodes by keywords and listen to individual podcast episodes without subscribing to the entire podcast. A few years ago, most podcast apps assume you wanted to subscribe to podcasts first then listen. Maybe I’m not a typical podcast listener. I don’t like subscribing to podcasts. I just want to listen to individual episodes then move on. Similar to consuming web contents, I don’t bookmark entire websites. I just search on google, read the web page, then move on :)

[1] https://www.listennotes.com

[edit] This is my Show HN of Listen Notes in early 2017: https://news.ycombinator.com/item?id=13310834


I love listen notes, It has made it easier to search for specific podcast which I had to discover myself. now I just use Listen Notes to search with few keywords and I’m able to find the podcast for my taste.

Anyways love your product and I definitely enjoyed reading your article on medium where you talked about how you ran the entire company as a solo dev and that’s how I discovered ListenNotes back then


Whoa! I'm glad I found this. I think it'll fit into my writing workflow. Thanks!


Most searches are about people's names, e.g., book author's name, a celebrity's name, a CEO's name...

And you can curate playlists of episodes by topic / people, e.g.,

- Scott Galloway https://lnns.co/0LWEK3dhSfy

- Female VCs: https://lnns.co/depiDjM1XvQ

- More playlists: https://www.listennotes.com/podcast-playlists/


I'm a huge fan of Listen Notes!


Thank you :)


I'm curious to know if you thought about building the capability of searching through the audio itself - did you run into transcription costs being a limitation?


I applied for listen notes API request but didn't get a reply back. I hope you get this sorted out. Time is valuable


The captcha experience was awful but the search was perfect.


I created intercooler.js in 2013 so I could do AJAX in HTML:

https://intercoolerjs.org

Last year I removed the jquery dependency and cleaned it up based on a lot of lessons that I learned, renaming it to hmtx:

https://htmx.org

Same idea: extends/complete HTML as a hypertext so you can build more advanced UI within the original hypermedia web model, with a cleaner implementation.

Part of that cleanup involved me pulling out some functionality around events and a proto-scripting language (ic-action), and I enjoy programming languages, so I created a front end scripting language to fill that need:

https://hyperscript.org

It's based on HyperTalk and has a lot of domain specific features for lightweight front end scripting, kind of a jQuery or AlpineJS alternative.


Thank you for intercooler and htmx! It’s a godsend for someone like me that doesn’t want to get too deep in the weeds with frontend dev but still get some dynamic UI actions and lazy loading/click loading done.


looks like you invented HTML-over-the-wire before it was cool :)


in fairness I created intercooler based on pjax and turbolinks, because I wanted to do everything in HTML without magic

and jquery has had load for a long time too: https://api.jquery.com/load/

so I can't claim to be the inventor of the approach

just the perfecter :)


Open source spinoffs from Tarsnap: https://www.tarsnap.com/open-source.html

scrypt: Because the world didn't have any strong password-based key derivation functions.

spiped: Because using stunnel or ssh tunnelling to connect to servers is gross.

kivaloo: Because I wanted a high performance KV store optimized for small values (e.g. 40 bytes) rather than larger "items" (each containing multiple key-value pairs) or "blobs" (e.g. cached chunks of HTML).


Did you also write bsdiff [1] or is that someone with a similar name?

[1] http://www.daemonology.net/bsdiff/


Yes, that was also me. A very long time ago -- I wrote it in 2003 when I was at university and I've barely touched it in the past 15 years.


You don't say:

>bsdiff runs in O((n+m) log n) time; on a 200MHz Pentium Pro, building a binary patch for a 4MB file takes about 90 seconds.


Can you elaborate on what makes stunnel or ssh tunneling "gross"? Is it the network overhead they add or is it the complexity of the protocol, or is it something else?


Protocol complexity, number of vulnerabilities in the codebase, and for ssh there's reliability as well. (SSH tunnels multiplex connections over a single TCP connection, whereas TLS and spiped map one TCP connection to one TCP connection.)


But what's wrong with stunnel?


Protocol complexity and the number of vulnerabilities in the codebase.


That's a rather gratuitous argument that applies to anything that uses TLS or OpenSSL libarary.


I agree that it applies to anything which uses TLS.

I disagree that it's gratuitous.


OpenSSH too?


It’s not a small project...


It's not Wireguard.


Not the author but the reliability of a simple SSH tunnel is horrible. I still use it occasionally because it's "always there".


I made a crappy little web tool[1] that lets me paste a timestamp into it, and will show me that time in my current timezone by default (I think that's just hardcoded haha), or in another one that I select. I mostly use it when reasoning about logs -- for some reason my mind has never gotten used to mapping UTC time to my time.

It also has a selector for the source timezone, but that doesn't work, for reasons I've forgotten. I think that part was intended to allow me to see times without tz data included, reinterpreted as being from other tzs, but I never ended up using it enough to fix.

I made this because I often read some logs, thought "hmm when was this again?", then typed "UTC to EDT" into a new tab, and then ended up on some ad-loaded page that with dropdowns to select things like the year. I don't want dropdowns! I've already got a timestamp. I made this over a year ago and I use it 2-3x/week.

1 - https://wcarss.ca/tz


Yeah, nice benefit of living in the UK is local time is at most one hour out from UTC. Downside is it's easy to code timezone bugs in winter and not notic until summer!


Cool simple tool! :)

I think the reason the source zone might not work (and I'm not an expert, and date times in code are hard), is that all "timestamps" are in Zulu / GMT0 time, so it won't make sense if you try to source a Zulu time from like Hawaii time or whatever.


That was a good guess! Your comment provided some helpful motivation to go look into it. :)

It seems to have actually been a typo-misuse of moment.tz's very similar but very different constructor patterns. To illustrate, imagine we have a date "2020-01-01", an offset tz "America/Vancouver", and "America/Toronto" as the locale's timezone.

moment(date).tz(offset) says "interpret date in its given tz, falling back to the locale tz if not given, then reinterpret it via the offset tz". So the date "2020-01-01" with no encoded tz data is interpreted as midnight in the locale's tz, America/Toronto. That is then reinterpreted in the offset timezone America/Vancouver, so calling .format() on it gives the result "2019-12-31T21:00:00-08:00" -- midnight on New Year's Eve in Toronto was 9 PM on Dec 31st in Vancouver. Offset here is changing the output, not the input.

moment.tz(date, offset) says "interpret date in its given tz, falling back to offset as the tz if not given". So the date "2021-01-01" with no tz data is interpreted as midnight in the offset tz, "America/Vancouver". Now calling .format() gives "2020-01-01T00:00:00-08:00", or midnight in Vancouver, which would be 3 AM in Toronto.

Glad to have it fixed.


Awesome!


i made something similar that lets you map times between arbitrary timezones: http://getwaqt.netlify.app/. thanks for sharing!


I wrote a library for animating image backgrounds on the web using After Effects style filters and transitions:

https://aeroheim.github.io/midori/

It's basically a post-processing pipeline implemented with Three.js and WebGL using a few shaders that I wrote.

I created it primarily because I needed it for one of my current projects, so the use case is fairly niche and mostly only relevant for stuff like creative coding. It was my first dive into computer graphics however, and I learned a TON about computer graphics in general and had a lot of fun writing the library.


This is really cool! I’ve never seen a slideshow look that nice


Thank you!


No, thank you!


Wow, awesome work on this!


Thank you - much appreciated!


This is shockingly good! I want to use this. Amazing work.


Here's most of the stuff that I built myself (and that I still use on a regular basis):

- fae [https://h3rald.com/fae] · a minuscule find and edit utility

- h3 [https://h3.js.org] · an extremely simple javascript microframework

- hastyscribe [https://h3rald.com/hastyscribe] · a professional markdown compiler

- hastysite [https://hastysite.h3rald.com] · a high-performance static site generator

- herald [https://h3rald.com/herald-vim-color-scheme] · a well-balanced vim color scheme

- litestore [https://h3rald.com/litestore] · a minimalist nosql document store

- min [https://min-lang.org] · a small but practical concatenative programming language

- mn [https://h3rald.com/mn] · a truly minimal concatenative programming language

- nifty [https://h3rald.com/nifty] · a tiny (pseudo) package manager and script runner

- nimhttpd [https://h3rald.com/nimhttpd] · a static file web server


Well damn!


I made a macOS script that OCRs text on screen. It is super handy for dealing with everything from scanned documents with account numbers to web pages with text in images.

When I press F16, I get an image capture cursor, and the area selected is turned into text.

Example:

https://files.littlebird.com.au/Screen-Recording-2021-05-17-...


Edit: I'll clean the script up and make it available as a "Show HN:" soon.


As promised, the Show HN is available here: https://news.ycombinator.com/item?id=27227925


Thank you!



You have a hit on your hands. Let me (@NetOpWibby) know if you need a website built.


Please share if possible, this is very useful


Not the original poster, but this (https://news.ycombinator.com/item?id=27177609) how I do the same thing


Can you share it? That sounds pretty useful!


Not op, but I wrote something equivalent based on a linux script that I found [1]. Here's the mac version of the script [2]. You need tesseract [3] installed for the OCR. For launching it using a keyboard shortcut I use a hammerspoon [4]. Example of how I bind it [5]

[1] https://github.com/sdushantha/bin/blob/master/utils/ocr

[2] https://github.com/JJGO/dotfiles/blob/master/scripts/bin/ocr

[3] https://pypi.org/project/pytesseract/

[4] http://www.hammerspoon.org/

[5] https://github.com/JJGO/dotfiles/blob/master/mac/.hammerspoo...


Nice, I started with tesseract, but moved to Google Vision as the quality was better.

Azure's offering was better for hand written notes.


Interesting, that doesn't really work for me as I sometimes use it to OCR sensitive stuff from scanned documents, and it wouldn't work offline. I found tesseract to work pretty reliably for my use and only fail when scanning non-ASCII characters.


Yeah, I only use it for things where the context wouldn't matter. For example grabbing an account number from a document.

I've been experimenting w/ CoreImage detectors (mainly for categorisation of scanned documents based on QR codes).

When I clean things up today, I'll see what can be used.


Last year I built a tool for visualizing log file volume (based on time) in your terminal [1]. Similar tools exist, but none were fast enough or easy to set up in a hurry, e.g. when you're investigating a production outage.

The year before that, a tool for visualizing ping latency as a heatmap [2]. My laptop's wifi had developed a severe latency stutter every ~500ms that was driving me nuts when using SSH and other interactive tools, not to mention killing my throughput. Once I could visualize it and saw the pattern, it was very clear that there was a system-level issue. Eventually traced it to a virtualization product's network driver.

It's definitely a pattern for me -- feel frustrated with diagnostic and data viz tools that are either too slow to handle or too complex to configure when I need them, and try to build simple tools that solve exactly the problems I keep bumping into. These smaller projects also provide good opportunities to practice with new languages or frameworks.

[1] https://github.com/acj/krapslog-rs (also krapslog-go)

[2] https://github.com/acj/pingrok


I've made a few things:

- Zola (https://github.com/getzola/zola) a SSG that works like I wanted and had a template engine close to Jinja2. I was using Hugo at the time but can't stand Golang template engine (imagine my sadness recently when I realised what Helm Charts are using...)

- Tera (https://github.com/keats/tera): a template engine pretty close to Jinja2/Django templates - at the time it was made only Handlebars existed in Rust and it was not enough for what I needed in Zola

- kickstart (https://github.com/Keats/kickstart): an equivalent of cookiecutter from Python but trying to be more interactive/powerful as a simple binary, see the GIF in the repo for an example

- validator (https://github.com/Keats/validator): my take on Python marshmallow validations, nothing existed at the time.

So yeah, most of my open-source is re-creating libraries I use with Python in Rust.


The last software team I was on used Zola for a lot of our internal documentation publishing. It worked great and we were really happy with it!


OwnTime.jl - A Julia profiling library [1].

It reports the time spent on each line of your project, and you can filter which files are included. For example, you might do a complicated map operation which calls a function in another library, if you filter out the library then it will report that all the time was spent in your code on the line that performed the map. However, if you don't filter out the library then it will report that all the time was spent on the lines inside the library.

I was tired of looking at profiling reports filled with lines of code I had never seen before.

I couldn't figure out how to get what I wanted from the existing profiling libraries so wrote this. It was quite easy once I figured it out. Julia has a built in sampling profiler you can turn on, and from there you can process the sampled stackframes however you'd like and display them however you'd like.

[1]: https://github.com/DevJac/OwnTime.jl


I gave it a try with a couple of calls of a framework which I have written and I suspect is littered with bad practices.

Not sure what to make of the results, since most of the time seems to be spent here: [1] 79% => poptask(W::Base.InvasiveLinkedListSynchronized{Task}) at task.jl:760

Is this just something internal to Atom to execute code?


Does task.jl contain code you wrote? Feel free to make an issue and we can talk about it on GitHib too.

Try setting a stackframe_filter, filter on the root path of your project and then only lines in your repository will be reported.


A Python program to make an embeddable search engine for a static website: https://github.com/joe-crawford/Static-Site-Search

It indexes the site and generates search functionality in JavaScript (the index is just a static JSON file) that can be uploaded wherever JS/HTML/etc. files can be uploaded. I thought it might be useful for people with sites on GitHub pages or similar who don't want to use one of the major search engine company widgets.

Also my back of the envelope calculations were that the simple search index for a small–medium site won't be that big.


That’s pretty clever, I guess that for most personal blogs it will be more that sufficient.


Cheers, yes that was the idea!

I looked for a while to find something similar. I'm sure it must already be implemented out there, but to answer the question in the original post about why build something, I couldn't find an easily downloadable "tool" for doing it when I looked (2019) and it was simple enough to be "fun".


Dbmate: https://github.com/amacneil/dbmate

I found most database migration tools tried to be too smart (I'm not a fan of automatic migrations), were too tightly integrated with specific languages or frameworks, or didn't support basic developer workflow pleasantries such as dropping and recreating the database.

I created a simple single binary migration tool that runs migrations (sql files), and keeps track of applied migrations in a table (inspired by Active Record migrations). Dbmate works with any language or framework, and generally tries to be useful while staying out of your way.

Comparison against other tools here: https://github.com/amacneil/dbmate#alternatives


I found that I wanted something really specific, and built a tool in Go that lets you write, modify and use plpgsql functions as if they were regular source code files rather than database objects. You can write them as Go packages, import those packages into other Go projects, and write tests - without maintaining migration files for anything other than the tables. Combined with IDEA’s plpgsql language support it’s made me super productive.

I’ve used it very successfully in production for a few years and have been curious if anyone else would find it useful enough for me to open it up.


can you give more details, i am generating golang and js functions corresponding to plpgsql functions in somewhat hacky way. I am curious to know what approach have you taken, is it something similar to this project https://github.com/gchaincl/dotsql


hey sorry i didn't see this earlier. it's not a code generator, as such, it's really just a structured way to write PLPGSQL functions alongside Go code, and have it deployed and upgraded automagically.

It would be great to generate Go stubs from the SQL definitions and I don't do that. I'd like something a bit like dotsql but ideally where it creates actual Go funcs out of the SQL function signatures.

The bit that I've done so far lets you package up SQL functions into a schema and treat them as a Go package. you can include them in go.mod and the functions are maintained without writing migration scripts.

I'm under the pump at the moment but I'll try to write something up about it and maybe ping you on github some time. Sorry i don't have much time at the moment!


I definitely want something like that.


Curious about the lack of comparison to https://flywaydb.org/ , which I've seen a ton of support for in the past and people seem to really enjoy


Flyway seems like a startup offering a hosted service, so it might be a strange comparison. But if you are familiar with it and care to submit a PR, I'd accept it!


I love Dbmate! Such a great library!


Looks similar to Sqitch (https://sqitch.org).


Used Dbmate in my pet-project. Absolutely love it! Thank you!


dbmate user here. used it for sqlite migrations and it has been a joy to work with!


https://webhook.site - for testing webhooks, evolved into a SaaS for building workflows from http requests and emails

https://instadj.com - simple app for making youtube playlists, wanted a simple app for playing music at parties (example: https://instadj.com/wam)

Both are open source.


Instadj looks great!

Tangent, but: I was dabbling with a YouTube webapp a while back, and ran into a problem where some of the music videos had a license that didn’t allow embedding - which was obviously necessity to play within my page. (Also at the time the api didn’t succeed in filtering out such videos within a search - tho I’ve not checked back to see if this is now fixed.)

Did you encounter this, and circumvent or fix it?


I'm not sure about that, I can't remember having experienced it. I also think it's dependent on the region.


Oh wow, you made webhook.site??! Thanks so much for it, it saved me just last week when I was beginning to worry about one part of a data pipeline that I'm responsible for :)


Thank you for webhook.site!

It's phenomenal, just works and doesn't require I jump through signup just to get started. Definitely the best webhook test tool I've seen.


Glad to hear it. One of the goals was to make it as easy as possible to get started with.


I built an open source library/mini-framework to seamlessly integrate a Svelte (JS) frontend with a Crystal backend. It's really fun to build apps with this!

https://nlh.me/projects/celestite

Still very much a WIP, but it works. You can build your front-end in pure Svelte components and it supports server-side rendering, client-side hydration, and all that good stuff.

I did this entirely to scratch my own itch - I love working in Crystal (it's a beautiful, elegant language) and I love working in Svelte (modular, reactive web components) and it all just fits together nicely.

Would love feedback / contributions!


Cool! I love working in both Crystal and Svelte. I'll definitely check it out.


I find Crystal fascinating but I haven't been able to internalize non-web languages just yet.


At this point I've made a habit out of building homebrew tools and languages. Very few of these are purely because I was dissatisfied with off-the-shelf solutions; many of these just exist because I thought it would be fun/educational/challenging to build an X for myself from scratch.

I've made

- A dynamic programming language, Ink (https://dotink.co), which runs in "production" (for whatever that means for side projects) for around a dozen projects written in it.

- A compiler to compile that to JavaScript (https://github.com/thesephist/september)

- A bunch of language tooling around that language, like syntax highlighters, editor plugins, code formatters (for example, the code formatter https://github.com/thesephist/inkfmt)

- A small UI library (https://github.com/thesephist/torus)

- A suite of productivity tools (https://thesephist.com/posts/tools/) like notes, todos, shared whiteboard, contacts/CRM

- Twitter client (https://github.com/thesephist/lucerne/)

- Theres a few dozen more at (https://thesephist.com/projects/) :)

Many of these end up building on top of each other, so across the few dozen projects built on top of these tools they form a nice dependency graph -> https://twitter.com/thesephist/status/1367675987354251265


I created a dotfiles management tool, yadm https://yadm.io

I'm a huge fan of Git, and the Git mental model just makes sense to me. I wanted to manage my dotfiles in the same way. At the time I had tried several other solutions, mostly similar tools like homeschick and vcsh.

I wanted a solution that was very portable, which both of those existing tools were. However, homeschick became a bit clunky to use because each time I wanted to do most operations I had to first "cd" into my "castle". In addition, that tool used symlinks, which doesn't work out well to manage a file that may get overwritten by a tool.

vcsh was the other main tool I tried. This tool adopts the bare repo technique, allowing files to be used instead of symlinks. The major drawback it had was that it was designed to work with multiple repos, and forces you to specify the repo with every command. Some simple aliasing helped mitigate that though. But by that time I also started writing some custom code to encrypt private data and was struggling to maintain different branches for different systems which had slight tweaks to configurations. It just became all fragmented and difficult to manage.

That is what lead me to create my own tool. The important features I wanted: portability, seamless interaction with Git, alternate files for different hosts, encryption for private data. I shared it as open source back in 2015 and it has grown in popularity some. It's been improved a lot over time including contributions from others. I'm glad so many others have found it useful.


This looks cool. I currently have a nest of dotfiles I manage with Git, and since I try to use them across Ubuntu, OpenBSD and macOS, the platform-specific stuff seems useful.

But is there a way to "promote" my existing Git repo to something yadm can use? If I just `cd ~; yadm status` I'm told a Git repo doesn't exist, when it most certainly does (`git status` behaves as expected). Or is the best way to just blow away my existing .git directory and start over using only yadm? I skimmed your docs for an answer to this but couldn't find it.


The best way would be to use “yadm clone” to clone the remote repo.

yadm will keep its repo in ~/.local/share/yadm/repo.git.

Once cloned, you do not need to be in any specific directory to run yadm.


I have been using this for few years. It's great and I think it fits well how developers want to configure and backup overall.


Not sure if it counts, but I made two Roslyn analyzers for C# code. A Roslyn analyzer is basically a C# compiler plugin which can emit additional diagnostics if it encounters a certain pattern in the code being compiled.

One of them lets a developer mark a C# method such that the method must be invoked with named args [1], calling it with positional args is a compile time error then.

And the other makes it a compile time error to discard a marked method's return value [2]. Kind of like it is in functional languages.

Was learning/practicing F# along the way, so the code is probably not so good. But it works :)

[1]: https://github.com/mykolav/require-named-args-fs

[2]: https://github.com/mykolav/must-use-ret-val-fs


#1 is awesome! I love the new record syntax for DTOs but I also like using named properties so that the ordering isn’t messed up as they are expanded (but also ensures that every property is filled out, unlike the class initializer syntax). This will be great for enforcing it!


I made a library called Restruct. https://github.com/go-restruct/restruct

It is useful for parsing and emitting binary structures. It uses Go struct tags to allow you to define fairly advanced arbitrary structures. Unlike some similar libraries, it supports both reading and writing data. I used it to write a quick program that manipulates FL Studio project files, and another that extracts PNGs out of a blob (like binwalk but more accurate for this specific task.)

https://github.com/jchv/pngextract

https://github.com/jchv/flsplit

If you’ve never seen Kaitai, it might take a while for it to click what you can do with it. I was actually inspired by 010 Editor’s binary templates, but found out about Kaitai Struct after creating Restruct. I stole their idea of having arbitrary expressions and wrote my own expression engine for Restruct, and also ended up contributing improvements to Kaitai’s Go compiler too, for good measure. The bottom line is, these tools make it extremely easy to work with complex binary files. I can incrementally sketch out an unknown file format, like FL Studio FLP, and quickly test my assumptions about structure shape. Kaitai also has their IDE tool, which lets you use Kaitai interactively:

https://ide.kaitai.io/

I actually hope to do the same for Restruct, but want to re-engineer some aspects of it to make it work better in an interactive context.


As a researcher in machine learning, I wanted to explore applications of Deepmind’s AlphaZero algorithm beyond board games (such as in automated theorem proving or chemical synthesis).

However, I noticed that existing open-source implementations of AlphaZero mostly consisted in complex C++ codebases that are highly specialized for specific games (eg. Leela Zero and LC0). Accessible Python implementations could be found but they were usually too slow to do anything useful on limited computing power.

Seeing this, I built AlphaZero.jl: https://github.com/jonathan-laurent/AlphaZero.jl

AlphaZero.jl is written in Julia and it is consistently one to two orders of magnitude faster than competing Python alternatives, while being equally simple and flexible. I just released a new version a few days ago with many new features (support for distributed computing, support for arbitrary MDPs...).

If you are a student, a researcher or a hacker curious about AlphaZero, please consider having a look!


I built https://www.dreamlist.com for my family. I had to make it simple, so my grandparents can use it; and private + completely ads-free, so it’s safe for our kids. It started as a collaborative shopping cart and wish list, but I’m growing it into a shared searchable brain bridge across generations. Grandparents can share and collect stories and memories, kids can share and collect things they want to do next, parents can turn wishes into more memories, etc. I love building it (Golang, JS) and every day I get some new delightful piece of feedback from a user that has done something fun with their DreamList (often use cases I didn’t know existed). It feels like a growing a child that makes you proud.


Sounds very interesting. I am particularly fond of anyone trying to reach less technologically inclined users (been working in that space for a couple of years now. You quickly learn that software devs do not reason about websites / apps and their UX in any way like a software dev would).

Do you have a pricing page anywhere? I am finding it hard to figure out where you make this project sustainable financially :)


As a system administrator, I’m frustrated by two things :

* The existing backup tools, that are all hyper-specialized, all configured differently, all have slightly different feature sets, and so on. You will likely have very different tools to backup, say, your /etc folder and your mariadb database

* cron is cool, but its logging/alerts capabilities are… subpar, to say the least

So I’ve developed tools to alleviate my pain in those two areas.

I’ve just open-sourced the first one : https://github.com/sloonz/uback/. Feedback is welcome ! I’ll probably make a Show HN post about it when it hit 0.5.

For the second one, I’ve made some tools to monitor cron jobs. Not released yet, but open-sourcing it is on my TODO-list too. It will need way more polishing for it tho.

On another shameless plug (but it’s the topic of this post, so let’s be shameless indeed), I needed something very similar to vouch-proxy but that could protect multiple unrelated domains and be configurable with a database (because I’m not the one who is going to configure the ACL, and the guy who is going to configure the ACL is perfectly able to use phpmyadimn). Since the guys behind vouch-proxy were not willing to accept the changes necessary to make this work, I decided to create my own alternative : https://github.com/sloonz/ngx-auth


Nice, I hear you on the risks of depending on 3rd-party libraries where you can't get your changes merged. It's nice to be able to roll your own in that case.

On the cron case, what are your thoughts on the email notifications? In what ways are the useful or useless? I'm not an expert in this stuff.

And I really like the look of uback. I have the kind of sense (and pride) that maybe this thread I posted is going to be the place where some people (who were maybe previously laboring on something in obscurity) first posted it to the world, or mentioned about it. Or maybe were even encouraged by this Ask post, or seeing other people post work, to post their own work. I'd love it if that were the case, if this post of mine was something that contributed to people posting their work that went on to become something good. :) Not trying to give you pressure, but uback could be that. :) It's pretty fresh


> On the cron case, what are your thoughts on the email notifications? In what ways are the useful or useless? I'm not an expert in this stuff.

I really dislike it. It’s probably the biggest pain point in cron IMO.

1. At the age of DKIM, DMARC and SPF, setting up a working MSA on every server that runs cron can be a real pain. Not always, it depends on the exact situation, but it can be.

2. The biggest issue is that it sends a mail whenever the cron job outputs anything, even if it is successful (exit code 0). This is batshit insane.

3. Nowadays, mail only notification are a bit limitating anyway.


For 2, see chronic[0] (part of moreutils[1]). It runs a command, muting stdout and stderr unless the command exits with a non-zero status.

[0]: https://manpages.debian.org/jessie/moreutils/chronic.1.en.ht...

[1]: https://joeyh.name/code/moreutils/


Yeah, I suppose you could send the mail to an intermediary and filter it. But I agree with you. I have a 5 minute cron on my server, and when I login to root. "You have 4487 messages". Thanks, cron! :P :) xx


I've built a few over the years:

- https://volt.fm - See, share and compare your Spotify stats

- https://postsheet.com - Send emails to contacts in a Google Sheets or Airtable document

- https://pikaso.me - Screenshots Twitter for sharing on Instagram

- https://github.com/soheilpro/zsh-vi-search - Adds support for searching the current line (in normal vi mode) to zsh

- https://github.com/soheilpro/catj - A better way to display JSON files

- https://github.com/soheilpro/mann - Never forget command line options again

- https://github.com/soheilpro/pgcmd - Non-interactive PostgreSQL query tool

- https://github.com/soheilpro/sqlmon - Collect events from SQL Server and save them to Elasticsearch

- https://github.com/soheilpro/sqltop - Find the most resource consuming SQL Server queries

- https://github.com/soheilpro/mon - Painless performance monitoring for Windows

- https://github.com/soheilpro/Ledger - Interactive CLI double-entry accounting


Cool body of work!!! :)


I made a CLI interface for the Microsoft Academic API so you can search for papers, download them, and generate HTML from a collection: https://github.com/mila-iqia/paperoni

I'm using it to automatically collect papers published by members of the research institute I work for, it's working pretty well.

I also made a small utility to sync folders on-demand on my different machines: https://github.com/breuleux/synecure

Most of the work is done by a library called bsync that someone else had written, which itself uses rsync, but I'm quite happy about the interface I made. By default it syncs home-to-home, so you don't have to specify both the source and destination directories. You can just type "sy -r remote" in ~/whatever to sync it to ~/whatever on the remote.


paperoni is awesome!

I once implemented the crossref api in rust in order to search for publications from the command line https://github.com/mattsse/crossref-rs


Thanks :)

How complete is crossref? I can't find my own publications on the search page they have on their site (well, I found one), but the search engine is garbage so I don't know whether it's because they're not in the database or if it's just an issue with the search. I assume it works better with the API.

Microsoft Academic is quite complete from my experience, I haven't really managed to find any paper that wasn't in their database unless they're very recent. Only downside is the need to get an API key (but they're free up to 10K queries/month or so, which is plenty).


make-portable.sh.

Makes any macOS .dylib or executable portable.

It (recursively) makes local copies of all the dynamic libraries on which you depend, and rewrites all load commands to prefer the local copy.

This lets you safely distribute your software to another computer (rather than discovering it only works on your computer because of your brew installs).

https://gist.github.com/Birch-san/e84cfa3b93ffa104af2bd9a047...

Explanation, diagrams: https://birchlabs.co.uk/blog/alex/juicysfplugin/synth/cpp/20...


I created Lua with C/C++/Java/Javascript syntax https://github.com/mingodad/ljs

Also forked a scripting language https://github.com/mingodad/squilu and extended it to accept a pseudo C++/Java/CSharp/Typescript syntax to make easy to reuse code and have fast iteration when creating new functions (with it and minor changes I could get https://ssw.jku.at/Research/Projects/Coco/ parser to work for easy grammar development).

Create a tool for create, document and prototype database applications at "user level" in near realtime (live) https://github.com/mingodad/db-api-server.

Extended GLPK to accept a bigger subset of AMPL and made some performance improvements in the GMPL interpreter https://github.com/mingodad/GLPK


Here I'm showing something I did some time ago https://meimporta.eu/htmleditor/ it's a crude WYSIWYG html editor for people that already knows HTML/CSS and uses basically the browser contenteditable capabilities. It was inpired by Webflow (https://webflow.com/) but very simple and basic, it has 4 main sections: editor shortcuts, wysiwyg editor, html editor, tree view. Some things are easy done in the wysiwyg editor but sometimes the raw html editor is better, we can edit in both of then and get instant result.

Don't bother comment on the UI/UX it's an personal naive tool straight to the point I needed. Anyway any feedback to improve is welcome.


What in the world does with C/C++/Java/Javascript syntax mean?

Four different languages with four different syntaxes.


An attempt to allow code reuse for a subset of all of then.


You should word it differently because I never would have guessed that is what it means.


I have built a few tools for developers:

- Mockit - A tool to quickly create mocked APIs.

- awsicons.dev - Quickly find AWS icons

- EventBridge Atlas - Discover and document your AWS EventBridge schemas

I'm currently building a tool that will hopefully help all open source engineers in the world.

I want to give open source engineers the ability to quickly create landing pages for their GitHub projects.

I think README.md files can only go so far, but (for me anyway) nothing beats a nice landing page selling your open source project.

Not all of us have the time or skills needed to create landing pages so I built https://gitpages.app that can help everyone.

https://gitpages.app is a tool that hopefully can inspire and help open source engineers. It's still in development but if people are interested let me know.

Cheers!


awesome project And I definitely love the idea of helping open source developers with the landing pages

Me as an opensource developer always have a hard time building a landing page as I suck at design


Thanks!

Yeah thats the whole point, just to help people out! I enjoy doing all sorts and writing tools so anything I can do to help people.

If your interested in the project, sign up for the early access, I give weekly updates on progress and hopefully soon give people links to give the tool ago for themself!

I just this second created another Youtube video with a weekly update if your interested:

https://youtu.be/rimFEf5grGk


No problem Just signed up looking to forward to it


We’ve created https://lowdefy.com and open-sourced it.

Over the past 5 years we’ve been building custom B2B and back office web applications. Nothing in the market had the level of customization we required both from a technical and commercial angle. So decided to standardize how we “configure” apps into a platform.

We did this by designing a schema by which to define web apps which is easy to read, write and understand, and works well with normal developer tools like editors, git and CI. While keeping it self hosted and even server-less.

Lowdefy has enabled us to rapidly deploy custom business apps like BI, CRM to MRPs.

Check it out :) https://github.com/lowdefy/lowdefy


This looks really cool! Just added a reminder to check out in more detail when I get to a computer :)

Thank you for sharing.


Thanks! We are having great fun building it and making it useful for others!

Please reach out if you have any questions: https://discord.gg/WmcJgXt


I looked over your page, briefly through docs, couldn't find: what kind of code are the finishes webapps pushed out as?

Are exported projects cloud host agnostic?


Essentially we've written an engine which renders the YAML / JSON as a React app. So the backend serves the json config to the frontend which renders and interprets the app at runtime. There are 3 parts to the front-end, operators - which executes page logic, engine - which evaluates all page blocks, and renderer - which renders the components using react. The logic executes in a recursive loop, so if you look at the renderer [0] or operators [1] code it's actually quite simple. The engine part is a little more complex, but thats written in pure js, so it's easy to test.

All of this talks to a stateless node backend [3] which talks external connections or data sources.

> Are exported projects cloud host agnostic?

Yes. Open-source and self-hosted. We run apps on Netlify (with functions), Serverless, Docker etc. We'll probably include a static build version which will allow to host a Lowdefy app as a static site. The docs is also just a Lowdefy app, so we can easily make it interactive [4] and autogenerate much it from a JSONSchema.

[0] - https://github.com/lowdefy/lowdefy/tree/main/packages/render...

[1] - https://github.com/lowdefy/lowdefy/tree/main/packages/operat...

[2] - https://github.com/lowdefy/lowdefy/tree/main/packages/engine

[3] - https://github.com/lowdefy/lowdefy/tree/main/packages/graphq...

[4] - https://docs.lowdefy.com/ButtonSelector


I built a "Pokemon for Sushi", where I kept track and collected all the different sushi that I ate, and where I found it. It connects to the Foursquare API, so I can just pick the sushi shop from a drop down through my phone

https://torodex.herokuapp.com/netas


I love the idea! Sushis are indeed very "pokemon-like" they all have very different characteristics and very cute looks. Have you thought about gamifying it somehow? Also I hope there's some kind of stats and lore on each sushi, so going on this page would be like exploring a pokedex.


I have thought about gamification, but I wasn't sure how you could verify the picture of the fish people posted was the actual fish.

At best, people would play against themselves, just a collections game.

Do you mean a made-up fantasy lore? Or actual background on the fish? What kind of stats?


I'm not a big sushi person, but stats could be stuff like freshness, juiciness, visual attractiveness, price, etc. Doesn't have to be that accurate but I think it'll add some appeal to the Torodex.

Lore could be just a description, if there's any history to this specific dish, what it's made of and how it's made, etc. and yeah could add some jokes / fantasy lore just to spice it up.

Mainly I'm hoping these sushis are all like manga characters and each have their own personality and stories (it's always exciting to read those manga data books), just my 2 cents!


Yeah, I know what you're going for. Kinda like there's a "spirit" character that thematically represents the fish.


Created TAO[0] -- a minimal syntax that's even simpler than S-expressions.

Building notations on top of it, currently focusing on one for data -- an alternative to XML, JSON et al.

Open-source, prototyping on GitHub[1].

Keepin' it simple ain't easy.

[0] https://tree-annotation.org/

[1] https://github.com/tree-annotation


Is this not too little syntax, though? The two JSON documents

  { "foo": true }
  { "foo": "true" }
would both have to map to the same TAO representation:

  foo [true]


Yes, Data TAO as shown on the website maps to a subset of JSON with only one primitive type (string).

However this is by design.

The idea is to provide only the most generic constructs, decouple specialized data types from the core syntax and enforce/interpret them on a separate layer.

An ad hoc (but perfectly sensible IMO) way to apply this idea to the example is: if foo is supposed to be a string then no need to do anything (that's the default primitive). If it's supposed to be a boolean, then parse it as such (e.g. 'true' -> true, 'false' -> false, error otherwise; or even error if typeof JSON.parse(value) !== 'boolean' in JS).

A non-ad hoc data type layer for TAO is something I've thought about a lot and it's still baking.

Two basic (non-exclusive) ways to do this:

1. Inline/mixed with the data -- type annotations, op-based literals

2. Separated from the data -- schema-based

Example designs for the 1st point:

    foo [`t]
    foo`: boolean [true]
    foo`: json [true]
All of these would associate the boolean `true` with `foo`.


It looks very neat. I quite like the example where tapping JSON code shows, which characters would be removed in TAO.


Glad to hear that, tried my best to present it in a clear and concise way. Quite the task in itself.


Is your project related to the tree notation project or are you the same guy?


If you mean https://treenotation.org/ then it's completely unrelated.

I saw this recently on the front page. Was pretty confused myself. Domain name is very similar and what the project is about seems to overlap, but ultimately it goes in a different direction.

TAO is my independent original project, but it makes sense that there is > 1 person making an effort to figure this stuff out.


Wearable computer (on-body Linux) with audial calendar reminders and "normal" hand-strapped "desktop" https://github.com/andrey-utkin/wearable-computer/wiki

TODO list and personal project tracker - like TaskWarrior but SQL-accessible https://github.com/andrey-utkin/taskdb/wiki

sqlhub.net - Access to public datasets in form via PostgreSQL server https://github.com/sqlhub-net/sqlhub-net/wiki


you mention "text message to miband 3" - did you, by chance, make MiBand hackable?


I used an existing open source python codebase. See https://github.com/andrey-utkin/MiBand3


I struggled to find a job board that was not filled with shabby recruitment agencies, that had clear high quality job postings and clear salary ranges in each job opening. I therefore created the job board[0] and it now includes a directory for Go developers[2], Companies using Go[3] and a salary trends section[4]. It's all been written in pure Go, basic HTML/JS/CSS and PostgreSQL and it's open source[1] :)

- [0] https://golang.cafe

- [1] https://github.com/golang-cafe/golang.cafe

- [2] https://golang.cafe/Golang-Developers

- [3] https://golang.cafe/Companies-Using-Golang

- [4] https://golang.cafe/Golang-Developer-Salary-Remote


How do you source the job postings for this site?

PS: I'm one of the moderators for fossjobs.net, a site for job postings about working on open source projects.


https://www.checkbot.io/

> Checkbot is a Chrome extension that tests 100s of pages at a time to find critical SEO, speed and security problems before your users do. Test unlimited sites as often as you want including local development sites to find and eliminate broken links, duplicate content, invalid HTML/CSS/JavaScript, insecure pages, redirect chains and 50+ other common website problems.

I build this to automate audits I kept having to do manually while doing frontend work. I also liked the idea of it helping to teach people about web best practices, so tried to make it easy to use with minimal jargon explanations (see https://www.checkbot.io/guide/).

It written with Vue + TypeScript + Firebase + Paddle.

There's a free tier with no sign-up required - give it a quick try on one of your own pages and it'll usually find something to fix you didn't know about.


I automate a lot of AutoCAD with Lisp, C++ and Python.

The two biggest things Ive built for with this tech is a multithreaded scripting engine using Qt, Lisp and acoreconsole, and a search engine for Autocad files that searches our entire CAD library for strings, etc.

Between these two applications, Ive saved my last company thousands and thousands of man hours.

I left because they didn't care.


I've written jql[0], an alternative to jq with much less syntax (being lispy) and much easier to use for me. The project may seem dead to some, but I'm using it daily, it's just finished.

[0]:https://github.com/cube2222/jql


s-expressions to tame json - quite my humour! And an eternal binary, awesome!


I have 112 mostly-maintained released project listed on https://github.com/simonw/simonw/blob/main/releases.md now - most of which are tools for loading data into SQLite database files (a surprisingly under-served use-case given how ubiquitous SQLite is) and plugins for my https://datasette.io/ project for reading data back out of SQLite again.

I realized a few years ago that SQLite was the perfect tool for doing data analysis at the small-data scale, where small data is less that 10GB which is pretty much everything you might want to analyze - especially for personal projects.

So I've been building tools to fill that niche!


I made https://CampAlert.live , a notification service for last minute camp site cancellations. It sends you an email and text message when we find a free spot of your choosing. Currently, it supports only Canadian campsites in BC, AB and National Parks.

I've looked into previous tools but their services were all too expensive. They also didn't have the granularity I wanted in the options you could choose for campsites. We tried expanding to the USA last year but since we made it a paid service, it's been hard to advertise and get into the market. When we were free for the first month, we got multiple local radio interviews and news stories which really helped bring that critical mass of users in.


would be great if you had parks in Washington state.


We did have USA up last year but nobody from the US used our site. The time upkeeping that code and managing it wasn't worth it unfortunately


This one is hardware and software.

I made webUSB postage scales and a webUSB thermal printer.

We used to use Dymo label printers etc, but macOS' printer queue would get stuffed up every 200 or so prints.

The advantage of a webUSB based solution was:

  - No OS based printer queue to get gummed up
  - Super-fast (just ZPL printer commands being sent down the wire)
  - One-click to create a postage label
  - Automatic label printing when a new order comes in (Websocket + WebUSB)
  - Live parcel weights and prices updated via Javascript
We used to create labels by copying and pasting between Shopify and Australia Post. This hardware saves us ~4min per parcel.

[1] Demo: https://vimeo.com/334547755/c387957a25


I built (or at least "designed most of, built parts of") Hyperscan: https://github.com/intel/hyperscan - a fast multiple regular expression matcher.

It was not originally open sourced; it was a commercial product for some time.

I wasn't satisfied with what already existed at the time (and neither were our customers) because most regular expression libraries are not focused on performance (using backtracking), streaming (again, backtracking) across multiple inputs - e.g. packets - without holding on to old data - and handling lots of regular expressions at scale.

re2 came along while we were still doing a commercial project but had zero impact on our sales, as it didn't handle streaming (weirdly, imo, as it could have) but it also didn't scale very well to large numbers of patterns.

I also designed simdjson (https://github.com/simdjson/simdjson), although others (chiefly Daniel Lemire and John Keiser) now do all the real work. It's still my design under the hood (although the "On Demand" stuff is significantly new and different). I built the first prototypes of simdjson because Daniel trolled me with a paper from Microsoft and I was bored.


https://github.com/GuB-42/dupfiletree

A tool to find identical files.

Compared to other tools that do similar things, it can also find identical directories, handles archives and is designed to work with millions of files efficiently.

It works is two steps: first make a list of files and checksums with xmd5 and analyze it with dupfiletree.

Made for personal use, not documented, designed for performance first, stable enough for me but use at your own risk.


I'm building a general-purpose data format for the modern age. The old ones are too bulky, too insecure, and too limiting.

* Secure: As a tightly specified format, Concise Encoding doesn't suffer from the security problems that the more loosely defined formats do. Everything is done one way only, leaving less of an attack surface.

* Efficient: As a twin binary/text format, Concise Encoding retains the text-based ease-of-use of the old text formats, but is stored and transmitted in the simpler and smaller binary form, making it more secure, easier on the energy bill, and easier on the planet.

* Versatile: Supports all common types natively. 90% of users won't need any form of customization.

* Future-proof: As a versioned format, Concise Encoding can respond to a changing world without degenerating into deprecations and awkward encodings or painting itself into a corner.

* Plug and play: No extra compilation steps or special description formats or crazy boilerplate.

https://concise-encoding.org

Reference implementation (golang): https://github.com/kstenerud/go-concise-encoding

Enctool, converter for playing around with the format: https://github.com/kstenerud/enctool


The pitch sounds a lot like that for Amazon Ion:

https://amzn.github.io/ion-docs/


Similar, yes, but Ion repeats a number of mistakes from the past:

* The text format doesn't have a version specifier, so any future changes to the text format will break existing codecs and documents.

* Uses ISO 8601, which has bad defaults (local timezone) and doesn't support real time zones (only offsets). ISO 8601 is also too big for what it does and non-free, resulting in inconsistent implementations (and thus security problems).

* Doesn't support chunking (so you must always know the full length before you start encoding).

* Lists must be prefixed with their length, so you must know the length before you start encoding. Converting a list from the text format to binary format would require loading the entire list into memory first.

* Only supports "struct" encoding, so you can't have real maps (for example using integers or dates as keys).

* Doesn't support arbitrarily long scalar values.

* Doesn't support NaN or infinity.

* Doesn't allow to preserve comments when converting to binary.

* Doesn't support arrays.

* Doesn't support recursive data or references.

* Doesn't have a URL or UUID or error type.

* Integer, date, and decimal float encodings are inefficient.

* Allows multiple sections, each with their own version declaration as a feature for mixing specification versions in the same document. This means that ALL codecs must support ALL versions forever, increasing bloat and attack surface.


I think your format is closer to ideal than a few others I have seen. Like json, you don't have to prefix size of lists or objects, which is a plus (or in some cases needed) for streaming. Strings can be chunked, which can serve the same purpose for huge strings. The power of json is in its simplicity though. And a binary format should retain that a much as possible. The big advantage of binary formats is not the 'conciseness' (there are compression algorithms that can help with that, where needed) but the reduction in lines of code and CPU cycles needed to serialize and deserialize. Json requires escaping/unescaping of strings and base-2 vs base-10 conversion of numbers. A binary format should mostly solve these and beyond that be as simple to implement bug-free and secure as possible.


Yes, that is exactly the aim of the binary format. It has a few basic concepts like being byte-oriented, 1-byte headers for most things, ULEB128 encoding for large values, same chunking mechanism for all arrays and string-likes, same "open/close" mechanism for all container types, etc.

The binary codec is VERY simple, and can be trivially implemented for an async-safe or otherwise constrained environment. In fact, I expect that many implementations will only build the binary codec, since you could just pass any recorded binary data through enctool [1] or whatever to see or manipulate its contents as a human. Most systems would have no need to process the text format.

[1]https://github.com/kstenerud/enctool


My ideal binary format would contain exactly the types supported by json + binary - arbitrary size numbers. It would be something like this, maybe:

- 'n' - null - 'f' - false - 't' - true - 'i' - int (64 bit, little endian, two's complement) - 'd' - double (64 bit, ieee-754) - 'a' - array start - 'o' - object start (must contain even number of values) - 'e' - array/object end - 's' - utf-8 string chunk (64 bytes, must be followed by string chunk or string) - 'b' - binary chunk (64 bytes, must be followed by binary chunk or binary) - 0x80-0xbf - utf-8 string (0-63 bytes) - 0xc0-0xff - binary (0-63 bytes)

It is at least an order of magnitude simpler to implement than almost all of its contenders. It is just not as concise and doesn't support a rich set of data types (but then again, json does quite well without a timestamp type). It supports streaming. It has a unique representation for every value (if you ignore that ieee-754 can represent many integers and objects have no defined ordering). Reading a single byte can tell you in every case the type and the amount of payload data that follows.


> Uses ISO 8601, which has bad defaults (local timezone) and doesn't support real time zones (only offsets). ISO 8601 is also too big for what it does and non-free, resulting in inconsistent implementations (and thus security problems).

Ion doesn’t use ISO8601; it has its own timestamp specification, which is free, and much smaller than ISO8601; its effectively a subprofile of a profile specified in a W3C note of ISO8601, but that relationship is mostly of historical interest, since it isn’t specified an a (sub)profile, but independently. It also has a UTC default.

> Lists must be prefixed with their length, so you must know the length before you start encoding.

True.

> Converting a list from the text format to binary format would require loading the entire list into memory first.

You’d have to process the entire list before starting to write it to the final format, because the format is optimized for read efficiency not write efficiency. But you don’t need to read the whole list into memory; you can use a state machine with a very small window to count items in this lost, write the total, and then start copying items from text to to binary, or you could just counts as you arr processing items to a scratch file, then write the length and copy the scratch file contents.

Length prefixing variable length data values is kind of important though; I’d consider it a major strike against a binary format if it didn’t do that. (Though Ion does have the problem that its length records are also variable length.)

> Only supports "struct" encoding, so you can't have real maps (for example using integers or dates as keys).

You can, since it supports type annotations and has formats that can with annotations communicate that. (List would probably be the normal choice.)

> Doesn't support arbitrarily long scalar values.

Yes, it does, which is why the length fields for most values, including most scalars, in the binary format use VarUInt.

> Doesn't support NaN or infinity.

Yes, it does. (It doesn’t, in text format or data model, distinguish different NaNs, but it supports NaN, +Inf, and -Inf.)

> Integer, date, and decimal float encodings are inefficient.

Ion has arbitary precision exact decimals, not decimal floating point, so, no, it doesn’t have an inefficient encoding for decimal floating point.


> its effectively a subprofile of a profile specified in a W3C note of ISO8601, but that relationship is mostly of historical interest, since it isn’t specified an a (sub)profile, but independently. It also has a UTC default.

Oh good, that's an improvement. Unfortunately, nobody thought to include real time zones :/

> You’d have to process the entire list before starting to write it to the final format, because the format is optimized for read efficiency not write efficiency.

Knowing how many objects are in the list won't help efficiency because you still don't know how big each element is. So you still need to walk through the list regardless. With that in mind, there's no advantage to a size field over an end marker, but there are disadvantages.

> Length prefixing variable length data values is kind of important though; I’d consider it a major strike against a binary format if it didn’t do that. (Though Ion does have the problem that its length records are also variable length.)

I did add typed arrays to CE to support efficient storage of monosized data types such as bool and int and float. Those have chunked size prefixes.

> You can, since it supports type annotations and has formats that can with annotations communicate that. (List would probably be the normal choice.)

I'm just not a fan of requiring users to massage data structures and annotations to get basic type behaviors. Technically you can get everything you need from XML too, but the costs...

>> Doesn't support arbitrarily long scalar values.

> Yes, it does, which is why the length fields for most values, including most scalars, in the binary format use VarUInt.

Ah cool, didn't know that, thanks!

>> Doesn't support NaN or infinity.

> Yes, it does. (It doesn’t, in text format or data model, distinguish different NaNs, but it supports NaN, +Inf, and -Inf.)

As I understood it, the text format didn't have "nan" or "inf" literals... unless I missed it somewhere?


I started (but many many others contributed) to the Go implementation of WebRTC Pion WebRTC https://github.com/pion/webrtc if you aren't familiar with WebRTC it is really amazing tech. Check out some of the projects on https://github.com/pion/awesome-pion. Stuff like https://github.com/giongto35/cloud-game continues to blow my mind :)

Along the way I worked on a Go implemenation of DTLS that gets used on its own a bit https://github.com/pion/dtls

In another life I wrote Therac https://github.com/sean-der/therac. A PHP debugger that was viewable from a browser. I was a remote at the time (2015) and it was a great tool to use with others.

I also wrote fail2web https://github.com/sean-der/fail2web a web frontend to fail2ban


I'm building a simple replacement for Apache Airflow to manage some small data pipeline projects: https://github.com/fieldryand/goflow. Airflow started to feel too heavyweight for these projects where all the computation was offloaded to independent services. I wanted a solution with minimal memory requirements to save costs and avoid the occasional high memory usage/leak issues I was facing with Airflow. I know there are tons of existing solutions in this space, but nothing I found seem to match the Airflow features I needed, so I decided to build it myself. It's been a great way to learn Go.


I created a Firefox addon to display tabs in the sidebar, grouped by privacy containers (isolated cookie containers)

https://addons.mozilla.org/en-US/firefox/addon/container-tab...


Wow, thanks!


I created dither, a library for image dithering in Go. And then I created didder, a CLI tool for the library. I had become interested in the aesthetic effect of dithering, but realized that there aren't many tools that do the math correctly, and also provide many different algorithms to explore.

https://github.com/makeworld-the-better-one/dither

https://github.com/makeworld-the-better-one/didder


Starting in 2014 I built Reviewable (https://reviewable.io) because after getting used to Critique at Google I couldn't stand the workflow of GitHub's PR code reviews. Reviewable gives you interdiffs (even when rebasing), tracks what you've already reviewed, has a rich comment resolution system for multi-party reviews, and lets you know when the ball is in your court. It has also grown more customizable over the years, including plugging in snippets of custom code to, e.g., group files, determine if a file is reviewed, or determine if a review is complete.

It's not open source because, frankly, it's still hard to run a business on open source -- especially without VC backing (Reviewable is bootstrapped). After all these years things are still growing nicely and while I've run the show solo for a long time I started hiring full time folks in the last year or so. (If you think you might be interested to join, ping me and let me know!)


This is cool! I used Phabricator (https://www.phacility.com/phabricator) in the past, which I really liked, and this seems to have many of the same features.

I think your landing page could be clearer though. If I hadn't had used better tools than GH in the past, I don't think I'd understand what I was considering signing up for.


Thanks! I agree that I suck at marketing and would love to hand that job off to somebody with more skill. :)


I'm by no means an expert - only a beginner when it comes to this - but I'm happy to help if I can :)

Just let me know!


Drop me a line at piotr@reviewable.io and we'll chat!


I built milkman[1], an alternative to postman because I wanted one tool to integrate with the whole web development stack. One tool that contains my SQL, rest and grpc calls to easily debug issues. Also nearly none of those kind of tools that existed supported SSE which I also use in my day to day work. [1] https://github.com/warmuuh/milkman

Edit: I love these threads. You read so many things and Frameworks and random stuff that you really learn in what silo you are actually living


I built audiomass ( https://audiomass.co ). it is not a dev tool but a general productivity tool (allow people to record, edit and manipulate audio directly in the browser).

I just wanted something that works as promised and respects the user (open source, 70kb total payload size, no ads, no tracking, feature complete, etc)


This is great, well done!


I tend to worry about criticism and whether my things are actually useful enough for others to be worth the trouble. So I've never actually published anything.

When I was a teenager I made a bunch of games for which I created level editors and the like. This was before there were free alternatives that I knew of which fit my needs.

When I started with graphics programming I made a library with OpenGL helpers for setting up shader programs, textures, double buffered textures, etc. configurable via configuration files. I ended up using it for a number of private projects and my master's thesis.

I've also made a Vim plugin for handling projects: setting up working directory, loading session files, setting a number of project specific paths and compiler options, etc. I wanted it to do enough little things that it was difficult to find a ready solution.

I once had a number of video files with subtitles which were out of sync, so I made a command line tool to resync them. It only took me a few minutes so it felt worth it and was very satisfying.


I encourage you to publish some of the things you’ve worked on. One way to reduce some of the concerns you mentioned would be to publish it using one or more anonymous accounts.

So long as your readmes have reasonable keywords, someone may just stumble on your project. If they don’t like it well it still won’t be attributed to you and you don’t even have to respond.

Either way, the practice of releasing and dealing with the weight of that is something that gets easier over time. You may find you enjoy it.


Thank you, I have thought about this before and might do so in the future.

At work I routinely make things which I know are used by millions of people and I've never had a problem with that. In fact, I'm quite proud of my work. And I've long since gotten over worrying about others reading my code. I'm also a big proponent of code reviews.

But the code being public still feels really scary, even though there's so much worse code out there which doesn't get sneered at.


You’re welcome. Those are good points and indicate this is likely a trick of personal psychology.

I felt the same way and it prevented me from releasing in the past.

There are a variety of ways to overcome this, but the absolute best is to just release something and find how little sneering happens.

In time you’ll realize that the gatekeeping and holier than thou view toward code can only be expressed by those who release even more.

If they hope to stay in the game, those who do will do provide feedback on your code do so with positivity.

Leaders know it takes teams of talented individuals capable of action to get stuff done. And they know abuse is anti ethical to recruiting talent.

So if it was someone you might want to impress judging your code unworthy, they’ll either offer constructive criticism or say nothing at all.


I created https://microapis.io to make it easier to launch API mock servers in the cloud with one click, no signup or login required.

The problem I found with most existing providers is they require you to create an account and often subscribe before you can use their services. For a quick test that's excessive overhead.

Until recently I was using prism[1] to run the mock server locally, but it's very limited when it comes to configuration and you have to install and run it locally, which again it's an unnecessary overhead when you want to run a simple test.

I've been using microapis.io for a few months now with my clients and it's proving a life saver to run integration tests and to help the frontend and the backend teams to work separately while ensuring they both comply with the same API specification.

[1] https://github.com/stoplightio/prism


Nice, will certainly check your project out. I created HttPlaceholder[1] some time ago. It is an open source mock server, which right now only has an on-prem version. I also have ideas to make a hosted version.

[1]: https://github.com/dukeofharen/httplaceholder

Edit: after looking through the comments, I found some 5 other HTTP mock server implementations XD Not that it's bad, just seems to be a problem that is tackled very often.


haha true, I guess there's a general feeling of dissatisfaction with the existing tools and frameworks, they're clearly not serving our needs so we have to create our own tooling. Thanks for linking your project, it looks awesome and very mature - gave you a star in GitHub!


As a frontend developer I often struggle with the component library to use. At last I set out to create my own https://nfui.js.org. It isn't complete yet. I still need to fix SSR and create a few essential component. But so far it is coming out great. With bundle size < 25K gzipped and more than 20 components already in place.

There are many I have tried before this project, but there is something always missing. I primarily work on React and so my experience can be a little different from people working on other view libraries.

Following are few libraries I tried and my pet peeves with them. Material UI was just too complex in terms of use. I had to always have a doc open in a tab and refer it for even a simple component like toolbar. Antd bundle size is way too large and few essential components like timepicker is > 100KB gzipped. React bootstrap doesn't have theming and looks outdated (Bootstrap 5 will fix this).


have you tried tailwinds and their uikit tailwindui?


I'd also recommend TailwindUIKit [0]. 100+ free Vue, React and VanillaJS tailwind components with light and dark mode support as well accessibility.

I did end up paying for the premium components, as I wanted to support their efforts, and it was reasonably priced, but didn't feel like I had to.

0: https://tailwinduikit.com/


I haven't tried it but I did go through their doc. Any thing slightly complex (like toggle) is behind a premium. I feel it would be great for creating layouts, but to create a component, I still have to go through the same trouble as material ui (but with classes instead of components).


I maintain an Emacs package that is fundamentally based around user interaction (called ido-completing-read+[1]). At some point I realized that I needed a test suite for it, but at the time there was no easy way to test Elisp code that reads user input without actual user interaction. So I wrote with-simulated-input[2], a package providing a macro of the same name that let you run a piece of code while supplying key sequence to feed to that code as if the user was typing those keys. So now my package centered around user input has a fully automated test suite, and that's pretty cool.

[1]: https://github.com/DarwinAwardWinner/ido-completing-read-plu...

[2]: https://github.com/DarwinAwardWinner/with-simulated-input


I built a little PWA just for me that I can use to scan ISBN codes of physical books I read. It then looks up the book and adds it to my spreadsheet of "Books I've read". I wrote a bit about it here https://nikhilism.com/post/2021/tracking-books-i-read-using-...

It was partly just a way to try out Svelte and XState.


That’s funny - I just started building something _exactly_ like this for my own website - I wanted mine to be more like a vintagy/iBooks-like proper wooden bookshelf that displayed the book covers of the books I was currently reading on the front page of my personal website.


Share the link please if available, would love to see it.


Last year I made Kondo, to clean up all the dependencies files from software projects you're not actively working on (eg node_modules). Awesome if you about to Zip up a bunch of old projects for archiving, or just want to reclaim disk space without deleting anything of value.

It was an experiment in Rust, and I even made a little GUI too!

https://github.com/tbillington/kondo


A couple of years ago, I built http://www.genewarrior.com

It's a suite of tools to work with DNA sequences (from simple tasks such as reverse complement to medium complex tasks such as multiple sequence alignments and primer design).

I built it because I wasn't very happy with the existing tools. It never got very popular, but after all these years I still think it's a very useful tool.


I wonder if some explanatory texts, alongside usability improvements could help you get more users. I can’t tell if your website is useful since I know anything about DNA sequencing - but from a UX perspective there is room for some improvements.


I agree, it could definitely use a make over.


Let me know if you'd like some help sprucing and getting the word out!


I built a safe PostgreSQL driver for node from scratch with zero dependencies in about 1200 lines. I did this mostly because of the plumbing required to make the goto node driver tolerable to work with. In the process I made it 4-5 times faster as well. https://github.com/porsager/postgres


Very nice! I'd ditch pg-promise in a heartbeat for something simpler. I kinda dislike the tagged template literal helpers magic. I'd be more comfortable with explicit references to substituted arguments.

Though it seems like I can just use the unsafe api all the time for that.

The rest of the API is very nice, .stream, .cursor, notify support, all that.


Thanks!

I'm curious - What's the reason you don't like the tagged template literal? It's those that give you implicit safe parameters and let's you avoid doing parameter counting when writing queries.


I don't like the sql() helpers part, and that they try to guess whether I want a helper for insert or update, based on some regexes and that helpers for different things all are sql(something) while doing very different things.

https://github.com/porsager/postgres/blob/master/lib/index.j...

I also tend to compose queries out of smaller static parts dynamically, and tagged syntax does seem to make that hard.

Patterns like:

    let params = {};
    let conds = ['TRUE']
    if (some_user_input) {
      conds.push('a = $test');
      params.test = some_user_input;
    }

    db.query(`
       SELECT cols, ... FROM xxx
       WHERE abc AND ${conds.join(' AND ')}
    `, params);
Not having a way to compose tagged literals and not having named parameters, only $1 just made it hard to port any of my existing projects based on pg-promise to your library.

I use named parameters, so I don't do parameter counting. Connector does that for me, when rewriting the parameters to native postgresql syntax.


That makes perfect sense. Thanks a lot for the feedback.


Anyway, I'm a fan of suckless (https://suckless.org/) and I try to use minimum set of dependencies (incl. transitive dependencies) in my node projects too. I like your project a lot from this perspective and that given its small size and lack of dependencies, it's also easily malleable to different needs, than the ones you anticipated.


I made https://esyvite.com to solve the problem of giving hosts an easy way to let their audience add events to their calendars. With the rise of online events (meetups, friend hangout, live streams etc) I realized that hosts simply post the times on their social feeds but their audience has to do the hard work of remembering and tracking the times.

Made it simple enough such that it requires no account, no payment and no app. Developed using React, Typescript, Firebase. Credit to all the libraries that I could leverage to build this.

Feedback on esyvite.com welcome at vamsi dot narla at gmail dot com


Nice! One bug report: the "Click here and checkout your esyvite" link seems to be missing the https protocol which makes the link broken (goes to esyvite.com/esyvite.com/e/xxx instead)


Ah, thanks for catching. Made some last minute change before posting to HN and broke it :facepalm: Truly appreciate the bug report.


I wrote a tool to tail CloudWatch logs called saw: http://github.com/TylerBrock/saw

I was so fed up trying to debug lambdas and the other AWS offerings that a tool like this to see program output in near realtime is indispensable. I use it all the time.

Is it bug free? No

Did i write tests for it? Also no.

Is it useful? Yes.


I was dissatisfied with SemVer and it turns out that most people are using it wrong. So, I created ChronVer[0] (Chronologic Versioning). There's a JS/Node plugin I made that will automatically increment your package.json version, and someone else made a Rust crate!

[0]: https://chronver.org


Interesting idea but it's probably only suited to end users of applications, not for npm libraries as the landing page proposes.

The reason sem-ver is better for libraries is that it indicates the compatibility of different version's API's. It doesn't matter when the version was released, just if it has breaking changes, or is just a patch release. Sem-ver is the simplest solution to this problem.


It works for me.

For people on large teams or orgs, this merging of SemVer and ChronVer would probably be best: https://twitter.com/grin_io/status/1390747474780901380

I may replace the branching model I have in the current spec with this.


Most recently, I wrote this Docker deployment tool because I was tired of having to either set up a Kubernetes cluster or manually log into my home server to update stuff:

https://gitlab.com/stavros/harbormaster

It allows you to run Docker Compose containers from a simple git repo (no Docker registry), and to specify all the Compose apps you want to run in a straightforward YAML file. It'll take care of automatically pulling/restarting/deleting everything for you.

I love it.


This seems cool! I recently moved my homeserver and dev environment to a big docker-compose file and am loving it. Just curious, what prompted you to use multiple docker compose files over a single one?


The big advantage is that you can reuse the Compose file for other computers. For example, in my model, Home Assistant would come with a Harbormaster-compatible Compose file, so to run it you'd only need to include the Home Assistant repository URL in your config file, and you'd get automatic updates and everything without having to do anything else.


- adhocify: I wanted to launch commands upon file system events, however without first configuring a daemon for that task. Therefore, I created adhocify: https://github.com/quitesimpleorg/adhocify

- qssb.h: A header library to make sandboxing applications on Linux more easier. Currently at an early stage though: https://github.com/quitesimpleorg/qssb.h The original aim was to make utilizing namespaces and seccomp easier without dealing with nuances. Currently working to get landlock in, then perhaps a CLI utility and test cases.

- raou: I found sudo too complex for the simple taks of switching users on a system. Given sudo also had some vulnerabilities in the past, I decided to write an alternative in Rust: https://gitea.quitesimple.org/crtxcr/raou

- qsni: https://github.com/quitesimpleorg/qsni At times I simply wanted to cut off a few applications from the network or to assign some specific firewall rules: https://github.com/quitesimpleorg/qsni


I've written...

* a Java library to open up ports on your router, because at the time nothing existed for Java that handled all 3 major port mapping protocols.

* a Java library implementation of Kademlia, because I wanted to learn the protocol.

* a Java coroutines toolkit, because I needed the functionality but at the time Javaflow was dead but Project Loom wasn't yet a thing.

* a Node library that extends markdown-it with my own special flavour of Markdown that I use for taking long-form notes. It automatically links terms in the document and lets me call into container images to generate outputs (e.g. generate chemfig/tikz/svgbob/graphviz diagrams, run a block of python and spit the output, etc..). I built this one because, at the time, Google Docs just wasn't cutting it in terms of features and had some bugs when the doc was hitting the low hundreds of pages. A sample of the MD <https://raw.githubusercontent.com/offbynull/learn/master/Bio...> and its rendered output at <https://offbynull.com/data/learn/Bioinformatics/output/outpu...>.


I recently built this: (https://github.com/Link-/github-app-bash) as a quick way to generate access tokens to use with GitHub's APIs. Now I can use tokens with a limited scope, and lifetime when I test which is safer than using a personal access token that never expires. It's fully written in bash and does not have many dependencies. It's quick does the job.


I made https://foragoodstrftime.com/ - which isn't open source (But honestly is just a few lines of code). I made it as I found it useful and hoped others would as well.


Great name


Still in the process but: a software renderer, a cross-platform gfx / audio api, a scripting language, code editor, sprite editor, ... (libs are all written in C for portability and easy scripting integration) Totally meaningless (to others) wheel reinventing, but it makes me 100x happier when making games and apps!

DIY everything is my path to programming happiness (in any field in fact). Tools should be extensions of your body, tools made by others are unlikely to be that.


There is a lot of truth to this. I'm generally good at avoiding this at work but I devolve into "DIY everything" in all of my side projects. It's certainly not the most productive way of producing software but it's the most fulfilling.


There are four things I quite enjoyed creating, in no particular order:

- https://github.com/rcarmo/pngcanvas because there was nothing similar that ran on Google App Engine

- https://github.com/rcarmo/imapbackup because I needed a migration tool from Sun IMS to Exchange

- https://github.com/rcarmo/rss2imap because it helped me survive well beyond Google Reader for a long time

- https://github.com/piku/piku because it enabled me to automate away 99% of my non-Docker deployments and is a lot more fun than Kubernetes on low-end standalone boxes

There’s also my blog engine (https://github.com/rcarmo/sushy), but that is in a state of flux right now and I haven’t published the new source yet.


looking forward to see the revamped sushy code. Am currently hunting for further inspiration for the OCaml successor of https://github.com/mro/ShaarliGo


I built an Android app that lets you create bookmarks for any kind of Android audio track (e.g. played by some music player or podcatcher app). My app, "Stop It!", captures which player app was used, which track/artist was playing, as well as the exact timestamp. I create bookmarks whenever I hear something I like (but I don't have time to pause the playback and make notes). Stop It lets me get back to that particular track and time stamp easily, hours or days later, and avoids that nice things fall through the cracks. The bookmark creation mechanism is to pause playback and immediately resume it, which you can do with any Bluetooth headset - no need to unlock your screen. The app is open source, but not on the Play store. I built it because nothing like this exists... More details here: https://www.augmentedmind.de/2020/08/16/stop-it-create-audio...


I created 2 things which I'm really proud of, because they solve the exact problem in elegant way.

First one is ProxyAV [1] - reverse proxy that scans any uploaded file with ClamAV. That tool/microservice was created, because backend developers didn't have experience to work with antivirus on their side, so the "mitm proxy" way allows to integrate antivirus scanning without bothering backend devs, they always receive clean files

The second one is etke.cc [2] service. I love matrix, but self hosting of matrix homeserver requires good level of expertise, even for skilled sysadmins. I just wanted more people join matrix with their own homeservers, so etke.cc service is perfect solution to that problem - you want your own matrix homeserver - you order it on etke.cc - you get your matrix homeserver, on your domain, on your server, with full control of it.

[1] https://gitlab.com/rakshazi/proxyav

[2] https://etke.cc


I needed a good Node wrapper for ExifTool and wrote https://github.com/photostructure/exiftool-vendored.js

When I saw how slow it is to fork child processes in Windows, I then realized I wanted to run ExifTool in "stay-open" mode, which meant I needed to manage 1 or more long-lived child processes that communicate via stdin/stdout, so I wrote https://github.com/photostructure/batch-cluster.js

In switching to TypeScript, I really missed Scala's `Option/None/Some` and `lazy`, so I added those (and several other small, helpful functions/classes) that I documented here: https://photostructure.com/coding/uncertain-lazy-forgetful-a...


- R package "matrixTests": https://github.com/karoliskoncevicius/matrixTests

I often have to perform statistical tests on genomic data. Which in practice means running the same test on every gene (row of a matrix). Running it separately on each row is slow, specially in R. Hence to speed it up I started an R package which is a lot faster and deals nicely with edge case scenarios (missing values, infinities, empty matrices, etc).

- R package "annmatrix": https://github.com/karoliskoncevicius/annmatrix

An S3 object for matrices with persistent metadata. This again is mostly relevant in genomic contexts where a matrix of, say, gene expression has to also keep information about rows (gene names, positions, chromosomes), as well as samples (disease status, age, sex).

- R package "basetheme": https://github.com/karoliskoncevicius/basetheme

On of the advantages of ggplot2 in R is the ability to specify themes. I prefer working in base plotting system and ended up creating a package that allows setting themes for base graphics.

- Vim plugin "sendtowindow" https://github.com/karoliskoncevicius/vim-sendtowindow

A minimal plugin which implements an operator for sending text to another window. Handy for sending code from the buffer to a REPL running within another ":terminal" buffer.


I developed a Lua IDE/editor/debugger (in Lua) [1], as I was looking for something that would be simple enough to use for high school students (for a project I was working on at that time) and have been working on it for the last 10+ years. I do have several commercial extensions/integrations on top of the IDE.

I also developed a Lua debugger [2] and serializer [3], as well as took on maintenance of wxlua project for the last 5 years [4]; all open source.

[1] https://studio.zerobrane.com [2] https://github.com/pkulchenko/MobDebug [3] https://github.com/pkulchenko/serpent [4] https://github.com/pkulchenko/wxlua


https://github.com/jhspetersson/fselect

A tiny tool I wrote to search within file piles (mostly unsorted downloads, torrents, and such). I could never remember `find` options, and more advanced queries are a pain. Now one can use some kind of SQL flavor to get the job done.


Looks awesome! SQL seems like a much nicer interface for this kind of thing over a pile of command line flags.


Quepid (http://quepid.com) and Splainer (http://splainer.io) both are tools I created a long time ago to work on improving search result relevance in Solr or Elasticsearch. Both are open source.

I literally started coding Quepid next to a colleague complaining about search so I could help them. It’s basically a kind of test driven approach to continually tuning and proving you’re not harming existing use cases too much. The first version that day was super hacky :) but it worked: it was a “single page server side application” ;). Just dumping a huge HTML page from flask about how our search was doing based on stakeholder feedback.

Nothing out there existed that did what I needed. Tooling around search engines for relevance just wasn’t great. In part because it wasn’t paid as much attention to ~10 years ago when I worked on creating these tools.


Cool! :) I'll definitely take a look at this. I'm working on a search thing now and need some way to measure and improve search result to query relevance. Thanks for posting! :p ;) xx


There is a new, emerging job role, the "relevance engineer", who ensures that search results stay relevant as content sets changes.

In London, the annual Search Engine Solutions meeting (each autumn/fall) organized by the BCS IRSG (Chartered Institute of IT's Information Retrieval Specialist Group) provides a forum for search practitioners to exchange best practices.

Here's the report from the 2020 meeting: https://irsg.bcs.org/informer/2021/01/search-solutions-2020-...

Talk proposals, demo proposals, tutorials etc. welcome.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: