Seeing tools like this pop up periodically and get so many upvotes it seems a lot of people aren't aware of great tools. It would be interesting to see a gallery of Unix classic/modern tools presented in a structured course. Is anyone aware of one?
Something like images (imagemagick), video (ffmpeg/vlc), audio (sox), phone (asterisk/etc.), SMS (pdus/various), linguistics (various), classification (NNs), version control, snapshots, clustering (PXE/corosync/pacemaker), security (kernel toolkits), transaction systems and databases (sqlite/RDBMS/noSQL/time series), networking (filtering/firewall rules, intermittent connectivity, local multidrop protocols, tcpdump/wireshark), kernel security toolkits, CI/CD, etc.
Scope would be essentially everything except tools lying within the popular web stacks.
The goal would be that a student with interest could work their way through the syllabus and emerge a capable multi-domain unix hacker, rather than needing to encounter these problem domains over decades of career, rediscovering well worn approaches.
I learned programming through the web, so I lack a lot of knowledge in basic systems / bash / *nix / I'm not even sure how to call that.
I've googled so many time "learn bash" etc but it's never really what I'm looking for.
I regularly discover tools that are so basic yet so powerful (all things you mention) yet I feel there is a lack of (great) resources online to "learn the basic tools of your CL".
You mean you don't just go read the man page for every bash command and intuitively understand it?
Sorry for the sarcasm, I just feel your pain. Learning even basic things in bash has been slow for me too. I love it. I prefer it to GUIs on most things I have figured out.
Well, the way this person uses ImageMagick can hardly be described as "basic". In fact, he's a wizard. If you look through the bullet points at the top, he links to an impressive CV and PDFs with mathematical descriptions of how some of his scripts work.
The best way to learn in my opinion is to find someone who knows more than you and have them be your mentor. You don't even have to ask them directly, though many people I think are glad to teach.
Identify someone at your work or school who you feel always has a nice perspective and who you trust; then seek out their opinion and advice when you have questions. You can learn a lot in a fairly short amount of time by applying this practice.
You can also find what I've heard called "silent teachers" - there's no formal or informal relationship between the two of you, but you're always watching what they say, observing the things they care about, etc.
If you're in a situation where you're not employed full-time in a technical position it becomes a little harder but you can still do it. Much of it depends on your local context.
That's how I learned a lot of things; by literally scanning through everything it had to offer and googling what I didn't understand. I did that for linux utils, language references, most programs I use and most codebases I had to work on.
One notable example is the time I had to learn C# 4 years ago. It took an entire morning to scan through the entire language reference and then later I found out I already knew a lot more than "advanced" devs who would just look up stuff as they needed it, without knowing what actually is available in the first place.
I feel its the only sensible advice if you want to learn a lot on your own with limited time.
I’m not saying you can’t learn this way. I’m saying it’s not practical for most people. Many people who would fail this way would succeed other ways. Reading man bash as primary learning material will lead to failure for 999/1000 humans.
I’ve wanted to read man bash, and believe me, I’d think I was the kind of person who would (I learned Go just reading the spec. It’s a great spec), but man bash and many other man pages have never worked for me. Maybe if I tried again now.
bash, incidentally, has a man page that sucks. I've found that most larger GNU projects have pretty horrible man pages. Probably due to feature creep in said projects - so the manuals have to be large to cover all the features.
I spent 10 years avoiding awk, because every time I typed 'man awk' I was overwhelmed by the GNU monstrosity. Then I found the man page for the plan9 implementation of awk (http://man.cat-v.org/plan_9/1/awk) and learned the language in fifteen minutes.
> I've found that most larger GNU projects have pretty horrible man pages. Probably due to feature creep in said projects - so the manuals have to be large to cover all the features.
Rather, info manuals are GNU's official (self-invented I think) documentation system. They make manpages only because of their popularity, but they direct you to the info pages for more info. When other projects have too much documentation for a manpage, they split it among several pages, like perl, openssl, zsh, git, borg, btrfs, etc. Personally, I like both, and wouldn't rank one above the other.
It's far better than Googling or guessing for them. Looking in those places provides exactly what is on your system and the man pages exactly describe what they are for.
That’s a good use case. Believe me I’m into man pages. Every 6 months I check the Apple store to see if anyone has created a good man pages app. But as primary learning material for most people: not so sure.
Such galleries in my opinion already exist and simply require an afternoon hitting github for "* awesome (list)" with every topic you've listed. Then devoting enough self-teaching time to learn the material. These lists tend to be curated by domain experts and are already in a structured format suitable to the domain.
Here's 1 such example from the 1st result from a Google search for "github awesome security list":
Hmm just tried it for audio and nothing I found in three audio-related lists off the curated awesome list of awesome lists even mentioned sox. I do think you suggest a good search strategy in very new areas, but I don't think it's quite the same thing as presented in the grandparent.
Personally I discovered a lot of great tools in my distribution's package management database.
You're right, not enough people know about sox. I can spend hours making random sounds with the synth just for fun. I've been contemplating making something like dwitter.net for sox as a side project. People would submit sox commands for various interesting sounds.
> It would be interesting to see a gallery of Unix classic/modern tools presented in a structured course. Is anyone aware of one?
Back when I used Gentoo, I discovered a lot of interesting software by simply browsing their portage repository and installing packages that sounded interesting. It's organized into sections like you describe. There were (probably still are) web sites to browse the package database too. FreeBSD's ports are organized in a similar manner.
I want a tool that does that kind of conversion for me. I've been looking(not actively or with any sort of effort) for such a too since my friend showed me a mac only text based converter that did something like the above. I've been toying with writing one.
A good book for learning the Unix basics is "The unix programming environment" by Kernighan & Pike. This is an old book but the concepts it teaches are invaluable even today. We used it in our university around 20 years ago but I still recommend it to unix neophytes!
It would need to be automatically collated by an AI with access to the Internet Archive that is probably quite a bit more sophisticated than what we currently have. DMOZ is the closest curated thing I can think of.
I can't seem to find the link now, but there was a post here a couple of weeks back about someone doing a very similar thing (in Python, IIRC) to clean up pictures of handwritten notes. It seemed conceptually simple, and worked quite nicely.
Wonder how close this is to the techniques apps like Office Lens use. I am always amazed to the extreme angles I can take a photo and the app still makes it end up as perfect pdf.
Oh man this page brings back some golden memories! I remember working with ImageMagick for the first time in 2010 or 2011 when I wrote a bash script to programmatically scale and center thousands of images for creating image gallery thumbnails on an internal social platform for winners of a design award. I somehow landed on this page and used the massive amounts of knowledge to learn how the ImageMagick scripting interface worked. Then it was a fairly simple task to build the script, but - like all bash scripts for me - I've forgotten every single parameter now :)
Somehow that feels much longer ago than just 7 years, maybe its just the website or the feeling of nostalgia looking at it.
I don't have the script on hand at the moment. But essentially I take two 180 degree fisheye photos one pointing straight up and one pointing straight down. (for this, I used a homebrew cardboard box and two mobotix cameras)
I then use OP's script to "unroll" each fisheye photo. I then stick them together top to bottom with looks like this:
Huh, I remember as a teen bring blown away by IPIX ('90s). It dewarped a fish eye image so that you could look around a scene from a fixed vantage point.
For those interested we implemented from scratch a subset of these commands as a REST API service available to call at request from any programming language.
Interesting to see this re-surface on Hacker News. This set of scripts has to be among the most useful things I've found on the internet, and I've used some of these in many of my projects!
These are great and I need to start using these. For instance the dominantcolor script extracts color schemes from pictures. Something I've tried to do by eye in the past with limited success.
If ImageMagic isn't your thing and like a GUI but don't like Photoshop or the GIMP I really recommend PixInsight (http://www.pixinsight.com/). It's normally used for astronomical image processing but it's well build with some advanced stuff in there. Unfortunately not free, but people have to pay the bills!
No no no, we can't have this, that's too old fashioned!
Proper way to do this is to create an image processing SaaS around these scripts, announce to store the filter presets on blockchain and create a marketplace platform for filter creators, have a $50M ICO, spend the cash on lambos, and fail awesomely.
I prefer the PHP version of ImageMagick as PHP is a better scripting tool than 'cli'. The manual is a bit hard to follow as things are essentially undocumented. However, if you want to carefully prepare a batch of images, going to PHP is taking it to the next level.
Particularly interesting are features such as being able to save images in different colour spaces, e.g. 4:2:0, going lo-res on the chroma for a lower ultimate file size in jpg.
If I had a lifetime spare I would complete the PHP Imagick documentation and write a version of Fred's scripts in PHP, so they can be chained together in a nifty object oriented way...
The GitHub wiki for VIPS has a page [1] very relevant to GP's question. It lists two dozen alternatives to VIPS (both command line tools and libraries) and compares how they performed on a load-crop-scale-apply convolution-save task. The source code for each implementation of the task is right there on the page.
I second that.
G'MIC has an impressive bunch of built-in commands to manipulate images, see e.g. the examples shown in the reference documentation (click the first magnifying glass, then use right arrow on the keyboard):
And as the commands are built-in, no needs for external scripts, those effects are available from any interface using the `libgmic` API (including C/C++ programs).
So everyone that ever figures out how to do something neat with a free tool and documents it gets to claim the rights to that and ask for money?
Might not even be against this as an idea, but the premise seems shaky to me, I guess
Maybe more to the point is- if you want to ask for money for doing work like this, don't publish the ImageMagick command equivalents? Though I guess that would be hard to avoid other than putting it in some proprietary UI/app or something. Oh well.
Something like images (imagemagick), video (ffmpeg/vlc), audio (sox), phone (asterisk/etc.), SMS (pdus/various), linguistics (various), classification (NNs), version control, snapshots, clustering (PXE/corosync/pacemaker), security (kernel toolkits), transaction systems and databases (sqlite/RDBMS/noSQL/time series), networking (filtering/firewall rules, intermittent connectivity, local multidrop protocols, tcpdump/wireshark), kernel security toolkits, CI/CD, etc.
Scope would be essentially everything except tools lying within the popular web stacks.
The goal would be that a student with interest could work their way through the syllabus and emerge a capable multi-domain unix hacker, rather than needing to encounter these problem domains over decades of career, rediscovering well worn approaches.