I've been a die-hard Linux user for about a dozen years. Recently I had to do some development with MS Powershell. I was very reluctant at first, but after getting familiar with the technology, I almost fell in love.
"Cmdlets", basically commands used in Powershell, output "objects" instead of the streams of text used in a more classical shell. Powershell has built-in tools to work with these objects. For example, you can take the output from one Cmdlet, pipe it through `SELECT` with a list of fields specified, and get a stream of objects only containing those fields. Other operations can be performed against those objects as well, such as filtering and whatnot.
Back to normal nix commands, we're starting to see more and more commands introduce direct JSON support [1]. There are even tools to translate output from common commands into JSON [2]. We'll probably see `jq` shipped directly with modern distros soon. Eventually we'll reach a tipping point where it's expected that command supports JSON output. Tools like `awk`/`sed` might get updated to have a richer support for JSON. Finally, we'll have ubiquitous Powershell-like capabilities on every nix machine.
Powershell _is_ available on Linux. The model of piping objects instead of JSON is both powerful and more efficient (For example, there's no redundant keys like in a stream of JSON objects, leading to less moving bytes, like how CSV headers aren't repeated with every row. Plus, binary data is smaller than text.) But, most developers are hesitant to switch out their shell and existing workflows for a completely new tool, which is why Powershell will likely only be adopted by a small subset of sysadmins.
Though it's pretty immature, nushell has a similar idea, with its own internal data model being streams of structured, typed data: https://www.nushell.sh/
And back to nix commands, libxo is used by a chunk of the FreeBSD base tools to offer output in JSON, amongst other things: https://github.com/Juniper/libxo
> But, most developers are hesitant to switch out their shell and existing workflows for a completely new tool, which is why Powershell will likely only be adopted by a small subset of sysadmins.
What I would like to see is some sort of stddata stream be offered by the kernel itself so devs won't have to switch their shells and object manipulation can be a standard.
Not only that, but pwsh support for objects doesn't stop on passing objects around and mapping to properties to parameters. There are number of mehanisms in place. All nix variants solve just 1 of those mechanisms.
IMO, powershell should be added to ALL mainstream distros as first order citizen. There is no downside to that given that MS is now legit FOSS player and that anybody can fork in case something goes wrong along the way...
That is a no then, as the MIT does not contain an explicit patent grant.
So Microsoft could very well sue anyone using Powershell, using patents. We don't know if they would win, as there might (or might not) be some protection in a "implicit" grant that a copyright license give. But it would be a fight most organizations would not be willing to take, and would settle out of - a win for Microsoft in practical terms. It is not long ago since they did this with FAT32 against practically all Android manufacturers.
> deal in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so
Looks like a corner case, but really, if implicit grant is not enough then ANY explicit grant will miss some of the real life moments - you mentioned patents, but probably many more exist or will exist in the future.
If somebody pulled this trick on me, and I were a judge, I would punish both laywer and a company he is representing. Its simply emberrasing and belitteling toward human intelect.
Also:
> the Open Source Initiative has stated its view that all open source licenses implicitly include a patent grant
Very neat project! I think visibility into intermediate stages of a pipeline can be enormously useful for data restructuring tasks, where the individual steps aren't necessarily "difficult" to understand, but it can be hard to keep track of what's going on without a good feedback loop.
Here's a demo of a prototype live programming environment I made for jq, which similarly shows step-by-step views of the data but also gives live feedback as you construct your pipeline:
By the end of that Twitter thread I ultimately morphed it into a tool for building interactive GUIs (eg, get API data in JSON, use jq to morph it into the right shape for your UI, output to a HTML template).
That is very cool! For other readers, "process" means submit jq-like document queries, and "various formats" means other JSON or JSON-like representations, such as BSON, Bencode[1], TOML, XML, and YAML. Thank you for sharing!
[1]: In 2001, prior to the huge popularity of JSON, it is an ASCII-coded dict+string+int+list format used in Bittorrent .torrent files.
Thanks for this link. If you'd like to create a GitHub issue, I'd appreciate it. My justification for linking to libjq is that it's a moving target: there are various builtins added across updates etc...
As a JS dev I tend to have node installed anyhow so I just use a shell alias to wrap ‘node -pe’ these days. It’s not really for shell scripts but it’s great for quick every day usage. Plus you can use JS if needed instead of their DSL.
Is there something similar for YAML? I've tried `yq` briefly but weirdly enough it doesn't seem to accept standard input in the way that jq does (ie pipe in some json, and output some pretty json)
Worth noting OP has tried it and has some critiques.
> I've tried `yq` briefly but weirdly enough it doesn't seem to accept standard input in the way that jq does (ie pipe in some json, and output some pretty json)
it shouldn't be too difficult to convert between yaml and json, funny i couldn't find a light weight converter easily. I think i will try to write one.
Basic YAML, sure, but YAML has some insane features that I don't think would be possible to replicate in JSON. Or, at least, you'd lose some information in the conversion.
For example, YAML supports the concept of reusable fragments. You can define a fragment in one place and reference it further down in your YAML file. A JSON converter could take the final YAML output and turn it into JSON but you would lose the context of the fact that in the original YAML it was an included fragment and not just the same section repeated a few times.
Yeah, YAML can also embed arbitrary code in the original Ruby incarnation and that clearly cannot be translated to JSON.
Yeah, no straightforward translation of references to JSON. You could provide both translation ramps, but it would be an implementation-specific convention and not something other JSON tools understood.
I don't know what the term would be, mental model, but I just can't get jq to click. Mostly because i only need it every once in a while. It's frustrating for me because it seems quite powerful.
I tend to only use it for relatively simpler queries than it is capable of. Ditto regular expressions. But I do not find myself missing advanced features of either very often. If I run out of my ability with jq, the manual page and playing around with the query sometimes illuminates the correct answer. And if not, it's often time (for me) to switch to using a less ephemeral program anyway.
I like jq for simple stuff which is what I mostly use it for. Whenever I have to dive into the documentation to do something complicated I die a bit on the inside.
I like how you have output for each process of the pipeline, however it would be much better in terms of usability if you could dynamically load the result just below the query rather than opening a new page.
Would like to have a kind of "rosetta stone" where each of these examples is rewritten by passing the json to "gron" and then using the standard unix tools.
I guess some of the examples would be simpler than the jq solution.
God forbid that someone should ever write (- y + x) rather than (x - y), what a useless use of the plus sign!
Why have a problem with this? Catting a single file is a well-known idiom for outputting its contents into a stream, plays well with positioning in pipelines, and has the nice property that you can erase as much of the pipeline as you want, to be able to peek inside it at any point.
Completely agree. Cut it if your file is 10GB of logs because it requires an extra stream copy, but otherwise, it is the best way to interactively drill down into data and write the pipeline as you go. It is almost never a valid criticism, and when it is, calling it Useless UOC (reads as an insult to the author!) was the worst way to communicate that to the masses.
not only that but I do the same when I use the CLI because it makes sense to say "I have a file. Now I will pipe it to ___" rather than "Using ___ I will pipe a file in then ___".
It just makes more sense to do things in order even if it's a couple extra characters.
I was going to suggest that shells should allow specifying the input redirect before the command, but it turned out that "<input command" in bash already works. Anyone knows about other shells?
This is so much easier with Powershell. V7 is totally cross-platform and I cant see why people have a problem to use it, if nothing else then for `ConvertFrom/To-Json/CSV/Whatever` cmdlets ..
As someone who started out scripting in Powershell but preferred Linux as an OS. I found myself missing powershell's object passing in opposition to bash's string passing.
That being said I now primarily use iPython for advanced shell tasks as I can leverage all of Python's libraries like JSON or YAML.
> That being said I now primarily use iPython for advanced shell tasks as I can leverage all of Python's libraries like JSON or YAML.
TBH they're both excellent choices. nushell looks really good too. The point is not to scrape text - rather than write a bash script that can handle JSON properly, write a pwsh/python/nushell script that handle everything properly (ie, by selecting fields rather than scraping text).
As someone fairly ignorant of Powershell, how well does it interact with the rest of the Unix-minded ecosystem? jq is nice because its plainly compatible with being piped into another command like xargs. I don't want a better jq if it means relearning and remastering everything else along with it.
"Cmdlets", basically commands used in Powershell, output "objects" instead of the streams of text used in a more classical shell. Powershell has built-in tools to work with these objects. For example, you can take the output from one Cmdlet, pipe it through `SELECT` with a list of fields specified, and get a stream of objects only containing those fields. Other operations can be performed against those objects as well, such as filtering and whatnot.
Back to normal nix commands, we're starting to see more and more commands introduce direct JSON support [1]. There are even tools to translate output from common commands into JSON [2]. We'll probably see `jq` shipped directly with modern distros soon. Eventually we'll reach a tipping point where it's expected that command supports JSON output. Tools like `awk`/`sed` might get updated to have a richer support for JSON. Finally, we'll have ubiquitous Powershell-like capabilities on every nix machine.
Powershell _is_ available on Linux. The model of piping objects instead of JSON is both powerful and more efficient (For example, there's no redundant keys like in a stream of JSON objects, leading to less moving bytes, like how CSV headers aren't repeated with every row. Plus, binary data is smaller than text.) But, most developers are hesitant to switch out their shell and existing workflows for a completely new tool, which is why Powershell will likely only be adopted by a small subset of sysadmins.
[1] https://daniel.haxx.se/blog/2020/03/17/curl-write-out-json/
[2] https://github.com/kellyjonbrazil/jc