Hacker News new | past | comments | ask | show | jobs | submit login
Introducing nushell (jonathanturner.org)
1477 points by bradleybuda 9 months ago | hide | past | web | favorite | 366 comments

The Powershell idea to push structured data through the pipe, which this project fundamentally replicates with a relational structure, has its merits; but it will always displease someone, and carries the burden of having to rewrite millions of utilities in order to unlock the real potential.

What if, instead, we pushed the approach “upstream”, asking systems to have an additional stream that all shells can access? We have stdout and stderr, we could add “stddata” - a pipe that expects and produces structured data. Then it would become trivial to add support around the userland, without displeasing anyone - because it’s just an additional interface, not a replacement. The pipe should support two or three formats to start with (csv, json, and maybe xml, so you can handle both tabular and child-parent relations with a varying degree of precision) and shells could have a special character for piping stddata (I like ¥) so everything else about them would stay the same.

I am at the stage that I am more than happy to rewrite whatever it takes to get better experience. This includes the OS. I think we have sacrificed too much on the altar of backward compatibility. I guess I am just tired of breakage because a file has a space in its name.

> I am at the stage that I am more than happy to rewrite whatever it takes to get better experience. This includes the OS

Ever tried to rewrite one (and the software on top of it, since we're "including" the OS) :)?

Shout-out to anyone who remembers being blown away by BeOS when it was brand-new and extremely promising

I now have the Virtual Void song stuck in my head.

Yeah, some places I have worked we had a research kernel because we could not measure HW performance accurately with Linux.

Hey, no cheating -- one that has a substantial baggage of applications on top of it. A research kernel is something you can do in one year working with a few dedicated undergrads. But then so is pretty much every single thing running on top of it. It looks easy until you figure out how much work gets into properly writing a PNG decoder or a PDF viewer. It's not fun, even in a memory-safe language.

I admire your optimism and I, too, wish we'd leave this POSIX hellhole behind us (and I'm fond of this POSIX hellhole!) but I think at this stage in the story of computers, it's no longer realistic.

I think the "POSIX hellhole" is really just the Linux hellhole; there's no reason the underlying implementation or the desktop on top of it needs to be that way (and Haiku is more or less succeeding at being POSIX and not a hellhole :)

Oh yeah, don't get me started on the Linux hellhole :). I like Haiku (in fact I keep putting off that weekend when I'll try my hand at fixing a bug or writing a new app for it, because I really liked BeOS back in the day, too). A lot of it isn't even POSIX' fault, it was just abused by things built on top of it, and now you can't use those things without making a mess.

I know a lot of it like the back of my hand, I've been writing POSIX code (or, at one point, POSIX implementations) for much, if not most of my career. That doesn't mean I don't want to see something better, or that I'm blind at the fact that Plan 9 is so fun to code for precisely because they skipped POSIX (although, that being said, I'm also not blind to the fact that it's on of the big reasons why it never really took off, either).

Totally agree with you. Now the real question, do you need to implement PNG decoder and PDF viewer? Or you need to design alternatives that are far more safe to implement?

Everyone has different criteria for what a new and improved OS needs. Some say make it safer. Some say it must be much simpler. Others say why is it so inefficient, make it faster and focus on mixed level languages. Nobody can agree and nobody needs any of these badly enough to effect a real change. So we have these brainstorms every year and the work that comes from them never gets far.

Might be a sign that what we have isn’t that bad after all

> Now the real question, do you need to implement PNG decoder and PDF viewer?

If you ask me, no, but that's because I remember (and not very fondly) an age when it seemed like we'd have a new arbitrary code execution bug in a PNG or JPEG or PDF decoder every week. I don't really wanna go through it either.

Edit: Lots of people don't remember (or don't know about) that period and think writing an OS and all the boring stuff that goes with it is easy. Four or five years and it should be all done, eh?

That's also why we got things like the security community, which is full of enthusiastic people and big companies that know exactly what OpenSSL needs and how it should be written. But when it comes to sitting down and doing it, or committing the money, the crowd gets awfully small. Foundational software isn't too glamorous, and writing things like a PNG decoder is pretty boring, too.

Why not just push all that data through stdout and interpret it at the other end accordingly?

Another option would be to just use the 4th fd.

Using te 4th fd is exactly what OP proposed?

Well, not exactly. He merely suggested using another one. But I thought it would be good to talk something a little bit more specific. And I am not sure he knows there are already others available (for shell scripts at least fd[0-9]).

After all, /dev/std{in,out,err} are more or less just convenience links (allowed extensions of POSIX). But using /dev/stderr might fail on some POSIX compliant systems.

Not breaking stdin and stdout is key. Looking at the output from that shell, text parsing tools like awk are not going to integrate if you dont have a compatible tool yet. Everything in a base Linux install has to be rewritten to work.

Personally i prefer the keep it simple approach. Id much rather /etc/hosts stayed the way it is rather than becoming xml, json or binary.

The whole idea of "Power" shell is broken if you ask me. Shell should be simple. If it isnt, write some proper code. And do some code review, and testing first!

Shell stopped being simple long long ago, at the latest when bash and ksh were introduced. (Re)writing the base utilities has happened many times, and is quite doable.

Yes, backward compatibility is important; that’s why I think adding a new named stream could be the way to go, since it wouldn’t break almost anything (I guess someone might have to reroute something, if it happens to be on the fd that is chosen for the link).

I honestly don’t know how much work it would entail from OS devs, but once you have it in Linux and BSDs/Mac that’s basically it. Microsoft could take the chance to be an early adopter, considering they already have all the machinery to output structured data in PS; at that point, whether it’s pure posix or not would matter little in practice.

I don’t particularly like PS for a number of reasons, but I do admit that its output facilities can be quite nice. Having something like that in Bash for the few times one really needs it, with standard interfaces and without having to learn a completely new paradigm, would be a nice win imho.

This doesn’t mean we should touch /etc/hosts or anything, btw.

Could you share more the things you don't like about PS?

I know quite a few places where shell scripts would probably blow up if some arbitrary FD number >2 would be used... Opening a file in bash (and I suppose other shells too) is done by specifying the FD number explicitly:

  exec 3<> somefile.txt
That might trigger some very weird behaviour when piping data from a 'stddata' aware tool into shell scripts - something which is guaranteed to happen somewhere.

The problem with adding a new standard fd is that lots and lots of code relies on first three fds being stdin out err. It could of course have a high fd number. Would be a great thing!

I am not sure how long "ls" will take with 1 million entries in directory, given this paradigm. This has been a problem so I use "ls -1" to list files in bash shell, otherwise it takes forever.

Probably it might benefit some. Good to see competition in this area with sh, bash, zsh, fish, conch and many others.

A csv output with one field is basically the same amount of data, but you would be guaranteed that the result can be ingested by a csv parser following the “stddata” dialect. It wouldn’t hurt your usecase, and it would make life easier for other ones.

Why not add a new parameter, --nushell, to any too that supports the 2-dimensional format? These tools would just be stored in nushell's config file and the shell would silently add --nutshell to any command it ran that was registered to support it.

The compelling idea here is that they convert the output of common shell commands into tabular data that can be manipulated using common operators, so that you don't have the remember sorting/filtering/grouping flags that may be different for every different shell command. So, imagine being able to use the same sorting/filtering logic on the output of `ls` as you might on the output of `ps`, and without relying on hacky solutions like pipelines to `grep`, `cut`, and `sort`.

It also means shell command output can be easily transposed into JSON and CSV. Actually pretty clever!

I've been delving into bash scripting a bit more than I'd like as of late and the lack of universally available consistent structured output for the CLI really got to me. Most of the script contents end up being these obfuscating and brittle "hacky solutions" that never should have been necessary. When I thought about pursuing fixing this I felt like the task was a bit overwhelming. I'm delighted that these developers are working on this!

The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines, necessitating all manner of argument escaping and field delimiters, when pipelines should be streams of typed messages that encapsulate data in typed fields. Logs especially (logging to files is a terrible idea, because it creates log rotation headaches and each program requires a log parser because of the loss of structured information) also. Line-oriented pipeline processing is fundamentally too simple. Settling on a common data simple/universal format robust enough for all purposes, including very large data sets, to exchange between programs and is just complicated enough to eliminate escapement and delimiter headaches without throwing away flexibility (by a new/refined set of processing tools/commands) is key.

It's not so much a flaw as a tradeoff. Because it's so easy to produce lines of text, virtually every program supports it, even programs written by hackers who aren't interested in following OS communication standards.

No, that's your opinion, not mine. It's a flaw because it creates escaping magic, delimiters that are in-band (spaces) and every program has to know how to parse the output of every other program, wasting human time and processing time. Structured communication, say like an open standard for a schema-less, self-describing, flexible de/serialization like protobufs/msgpack, would be far superior, more usable, more efficient and simple but no too simple to process streams of data with structure and programmability already there.

Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.

So, if you're not a fan of the UNIX philosophy, maybe check out Powershell. Or take a look at WMI and DCOM in Windows. Eschew shell scripts in favor of strongly-typed programs that process strongly-typed binary formats, or XML, or whatever. The alternatives are out there.

"Worse is better" isn't for everyone.

Nushell does not seem like a violation of the unix philosophy, or at least the version of it that I like best.

"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."

Perhaps I'm wrong, isn't nushell simply adding one more dimension?

Instead of a one-dimensional array of lines, the standard is now... a two-dimensional array of tabular data. Perhaps it is not strictly a "text stream" but this does not seem to me to be a violation of the spirit of the message.

Simple line-based text streams are clearly inadequate IMO. 99.99% of all Unix programs output multiple data fields on one line, and to do anything useful at all with them in a programmatic fashion you wind up needing to hack together some way of parsing out those multiple bits of data on every line.

    maybe check out Powershell
I left Windows right around the time PS became popular, so I never really worked with it.

It seems like overkill for most/all things I'd ever want to do. Powershell objects seem very very powerful, but they seemed like too much.

Nushell seems like a nice compromise. Avoids the overkill functionality of Powershell.

Semantically, a one-dimensional list/array of “things” can be trivially transposed into a series of lines with a linebreak separator, but I don’t think the same holds true for a “list of lists” (2-dimensional data) or a “list of lists of lists” (3D data) etc. At least without a standard begin-end list delimiter that allows nesting.

Just thinking about a way that perhaps an old tool can be “wrapped” to be tricked into working with 2+-dimensional data by somehow divvying up the 2+ dimension input data into concurrent 1-dimensional streams, but this seems to require a way to represent more than 1 dimension of data without breaking existing utilities (unless there was, like, a wrapper/unwrapper layer that handled this...)

It’s worth noting that Powershell is available on Linux. Objects are pretty cool. https://docs.microsoft.com/en-us/powershell/scripting/instal...

It works pretty well on the Mac now, too.

"Worse is better" is not some absolute truth or philosophy and *NIX has won: Android, MacOS, WSL.

There's no real alternative for many profesionals, if they want to be employable.

Maybe we should accept this fact and try to make everyone's life easier than be stuck up and push people away.

There's no law that says that Unix tools can't be extended.

And heck, basic POSIX tools violate so many Unix principles...

When perl came along it killed the 'Unix one tool philosophy' dead as a doornail. And since then and people have just kinda ignored the smell coming off the rotting corpse.

I don't write complex scripts in shell anymore because it's insanity. But ad-hoc loops and crap like that... hell yeah. At least a few a day. Sometimes dozens.

People need to be reminded, I think, that shell isn't a programming language first. It's a user interface. And when I look at Powershell scripts and other things of that nature and think about living in Powershell day in and day out I don't see the big pay-off over something like fish.

'Is this going to make it so I can get my work done faster?'

'Is this going to be more pleasant to use as a primary interface for a OS?'

When I go into fish or zsh and use curl to grab json and 'jq' to get a idea of how to use the API in python or node...

versus running 'curl' in powershell against some random API I have never used before..

I get the distinct impression that 'This is fucking garbage' in that It would take me a lot longer to figure out how to use powershell in a useful way then the time I would save by doing so in the long run.

Yeah, I tried using Powershell as my shell and that's when I found out Powershell is more about the scripting language than an optimized shell used for everyday use. I was confronted with this almost immediately because one of the things I rely on most in Bash is 'dirs -v', pushd and popd. I have dirs aliased to dirs -v and I usually have 15-20 paths on my stack. I'll leave implementing the same functionality in Powershell as a user exercise.

Im confused... Why doesn't Push-Location (aliased to pushd by default), Pop-Location (aliased to popd by default), and Get-Location -Stack (trivially aliasable to dirs) not work? You can even have multiple location stacks and, if you care about the registry, you can also use the same commands to handle registry navigation as well.

‘dirs -v’ shows an enumerated list of directories. If I follow that up with pushd +11 I will be taken to the 11th path listed in the prior dirs command. As far as I know this isn’t implemented out of the box in PS

An addition to `{Push,Pop}-Location` - `Get-Location -Stack` will show the current directory stack. I don't know offhand of a way to get the equivalent of `pushd +11` though.

Not out of the box but I wrote a module[0] that gives you `cd+ 11` (and `dirs -v`).

[0] https://github.com/nickcox/cd-extras#navigate-even-faster

The irony is that the very attempt to be one tool for everything caused Perl's own destruction. Perl 5 is still used by some veterans for small scripts but who wants to use Perl 6?

Unix follows the KISS principle, and that is key for success. Albert Einstein said: "Keep things as simple as possible but not too simple". In that sense Unix and Posix are well done. However, that doesn't mean that good ideas like Nushell are not welcome.

I think the failure of Perl 6 was caused by a lack of community building and implementation leadership, not by trying to be too many things at once.

Sounds like the CLI tools, GNU and otherwise, could benefit from some kind of "--format" switch to define the output in detail. I mean something like ls --format "{ \"filename\": \"%{filename}\", \"size\": %{size} }" for a JSON-style output (or whatever the user wanted).

Or even something like --format-rfc4627 filename,size,permissions-numeric to get only those fields out in a valid JSON format.

This wouldn't remove the "every program has to know how to parse the output of every other program", but I am not convinced it is needed. For instance, how would e.g. grep know what field you want to process? And does e.g. "size" carry universally the same meaning and semantics in all programs there is and can ever be? Ditto for "permissions". And what about "blacklist".

As a completely fictious toy example:

  (ls --format-rfc4627 filename,size,permissions-numeric | json-shunt -p "filename" --cmd grep "MegaCorp") | json-print -p filename,size
The fictious "json-shunt" (for lack of better name) would pass only that input-parameter to its --cmd as an input, in this case grep command, the | part would be done for things for which the grep matched, but with all the other parameters intact. So it'd print the filenames and sizes of filenames with case-sensitive "MegaCorp", and output it in JSON.

Yes, I know there are more concise and elegant ways to express the same thing of printing out file sizes and filenames of matching filenames... Also, when scripting pipelines the verbosity doesn't matter, IMO it'd actually be a benefit to be able to read the script.

Edit: fix pre-caffeine wonky redirection and brain fart

What about JSON vs protobufs? Is there a schemaless system that you use?

If GNU decided tomorrow that all utilities need to have a --json output argument then that would make me a very happy person.

Better than nothing, but the problem with that is that you can't construct a deep, lazily evaluated pipeline. JSON can't be outputted until all the data is collected.

There's a streaming format which is used for this: JSON Lines [1] AKA newline-delimited JSON [2].

[1] http://jsonlines.org/

[2] http://ndjson.org/

That still limits us too much due to the rather severely limited available data types (although arbitrary types could be serialized into binary/string data, I guess...)

Funny that you should mention this. I just hit that problem yesterday. The lack of a binary type is a problem. This is the same thing that hit XML. Unfortunately (or fortunately), the best solution is still the MIME stuff.

AFAIK protobufs are not schema-less.

Yeah, those sentences are too close together.

I'm trying to elicit more of a response from the comment author. I think they make some good points. I would like to learn more about their ideal system.

I was just looking at MessagePack yesterday (researching best ways to encode an event stream) and was very impressed.

Yes, I agree that it's a tradeoff. Although that tradeoff was made when memory was sparse and expensive, a modern shell should certainly be more structured - e.g. like PowerShell or Nushell here.

I can understand your frustration.

However, the unstructuring of data and keeping a single form of communication is what has kept UNIX around for so long and why Linux hasn't collapsed under its own weight long ago.

I was voicing your opinions exactly when I was new to the professional computing world. Over time I saw a lot of structured data schemes come and go and they all fall down with this: inflexibility and improper implementations.

Where do you think does the structure inside the data come from? You need to come up with a standard. Now you need everyone to stick to that standard. You need to parse the entire data or at least most of it to get to the information you need, finding new security footguns on the way. Soon you will realise you need to extend the standard. Now you have not only conflicting implementations (because noone ever gets them completely right) but conflicting versions.

And this needs to be done for every single fucking little filter and every single language they are written in.

Take a look at the various implementations of DER and ASN.1. The standard seems simple at first glance, but I haven't seen a single implementation that wasn't wrong, incomplete or buggy. Most of them are all of that. And DER is a very old standard that people should have understood in the meantime.

In order to get at least a bit of sanity in all of this, you need a central source of The Truth wrt. your standard. Who is that going to be for Linux and the BSDs and for macOS? Linus? Microsoft? Apple? The ISO standards board? You?

And all of this is equally true for log data.

I'm okay with tossing backwards compatibility over board. But not in favour of the horrible nightmare some people here seem to propose.

It was a decision that lead to success. Before Unix most filesystems were record oriented. Unix's philosophy of treating everything as a stream of char was different and very successful.

> The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines

not so sure about your assertion there eugene :o)

interfacing with funk formatted lines of text is waay more easier because (imho) of the 'wysiayg' principle, and encourages, for lack of a better term, 'tinkering'

> Settling on a common data simple/universal format robust enough for all purposes...

output from 'ls, gcc, yacc/bison, lex/flex, grep, ...' should all follow the same fmt, sure...the good thing about standards is that we don't have enough of them to choose from, f.e. xml, sgml, s-expressions, json, csv, yaml, ...

having said that, when dealing with copious volumes of data, structure doesn't hurt, but in such cases, my guess is, there are only a handful of data-producers, and data-consumers for automatic consumption, and are all tightly controlled.

I used to think this, and am still sympathetic to the motivation behind this. But I've since changed my mind when I saw the challenges faced in Big Data environments. Structured data is nice at first when you get into it. But the need for data prep stubbornly will not go away. If only because how your stakeholders want the data structured only sometimes aligns with the pre-defined structure. And when you get those numerous requests for a different structure, you're right back to data munging, and wrangling lines of text.

In my limited experience, you're either taking the cognitive hit to learn how to grapple with line-oriented text wrangling, or taking the cognitive hit to learn how to grapple with line-oriented text wrangling AND on top of that a query language to fish out what you want from the structure first before passing it to the text wrangling. I'd sure like a general solution though, if you have a way out of this thicket.

I don't think I'd go as far as typed messages as that doesn't strike me as a simple or light-weight format. Something like JSON is almost good enough, but perhaps something even simpler/lighter would be ideal here.

JSON should have included comments from day 1. Big oops.

Comments are great for human-edited config files, but not a clear win for an stdout/in format.

The trouble is, comments are normally defined as having no effect, not part of the data model. But iff the comments contain useful information, how do you pick it out using the next tool? Imagine having to extend `jq` with operators to select "comment on line before key Foo" etc... And if it is extractable, are they still comments?

How do you even preserve comments from input to output, in tools like grep, sort, etc.

XML did make comments part of its "dataset". That is, conforming parsers expose them as part of the data. Similarly for whitespace (though it's only meaningful in rare cases like pre tag but that's up to the tool interpreting data), and some other things that "should not matter" like abbreviated prefixes used for XML namespaces). This does allow round-tripping, but complicates all tools, most importantly by disallowing assumptions that some aspect never matters, e.g. that's it's safe re-indent.

I'd argue the depest reason JSON won over XML is that XML's dataset was so damn complicated.

It used to. They were removed later

How should logging be done?

Nothing prevents the output from having more than 1 stream (stdout, stderr, etc), or to use either for logging.

> /dev/null

is "destructuring deserialization and reserialization based on lines" really an aspect of the UNIX philosophy or just an implementation detail? I thought it was more about doing one thing and doing it well [1]. It could be argued that nushell follows the UNIX philosophy.

[1] https://en.wikipedia.org/wiki/Unix_philosophy

“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”

Doug McIlroy (2003). The Art of Unix Programming: Basics of the Unix Philosophy

That said, I’m looking forward to testing our nushell!

You can also just use PowerShell. Its chief crime is its verbosity in naming, but it has the "universally available consistent structured output for the CLI" you're asking for.

I spent a lot of time in Powershell on Windows. Almost 3-4 years. I simply could not use it as a shell.

I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable.

It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else.

Bash felt far more comfortable as a daily shell because what you see is what you get.

I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.

PowerShell is more like interactive Python with very smart tab completion than "bash with objects." Basic conveniences like "grep" require some memorized scripting gymnastics because the commands got generalized a little too much.

For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid.

The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.

This is not true. If you want, you can literally pipe PowerShell output into grep (or select-string, which does the same) and it will filter by the tabular representation of each row. That can work just fine in interactive use cases and while it is less accurate and performant, it is the same bash.

The key to success in using PowerShell as an interactive shell is to memorize aliases and not caring about casing and spelling everything out. PowerShell is case-insensitive, has aliases (e.g. ? instead of where-object, or % foo to get the foo property of every object) and will accept abbreviated command and parameter names if the abbreviation is unambiguous. It is also useful to know about the different formatting cmdlets: mostly format-table (ft), format-list (fl), and that you can pick properties with -p (-Property). While PowerShell usually does the right thing, with these you can get the most appropriate formatting for any use case.

(Disclaimer: never used powershell, so unadulterated opinion here)

Scripting is programming, so you want data structures. For interaction, you want human-readable syntax.

e.g. programming languages mostly have a syntax: C, python, bash, fortran (exception: lisp), but they operate on data structures.

We can see powershell, nushell and other structured shells (e.g. some use XML) as an attempt to "get out of the syntax business", which XML began and JSON finished (https://blog.mlab.com/2011/03/why-is-json-so-popular-develop...).

This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.

> It's not clear to me why

I commented on this earlier but the defectiveness of PS as a shell hit me almost immediately when I tried to recreate my workflow of 'dirs -v', pushd and popd with a large number of paths on the stack.

The chief crime of PowerShell is actually how long it takes for a prompt to appear after starting it - especially on Windows. I still use it as a login shell on MacOS because it offers useful features, but it's (to put it bluntly) slow as hell to start.

Is it actually useful outside Windows? I know the latest version is now portable because of .net core, but I haven't tried it on *nix.

Yeah I'm using it as my primary shell on macOS. You can do everything you do with bash, but with objects, autocompletion etc

The only thing I cant find in PS is fishlike predictability when using it as a shell.

Fascinating. Is there a command for getting processes for example or do you need to use ps?

Naw its chief crime is not having `&&`

Im speaking from a position of naivete, so dont attack me;

Would an ideal situation be “do + while”

Whereas one may write

While x is true do “blah blah” && if output is still true then end elif do &&?

Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))

There are ways around it but nothing as easy as `&&` in other shells

pwsh 7 is adding it

I've generally found you can get most of what you need done with head, tail, grep, cut, paste, sed and tr (and maybe a couple others I'm missing). Oh, and test (both bash's and coreutil's). It can be a little hacky, but once you realize the crazy output you sometimes have to parse from third party programs, you realize that the ability to handle any free form text is pretty essential.

That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).

I agree. The idea of Mario is to use Python objects instead of text lines. This means it's much more structured and has a familiar syntax as well.


Thats when you switch to a real programming language. Relying on output from other programs will always be brittle - as error messages etc will change every new dist release.

Yeah just iterating through a directory with filenames with spaces..

I think they took one step in the right direction, where two more were needed.

The UNIX idea of piping text output from one command to the next was fine at the time, but it means that the downstream command has to parse text. With nushell, you are parsing slightly more structured text.

The two steps they could have taken in addition:

1. Instead of generating rows of a table, why not lists of strings? E.g. ("abc", "def", 123) instead of "abc | def | 123". Much easier to deal with in the command receiving this input.

2. And instead of just lists, why not objects? E.g. a process object, a file object, or, if you'd like a list as above.

I have implemented this idea: https://github.com/geophile/osh. For example, if you want to find processes in state 'S' with more than 3 descendents, and print the pid and command line of each, sorted by pid:

    osh ps ^ select 'p: p.state == "S" and len(p.descendents) > 3' ^ f 'p: (p.pid, p.commandline)'  ^ sort $
^ is used to denote piping. ps yields Process objects. Select examines those processes and keeps those with the right characteristics. Then f applies a function, mapping to the the pid and commandline in a list, and then sort sorts the lists, by pid (the first field of each list). $ does output.

The "abc | def | 123" is just a textual display of the data, not the data itself. Each row of the table is a structured record, like you suggest.

I think in nushell and powershell you are parsing objects.

'where' in pwsh / nush literally looks for keys with particular values.

I agree objects are way better than text for manipulating structured data. Python is great at this, so I use Mario to do it in the shell. https://github.com/python-mario/mario

For example, to sort filenames by length

    $ ls | mario apply 'sorted(x, key=len)' chain

Seems its only one level deep, no tables of tables are shown. Powershell has a lot of issues but one cool thing is you can access output as nested object data. Seems like the logical evolution.

You can have tables of tables. To get to the inner table data, you can use the 'get' command to retrieve it and work with it.

They have an example in their video.

This is pretty much what I've idealized for a new shell. Where every program can just output a standard table of information for each action. No need to have the program itself be queried for it every time, or find all the right options. You just get all the data, and filter what you want programaticly.

> You just get all the data, and filter what you want programaticly.

There are definitely cases when you're better off with producer-side filtering. For example, `find` allows you to not traverse certain directories with -prune, which might save a lot of resources.

PowerShell has a lot of "corrections" of this kind, to get producer-side filtering. e.g. Getting all Windows Eventlogs, then filtering client side turned out to be quite bad, so Get-EventLog became Get-WinEvent with a lot more filtering controls.

Cmdlets which get data, like Get-AdUser or Get-MessageTrackingLog in Exchange, or Get-CimInstance often have a lot of common filtering parameters as well as a more general -Filter or -Query "build your own".

Big source of annoyance where you can't do that, but might want to.

It also generally applies to concept of wildcards/globs not just to file paths, but many parameters. So a cmdlet like Get-User with a -Name parameter to get a user with a certain name can also be called with John* to get all users with first name John. In my experience the filtering options are pretty consistent

Do they actually convert the output of the existing commands? Or are they reimplementing then one by one? It looks like the latter.

In their examples it looks like 'ls' is built-in to their shell instead of from e.g. coreutils.

I love the core idea of nushell, but reimplementing them seems rather insane, given the incalcuable number of programmer-hours that have gone into coreutils.

That seems like a real mistake, rather than simply having nushell's "ls" be a wrapper for coreutils "ls -la" or some such.

I understand the benefit of reimplementing everything from scratch, as that way you have a more consistent nushell experience everywhere, regardless of which version of "native" ls your system has. And allowing "^ls" is a useful and necessary hedge.

But, wow reimplementing nearly everything seems like an enormous undertaking.

(It is of course possible that I'm completely misunderstanding things! Perhaps the devs can comment?)

Check the ls man page and remove anything related to sorting, filtering, or formatting information (as under this system all this functionality is provided in a shared and generic way). There is not much left.

Coreutils has been reimplemented N times: BSD, GNU, BusyBox, the Rust one, countless others I'm forgetting. It's doable.

ls is a shell builtin. When he types ^ls he uses the one from coreutils.

This idea is already preset in xonsh

How do you think it compares to xonsh? I ask because I had been thinking recently about investing some effort into xonsh and now nushell has appeared. I’d be very curious what any serious xonsh users think about this newcomer.

There're already numerous ways to do this without rewriting the world. For instance


> So, imagine being able to use the same sorting/filtering logic on the output of `ls` as you might on the output of `ps`, and without relying on hacky solutions like pipelines to `grep`, `cut`, and `sort`

That would awksome.

Seems similar to the purpose of `osquery` which is SQL based.

This is almost exactly what I have dreamed with a shell. I'm building a relational language (http://tablam.org) with the plan of something alike.

Some extra ideas:

- Tables are amazing. But I think we need at least support for this "shapes" of data: List, Tables, Trees and maybe graphs

- I think will be nice to have an equivalent of Request/responses alike http. And then a way to introspect what a command do, like auto-swagger.

- Having widgets like table-browser, graphs, etc AS COMPONENTS.

In FoxPro you could say "browse" and it display a DataTable grid with full editing capabilities on the current table (or it ask for a table to open). In this case:

    ls | ui_browse
The key here is that the widgets work stand-alone, so is not necesary to fully build a terminal app just to see nicely data. Look like https://goaccess.io and more like this idea: https://github.com/sqshq/sampler

This mean we could get a jupyter-like experience here!

- Some way to do curryng and maybe environment alias: You could setup an environment for the system, the user, this directory. In the environment is a .toml with vars and stuff.

The other thing where I think shells fail... is that are still old-style terminal. Having text output is cool, but having more rich widget system will unlock better the potential

"The other thing where I think shells fail... is that are still old-style terminal. Having text output is cool, but having more rich widget system will unlock better the potential"

You might find DomTerm (http://domterm.org) interesting. It combines solid traditional-terminal support (including solid xterm compatibility) with a data model based on HTML/DOM. So you can "print" images, HTML tables, and more. DomTerm "understands" shell/repl commands as consisting of prompts, inputs, and outputs. You can add markers to group the output into logical groups, so it is automatically re-formatted on window re-size. And more.

Like PowerShell's different output formats (table, list, wide, ...)?

No output. Declared the input/output is in the shape table, list, tree.

Like when a http response say is json, text, csv, etc. This information unlock certain capabilities and will allow to provide efficient execution of queries.

If psql (postgresql terminal cli) declare their data is in the "tables" shape then it could take the |where name = "" query itself and execute it instead of output all the rows...

There's a lot of comments here already so I don't know if this has been addressed already, but I don't get this shell. It seems to be fixing a problem that doesn't exist, and a lot of comments are echoing it. Current shells are not deficient, as Doug McIlroy originally stated, the universal interface is text and tools should take input and generate output based on this. You obviously can then pipe things together with these tools not necessarily knowing in advance how they will be used in a pipeline. You can get very far with sed, awk, etc, you just have to spend a little bit of time learning them, but they are not complicated. You can of course then introduce a command like jq for specialised purposes. But hte point is the shell isn't providing this, they are separate tools that you orchestrate using the shell. There is not a flaw with how it works at the moment, I truly believe and this isn't a dig at this project or anyone in particular, that people do not spend enough time learning these tools that have been there for decades. The re-inventions that appear on GitHub are not a product of the deficiency of the existing tools, I don't know what they are but it's not that. Unless I am missing something.

You state the problem yourself

> [...] you just have to spend a little bit of time learning them

I can still remember the many times I tried to learn shell scripting and gave up too quickly. So 'little bit of time' might be an understatement. Actually, you need a good use-case, sufficient motivation and enough time at hand to bite through the first problems...

And show me a serious, modern programming language that is so susceptible to whitespace usage. Yes, there are some that care about whitespace (e.g. Python), but rarely it will cause a syntax error to not place a space before a bracket.

Another problem is that those text formats are a bit brittle. I am talking about having to use IFS="" and ls giving back something different to humans than to pipes. Most of the time you can solve it one way or the other but having to search frequently for solutions to such problems just sucks.

I am not saying that the old tools aren't worth learning. In fact, I love writing shell scripts (for the good parts) and can't remember the last day I haven't used a shell. But saying there are no problems to be solved is just wrong. So I think a little discussion about what could be done and some prototyping shouldn't hurt anybody.

I've now read a lot more of the comments, people are really excited about this, it has taken me a bit by surprise. There are a lot of comments in the vein of "bash sucks but it's everywhere". My dream shell is pdksh, which already exists, I would only favour bash because it has more advanced completion logic. Aside from interactive use I write everything in POSIX shell. I think popularity is being conflated with portability and that's a property that isn't valued enough. I have no problem at all using Bourne-compatible shells, I can't quite formulate it in my mind but I think they are probably misunderstood. To me, who hasn't been doing this too long, I don't think that bash sucks, I don't find bash/POSIX shell to be a bad language. In fact in terms of different things I write (not LOC) I think I write more things in POSIX shell than anything else, writing wrappers in shell is not to account for a deficiency, it's utilising that power to change your shell experience and workflow into what you want, into what fits your brain. This is my experience, I don't understand (it would be good to hear from others) why people have such a bad opinion of the current state of shells.

EDIT: wording.

When PowerShell first came out, I was very excited by the idea and then barely even gave it a chance. Instead I gained an appreciation for input and output being just text. Yeah, I might have to awk’wardly massage the data a lot of the time, but the fact that I can just run a command and see the output makes the flow of incrementally solving a problem really nice.

I’m gonna give nushell a chance, because tabular textual data seems like it still fits the bill while also making it less awk’ward to massage the data as needed for the pipeline.

I too don't understand the excitement these object-oriented shells elicit in some circles. It's exactly like you say, the job of a shell is to manage jobs and run programs. The only data processing that it performs is the processing of the command line arguments and nothing else, as far as I can tell.

You are missing something. Current shells are brittle and they're just bad tech in 2019. We've made them half decent by brute forcing tools and hacks for them.

Everyone has to reinvent the same basic tool output parsing because ls (or top or ps or...) can't spit out real structured data.

I disagree. I'm not saying that everyone should think the way I do, I guess realistically all I mean is that when I'm confronted with a parsing issue in this context I just write a shell script. Ok, this can be taken as reinventing, which I'm advocating against, but I see this as utilising the UNIX philosophy and by sticking to POSIX shell specifically (and using sed, awk, tr, et al) it is still a built-in and portable way to deal with this problem. Also I am not saying this because of some irrational submission to an old philosophy, I advocate this style of working because I find it easy, convenient and fast, and it works for me in the real world. But of course people have different ways of working and I respect that. This is just what works for me and how I see things. Definitely not meant as an attack against this project or different ways of working.

Why the hell was this downvoted?

yes, I have a feeling that Windows people are coming to Linux through "Windows Subsystem for Linux" to reinvent almost 50 years old concept which have been working well. Instead of using small bash which is everywhere (even in inird), more convenient zsh or Python, they are going to setup Rust environment, download gigabytes of crate dependencies and build a new shell in order to run their web development workflow script. Maybe in the 10 years, there will be Nodejs/Electron-based shell with fancy css animations and primarily mouse control. And tables will be sortable by a mouse click on a column header and searchable in real-time and there will be speech commands for common tasks...

The point of this project is standardized inputs and outputs, that's all. Working with commands is not standardized on traditional posix systems, in terms of either input or output.

The other comments about adding a "stddata" and handing this responsibility to userspace is a good one.

You shouldn't need to know a specialized DSL for each tiny core util.

One downvote and counting. This is just my opinion. Let's crank this up a notch. Those advocating for more advanced interfaces are missing the point. Text is all you need, no graphics or objects or widgets, just text. I have no problem with this, I enjoy it like this, we don't need an improvement. There I said it, any downvotes will be worth it, because I get to voice my opinion. In the end I will still be happily using my POSIX shell scripts, awk, sed and enjoy my life.

"Goto is all you need, no ifs or loops or dynamic dispatch, just jumps. I have no problem with this, I enjoy it like this, we don't need an improvement. There I said it, any downvotes will be worth it, because I get to voice my opinion. In the end I will still be happily using my assembler and enjoy my life."

I'm sorry, I couldn't help myself. Your comment reminds me of an anecdote I heard from the early times of structured programming. When structured programming was just gaining its feet, there was a certain class of programmers who just could not understand why people would want to write structured code. You can do everything in assembly they said, you have much more control over performance, etc. They looked down on structured programming as not "real programming".

There's a lot of benefits to adding some structure to text. I don't think that Nushell's approach is the best one, but to say that there are no problems and we shouldn't look to improve things is just backwards. We should always look to improve our tools and our craft, otherwise we would still be stuck writing assembly.

There's a lot of benefits to adding some structure to text.

There are benefits, true, but there are also potentially serious drawbacks. Chief among them, I would hazard, is the risk that we get locked into a format that didn't anticipate a (completely unknown) future need and we have to go back and rewrite everything again.

The beauty of text's lack of structure is that people are free to interpret it any way they please.

I keep seeing comments like this one, but just as you can parse textual data into something more structured, like the examples in the post, you can also print the structured data as text. So the tools that use and generate structured data are no less compatible because it's easy to go back and forth between the formats at will. You're not losing anything with this approach.

> You're not losing anything with this approach.

True, but you are adding something. And if you add things you don't really need, it becomes plain old cruft.

Like the others above, I understand the urge to make some things simpler by adding complexity, but for the long haul I'm convinced it's better to keep tools simple and add any needed complexity to the code you're writing (whether it's shell scripts or anything else). And then document the complexity so future you or some poor stranger can understand why the hell you did that in the first place.

But I don't look forward to reading comments like "X was deprecated so we're converting these tables back to text streams so they can be converted back to Y properly."

But isn't that precisely the problem? A lot of people are unsatisfied with the current solution which doesn't lend itself to a particular goal?

Aren't we now locked into a format (plain text) which is now becoming more and more of a pain (what type is this? is this text related to this other text?) and we're now having to go back to old utilities and add --json?

I get it, but to clarify, and maybe it's to do with the work I do, I have no problem with current shells, I think they are great. As I learn more about them and use different features I like them more. This is my experience.

Maybe you're just not very imaginative. I can see my workflow improve considerably by using a structured-io shell, and I'm already quite proficient in bash. So, I'm very happy some people are trying to make it happen.

Also, I'm not sure what your argument is, except "I don't get it"? It's fine, lots of people don't get stuff. Just move on, and maybe in a few years when it's mature, you'll see it again and go "Ah!"

Also goto being considered bad is already a very old idea. Yet it is still used, sparingly in the right places. Shell underpins a lot of stuff, it is tried and tested. Text is already structured, or can be, IFS, new lines, etc.

"Text" is a useless specification when building a program that produces data that other programs consume, or consumes data that other programs produce. There is nothing actionable in knowing that you have text to work on other that you can presumably discount the appearance of nulls in it.

I find it basically impossible to compose programs where I have things like a list of filenames with spaces. It turns out some programs support some kind of option to separate with nulls, but they are inconsistent and not everything supports it.

So I end up having to put everything in one giant call to 'find', which defeats the compositionality of shells.

Null-terminated mode: turns out tons of tools have these: https://github.com/fish-shell/fish-shell/issues/3164#issueco... But yeah, 2nd "standard" hurts compositionality.

Default posix/bash splitting of output by spaces is very big-prone and having to type "$array[@]" all over the place is annoying. But splitting by lines is much saner. Now newlines in file names, that's just evil and should be outlawed at OS level. I think OSX did this (?)

In bash life can be easier by setting IFS (Google "bash strict mode" though I have reservations about the IFS part).

But really, its easy to fix in a non-posix shell. Fish splits by lines and interpolates variables as arrays by default.

Text is all you need

In essence, yes. Yes you can get pretty far with sed and the likes. But it's not because it's all we really need, that other things (objects in this case) don't offer an improvement and make things better/easier/more convenient. Just like we theoretically could get along fine with just the CLI, doesn't mean that it is the single best thing in the world for all possible occasions.

True, but also when using shell for scripting, when you want to use an associative array or some other "advanced" feature, from a shell point of view, well then you should move on to a different language. So in the same sense in such cases you should move on to a different language rather than trying to get the shell to do all things for all people. I am a very strong proponent of shells and shell scripting but realistically it's only designed to do certain things, those things it's perfect for, for some things it's bad for, use something else.

Text is awful and annoying to deal with.

A brand new shell, especially in Rust is a great thing to see and also how it tackles the problems that exist in other shells.

However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked) and as shells have always been portable across many operating systems, I wonder how portable nushell would be since it wishes to replace my current shell.

I think the JavaScript/npm world has trained a generation of devs that high dependency counts are a code smell. This can be true, of course, but that's largely because of external factors.

In a perfect world, a language's ecosystem would have a tiny stdlib and allow importing of libraries for everything else. The Linux philosophy would be followed strictly.

The problem is just that the overhead in securing, maintaining, and organizing all those libraries is pretty large, as we've seen by npm's repeated failures. Of course the *nix community seems to have largely solved the problem, but there's also a massive culture of volunteerism benefiting them.

I think there’s definitely room for mirrors of package repos that focus on a stablized and audited subset of packages, with frequent security and bug backports and medium-length cycles for new features — like what Fedora and Debian do, but for language ecosystems.

Agreed. ActiveState has had that as a commercial offering for quite some time (https://www.activestate.com/products/) and Red Hat is getting into it with (OpenShift) Application Runtimes. Given that providing such a thing is hard not-so-fun work I have a hard time seeing it getting available for free. (ASP).NET Core might be alone in being well-financed and free, something I think will lure (enterprise software) developers over long-term.

I believe Deno is doing this.

For a FOSS language, a large stdlib can be similar to a curated repo in practice.

UNIX philosophy. You mean the UNIX philosophy.

>…the Linux philosophy is "laugh in the face of danger". Oops. Wrong one. "Do it yourself". That's it.

- torvalds

I did. Thank you for the correction. Linux did, for most of its history, do an admirable job of upholding that philosophy though.

Care to elaborate? Linux is a huge monolithic kernel, isn't this the exact opposite of the UNIX philosophy?

The ecosystem, not the kernel. Linux packages are relatively good about doing one small thing well.

It is safe to say that it is unrealistic to expect that developers in the real world audit their dependencies.

I'd imagine a feasible solution being a permission-based system (which might already exist), where programs and their dependencies do not have access to certain facilities, like the filesystem, other processes or the network, without them being declared in a manifest file of sorts. Permissions should be inheritable when explicitly specified, and a sub-dependency could not gain a permission it didn't inherit from its parent. Unfortunately this does not work so well with a shell, since the shell needs the ability to spawn arbitrary processes. At least the shell itself would not have network access, forcing it to rely on tools already installed in the system

Also, if we resort to some magical wishful thinking, then all the tools in the system follow this permission model and the package manager is aware of the permissions. You could then query the package manager for all installed software with a certain capability, like network access, disabling tools like curl and netcat.

The key difference between typical GNU/Linux distributions and the npm/cargo model is that the set of packages available in the distributions is to some extent curated.

I hope that I implied that point when discussing the labor required for a highly modular ecosystem to function well.

There are 464 dependencies in the complete tree when doing `cargo install`.

That's a bit too many. But it's not often you compile your shell from scratch.

It's definite a lot. We're looking into making more of the features optional, so you can streamline it if you want.

That is some impressive insight to the behavior of the lib. Have you ran into that before and knew it was an easy thing to get rid of?

I’m always impressed when people show up with fixes like this (that I wouldn’t even know to look for).

I did see quite a few that double up, like 2 versions of url and a few others.

Just had a quick play ... on arch linux `yay nushell-git` already works!!!

One area where I got a bit stuck was around help. `man where` gave me nothing, `help where` also. I tries stuff like `ls | where type = File` and got a type error. I think it would be amazing if this thing onboarded people a bit nicer, "where needs a condition, to learn about where type "help where" ... stuff like that

Overall really enjoying the ideas here and I am absolutely going to be following this!

And 484 when installing with rawkey and clipboard.

> shells have always been portable across many operating systems

Most shells work on just one of either Unix or Windows; the blog post specifically mentions Windows support as being something on the radar of the developers, which Rust is arguably a better fit for than C/C++ (which most shells are written in) due there being a single compiler, standard library, and build tool with first-class support for both Unix and Windows in the Rust toolchain.

Which shells are those? Bash works fine on both Unix and Windows.

Mario gives a lot of the benefits of using objects instead of text, but since it doesn't control the whole shell it's less of a commitment.


The risk of a supply chain attack, roughly speaking, is then multiplied by all 73 dependencies. If any of those are compromised, the shell and system are compromised.

To ameliorate the risk, the security of all 73 dependencies would need to be at least an order of magnitude greater, just to catch up to shells with no dependencies.

The irony is that this lack of "dependency-safety" is far more easily exploitable than any previous lack of "memory-safety".

Of course, this can be easily fixed if developers stop multiplying third-party dependencies, so that importing a dependency involves O(1) risk, not O(?).

> But this can be easily fixed. Developers need to stop using third-party dependencies, so that importing a dependency should involve O(1) risk, not O(N^2).

How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.

I guess that if nushell had a zero dependency policy, it would have never happened.

And there is a significant risk to spreading your resources thin when reimplementing sensitive dependencies, like crypto or network / http libraries.

> How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.

At some point, developers need to take responsibility for the security of their projects. If your project is larger, you will need to maintain more code, whether directly or indirectly. You can't simply import a dependency but never audit it. And if you're taking the trouble to audit a third-party dependency, at some point it becomes worth maintaining directly.

> sensitive dependencies, like crypto or network / http libraries.

I am not suggesting that developers reimplement TCP.

I've had to reinvent the wheel, over and over again, throughout my career. Sometimes it was because our environment was constrained (no cloud, needs to be up for major disasters), or because the architecture was unique (no primary, all protocols must be decentralized), or because what I needed didn't exist for my stack.

It's pain and sorrow all the way down and I find more bugs in my reinvented wheels than in the actu code I set out to write.

How about this:

0. Rebuild from latest stable dependencies, or latest updates to your LTSB, whenever you or your dependencies have a bug.

1. Prefer static linking or lib incorporation if feasible, or use a package manager that properly guarantees you the right versions.

2. Stick to popular repos with responsive maintainers, or

3. Fork the dependency yourself, audit the code and know how to debug it.

4. Watch upstream for security bugs.

Sure, and 5 would be:

Prefer dependencies with smaller flatter dependency trees.

> At some point, developers need to take responsibility for the security of their projects.

Agreed, but that could be through careful selection instead of rolling your own. And it would be great if there was better tooling for that.

Imagine an ecosystem where every module author automatically gets not just a public repo and issue tracker, but also a way to deal with security issues, an indication if all contributors used 2FA for code contribution and package uploads. Then there could be a search for packages where you can say "only give me those with 2FA and a defined security disclosure process, including all transitive dependencies".

That wouldn't be perfect, but much, much better than any ecosystem I've seen so far.

This is sorta why things like Guava and Boost exist - audited / trustworthy libraries for common asks where you know more or less what to expect in terms of quality.

The approach the Rust community is taking with a sort of decoupled-STL approach is interesting, but I do wish they found a way to collect the known-good-bits in a common namespace.

What shells, if any, have no dependencies?

An example: https://www.archlinux.org/packages/core/x86_64/dash/

Or look at bash:


I would say many shells that are written in C will have an extremely short list of dependencies in contrast to Rust.

On my phone, but ldd bash? Also compare cost of maintaining your own custom secure functions Vs using well known ones.

You do not have to maintain your own, you can use a popular and audited one that many other projects depend on so it is typically installed. The point is not to not have any dependencies, just do not have in the hundreds, especially not if they are not shared; it would take up too much space!

As per your request:

        $ ldd /usr/bin/bash

        linux-vdso.so.1 (0x00007ffce8b8c000)
        libreadline.so.8 => /usr/lib/libreadline.so.8 (0x00007fd76a0dd000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fd76a0d8000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fd769f15000)
        libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007fd769ea6000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fd76a266000)

Those files are generally installed on all major Linux distributions.

For completeness:

        $ ldd /usr/bin/dash

        linux-vdso.so.1 (0x00007ffdbadce000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f6f2cd28000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f6f2cf58000)
Note that these files are shared between projects, you do not have to download and compile them per project. Can you do the same using Cargo? According to my experience with it from two years ago, I had to download and build over 400 dependencies for all projects, separately. I do not know how many dependencies Rust projects have on average.

This isn't a problem if versions are pinned.

> However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked)

The problem is that Rust has two issues:

1) In Rust, you don't pay for what you don't use. This means that a lot of stuff tends to be an external library since not using something means you can exclude it COMPLETELY. The issue is that things tend to get atomized more finely than most languages would do.

2) Rust is trying to not bake things into a standard library too early. This is a good comment about standard libraries from Python:

"The standard libraries constitute one of Python’s great features, but they tend to suck up all the oxygen. Programmers are reluctant to write libraries that duplicate their functions, so poor libraries in the standard set persist. Only a few people, like Reitz, are willing to write modules that compete with the standards."


This looks great, but what exactly are the benefits over PowerShell? The article mentions PowerShell, then says:

> What if we could take the ideas of a structured shell and make it more functional? What if it worked on Windows, Linux, and macOS? What if it had great error messages?

- Any shell with piping is already very functional, and PowerShell has script blocks (higher-order functions). Or does this mean "functional" in the literal sense? What functionality is missing? - PowerShell works on macOS, Linux and Windows - The examples of error messages further down look almost exactly like PowerShells error messages (underlining the command in the pipeline that failed)

It is not clear to me what exactly the authors sought out to do different

I wanted to like PowerShell but I find the syntax choices absolutely exhausting. This looks far more comfortable to me even as someone who mostly uses Windows but also knows Unix shell.

I know the long cased commands like Foo-Bar can be alienating to Unix folks at first, but have you tried just using aliases and not caring about casing? PowerShell will accept any unambiguous case-insensitive abbreviation of commands and parameters, and has almost all frequently used commands aliased (see this thread, it's literally the same as nushell).

I often wonder if people just don't like it because all they see is PowerShell scripts, where people like to spell out commands so it's clearer what they do. Imo it's a nightmare that we write bash scripts with all these 2-3 letter commands and flags that no beginner could ever understand. (but if you want to write that way even in scripts, nothing is stopping you)

It's not the name of the commands that bother me; it's literally everything else from the operators, the sigils, to how functions are defined.

There is no excuse to make a new language that re-invents nearly every basic syntactic principle or borrows from the worst examples (bash, perl).

Although the naming isn't great either; the use of namespace prefixes rather than a flat namespace of long names might have been a better (and more logical) choice. It would have even allowed for better auto-complete.

I'm more of a Windows and C# guy, and even I just can't get along with the naming and syntax.

I tried using aliases instead, but there is a danger with this - some of the aliases are commands on Linux. For example, if I use the `ls` alias it will work as expected on Windows, but if you run the script on Linux, it will run the Linux `ls` command, rather than whatever the Powershell alias is for.

> I know the long cased commands like Foo-Bar can be alienating to Unix folks at first, but have you tried just using aliases and not caring about casing.

You'd have to alias every command and every option to make it enjoyable for interactive use.

And of course those aliases would need to be part of the standards powershell.

"PowerShell will accept any unambiguous case-insensitive abbreviation of commands and parameters, and has almost all frequently used commands aliased"

> PowerShell will accept any unambiguous case-insensitive abbreviation of commands and parameters.

Which due to the naming conventions are still too verbose.

That's why I wish this shell could replace both bash and PowerShell in the real life. That's a pity this is highly improbable.

Why not? On your own system you can use whatever shell you want and if you develop shell scripts that you plan to distribute you can just list it as a dependency.

They clearly know about PowerShell. They acknowledge it as an inspiration and the core concept is the same.

That doesn't mean there aren't ways to improve upon PowerShell.

As a light user of PowerShell I consider it a poor implementation of a great idea.

Commands are way too long, control structures have poor syntax etc.

To make this very concrete, take this example from nu:

    ls | where size > 4kb
Now give me an equivalent in PowerShell and then let's talk about how different Nu is from PowerShell.

The equivalent, btw, is:

  ls | where length -gt 4kb
The only difference is powershell uses “>” and “<” for redirection, and so uses mnemonic operators for comparisons.

Nu doesn't seem to support redirection (or even have anything other than one input and one output stream, though metadata might serve the purposes otherwise served by additional streams; it's not how a program would return something Nu would treat as metadata, though.)

I guess that with a structured-data shell, error type of info could just be returned in-band under a different tag, in any case.

Take your pick:

    # most common in script, vs at shell
    Get-ChildItem | Where-Object { $_.Length -gt 4kb }
    gci | where length -gt 4kb

    # longest most restrictive with namespaces, 
    # most stable over time, 
    # least risk of hitting the wrong command 
    # or wrong property as surroundings change
    Microsoft.PowerShell.Management\Get-ChildItem | Microsoft.PowerShell.Core\Where-Object -Property Length -gt 4kb

    # shortest(?)
    gci|? le* -gt 4kb

    # because you can
    gci | & { process{ if($_.Length -gt 4kb) { $_ } } }
    $larges, $smalls = (gci).where({$_.length -gt 4kb}, 'split')
    [linq.enumerable]::where([string[]][io.directory]::EnumerateFiles($pwd), [func[string,bool]]{param($f) [io.fileinfo]::new($f).Length -in 0..4kb})
In Powershell you can tab complete "Length" in the filter, from the available properties of the objects coming out of gci, which is neat.

Complaining about command length when PS explicitly had full fledged named commands (for readability) and default aliases (for terseness) doesn't make sense imo.

  dir |where length -gt 4kb
Let's talk.

PowerShell has some great ideas, but I can think of quite a few things that can be improved in (or over) PowerShell: its abysmal speed, its verbose syntax, verb-noun structure which is much less useful than noun-verb would have been...

Is there any noun-verb shell?

Don't think there's a noun-verb shell, but in PowerShell the deficiency is obvious. Here's an example posted in a later link in this thread:

"Get-Process | Out-GridView -PassThru | Stop-Process"

Now, is it "Stop-Process" or "Set-Process -Status Stop"? "Out-GridView" or, say, "Show-GridView" or "Out-View -Grid"? The other examples are all technically legal, and other modules can and do have such inconsistencies.

With verb-noun, I can't meaningfully complete for the verb when I don't know it, I can complete for the noun if I know part of it. But I usually know the noun, so that's not so useful.

With noun-verb, I could get a meaningful verb completion and still be able to complete nouns - with some duplication of results, though I think there could be a way to have some semi-completion for the noun only. A semi-completion isn't useful in the verb-noun case, since the program can't guess the noun yet and different nouns may have different verb uses.

Last I tried Powershell (on Linux, that is) it was so slow that I simply gave up, frustrated.

I really like the concept, but the slowness ruined the user experience for me.

> This looks great, but what exactly are the benefits over PowerShell?

To my perspective, making parsing for a variety of text-based formats a default action is an advantage, because it means tools don't have to use a particular runtime (e.g., .NET) to feed objects into shell pipelines without the shell user doing explicit conversion.

Maybe I'm misunderstanding you, but isn't this what the ConvertFrom-{Json, Xml} etc are for?

For one Powershell isn't libre as in freedom am I right.

More than enough libre :)

Pretty cool idea and project. Hope you take it further and it turns into something truly spectacular!

My only comment is on the article. This line:

> Nu would not have been possible without Rust.

That's just patently false.

Perhaps you wouldn't have chosen to code it in another language. Or perhaps Rust had one or two benefits that assisted in the development of Nu. But you don't word it that way or go into any details as to what made Rust special/better than X, Y or Z language (not that you need to, but if you're going to make a wild statement, you should be prepared to back it up). The only 3 points made following that were about features that have widely existed in other languages well before Rust came along: async/await, async streams, and serde.

Rust is cool and is still exploring some really great ideas in making programs safer and more maintainable. But, it's not the greatest language ever (at least not yet), nor does it make the impossible possible. It can merely make some things easier at the cost of something else.

I'm glad you were able to use it effectively to create a tool you - and others - find useful.

It may be that they intended the statement more like "Rust's collection of capabilities uniquely inspired us" rather than "Rust is more turing complete than all the others."

For example, the error reporting style in nushell appears to be a direct decendent of Rust's visual error message style.

I was about to say a similar thing. Yes, in a CS-sense anything you can do in one Turing-complete language you can do in any other Turing-complete lang. They almost certainly were not disputing that.

If you factor in the real world, the number shrinks as you get to things like performance, cross-platform support, ease of packaging and distribution.

But still, Nu could plausibly have been written in Assembly, C, C++, D, Go, Nim, Zig, Haskell and so on.

You would have lost some of Rust's strengths and weaknesses and imported another set (regarding the language, tooling, performance, ease of debugging and so on).

I took the comment to mean that as far as they (all people intimately familiar with the language, tooling and ecosystem) are concerned, this would be a trade-off that would be negative enough to not begin or finish the project in the first place.

I don't know what plans and requirements the Nu authors had. But for my own game (a roguelike written in Rust) the requirements on the performance, distribution, stability, ease of development and dependency management meant that Rust was (at the time) pretty much the only viable choice for me.

Could someone have written it in C? Sure. Could I have written it in C? Likely, yes. Would I have ever finish the game in C? Almost certainly no. For various reasons none of the other languages would have done in those circumstances, not really.

Human language is an imperfect method for transferring information and sometimes one trades off brevity for nuance.

The error reporting style also looks almost exactly like PowerShell's (underlining the command in the pipeline that failed), which is written in C#.

It's a 3 person side-project over 3 months. I suppose he means that Rust is to thank for this level of polish and reuse of 3rd party libraries. My experience with Rust is similar and there are many projects I would consider impossible (practically not technically) without Rust.

Programs like shells or servers are expected to perform well and usually cannot afford the GC tax (more memory usage and pauses). I wouldn't use a shell written in a GCed language. That would leave them with non-GC'ed languages. C or C++ simply don't have a usable equivalent of Cargo/crates.io which makes 3rd party reuse slower, and then they have much weaker execution safety guarantees.

"That's just patently false." Why? Projects have constraints including time and skillset of partcipants.

Yeah I agree. That was intellectually dishonest.

I really like the idea and look of this project. To me it feels like PowerShell done better (and implemented by non-aliens).

I was about to post the same thing.

I could put this together in Common Lisp. I know precisely what CL technologies I'd use to do it, and perhaps after finishing the work I'd say "this wouldn't have been possible without Lisp", which would be true in some respect, but just as dishonest as the original article.

That doesn't mean I don't respect the work done. It's an interesting project and most definitely something that deserves attention. The language used to develop it is the least interesting aspect of this.

Yeah, a program written in X would not have been possible without the existence of programming language X.

Maybe it should be read as "my child not be possible without me".

I think you got it wrong. They meant "It would have been impossible for us without rust".

Of course it is possible in any language you can think of, but not with their skills and time.

Interesting! I am also looking forward to setting aside time to play around with https://elv.sh/, which takes a completely different approach with a similar philosophy.

Elvish author here, thanks for the plug :)

I pretty much like the idea, but my biggest problem with current day shells is error handling.

Errexit (aka `set -e`) is a great idea badly executed. There are many things which can be improved (e.g. printing a stack trace by default), but at its core, there are two things which break it.

1. Inconsistent behavior: https://www.in-ulm.de/~mascheck/various/set-e/

2. You can't rely on it.

Inconsistent behavior should be no problem if a new cross-platform implementation is coming, but not being able to rely on it sucks. To give a quick demo:

  foo() {
    set -e
    echo "Sucks"
  printf 'Normal: %s\n' "$(foo)"
  printf 'Condition: %s\n'  "$(foo || true)"

  Condition: Sucks
So if someone is about to bring a new shell, please fix the error handling along the way. But hey, Yehuda Katz is part of the project. So I am looking forward to what is coming next :-)

After running into this and many other issues in sh/bash scripts (such as the pitfalls of handling newlines and spaces correctly with the splitting rules, the awkwardness of basic features such as lists or arithmetic, and the generally inconsistent and obscure syntax), I've moved all my shell scripts to Python3 and I have no regrets about it. Calling external programs is still very easy with the subprocess module. And it's installed pretty much everywhere you have sh/bash on.

To be fair, I am not even sure if creating a new shell scripting language is something worthwhile in this day and age. There's some value in the convenience of instantly turning a sequence of commands you just manually typed into a script, but there's a conflict in that the shell has to provide you a way to hack a sequence of actions together with as few characters as possible, while a script has to be comprehensible, maintainable and extensible. And the inertia you would have to overcome to successfully introduce a new programming language in addition to a new shell makes it intractable.

I have fixed this in https://oilshell.org/ :


Well actually there's one more thing I want to fix:


That will fix your case, because foo is a function, and the disabling of errexit in || is indeed problematic there.

Please subscribe to the bug, or feel free to e-mail at andy@oilshell.org if you want to know when it's fixed. I have another person "on the line" (author of envdir I believe) who also wants this fixed.

It would be great to have you both try this out and verify that errexit in Oil has no more pitfalls :)

I really like this conceptually. I feel like with Powershell, you need a bunch of commands that actually pass objects around or act upon them; and something similar in Linux would require writing a lot of extensions first.

I've been using fish shell for years and glad I started down that route. It's been incredibly useful.

I really want to give this a shot too. I feel like too often I'll open up Python, read a file and then manipulate it when I need to something more complex than simple grep/sed/cut/wc etc... This feels like it really fills in that gap. It starts with something really simple: get your unstructured output in a structure, and then build from it. I could see people writing more custom extensions and objects later on if this gets popular.

PowerShell works on Linux today. It's consistent and works fine for this Go-developer's needs.

It's spectacular to have a shell with consistent verbiage and parameters.


The big problem with Powershell, and it makes it borderline unusable on Linux, is its crazy error handling. The equivalent of bash's `set -e` is `$ErrorActionPreference="Stop"`.

The problem is that it treats any output to stderr as an error and ignores actual posix errors (non-zero exit codes.) What's worse is in some contexts the error it throws on stderr is impossible to catch.

I really love Powershell in a lot of ways but this makes a lot of really basic scripts in bash very cumbersome to write in Powershell.

Do you have any examples of stderr errors being impossible to catch in PS, and/or a basic script that is very easy in Bash but cumbersome in PowerShell?

Recently I was trying to automate something with gsutil (Google's storage CLI) using Powershell remoting to a Windows host. gsutil is a totally well-behaved posix app, which emits status information to stderr. The bug is specifically that when using remoting (Windows equivalent of ssh) the error is uncatchable.

[Remoting Bug] https://github.com/PowerShell/PowerShell/issues/3996

[RFC for improved error handling] https://github.com/PowerShell/PowerShell-RFC/pull/88

Ultimately I just want Powershell to handle well-behaved posix apps in a sane way, but honestly the reaction of some of the people on the thread seems like there's a huge cultural gap between Microsoft and the Posix community. (Which, Microsoft claims to be a part of but clearly is not.)

Python has a similar idea with Mario. But because it's not a shell itself, it requires less of a commitment.


Or even more similar to the project being discussed, https://xon.sh/

If you like xonsh, I might recommend the tako shell (by a former xonsh dev).


Tako seems to handle background tasks better than xonsh. The link to the code isn't working though, so I am not sure why.

Can you tell me which link is broken? The author's a friend, and I can poke him to update it.

Thanks for making me aware of this. It seems less fluent to use mario

I read this, and read all about it, and the whole time I thought the name was “nutshell,” until I saw they abbreviated it “nu.”

Now I’m really disappointed that it’s not called “nutshell.”

But this is still very cool.

Nushell in a nutshell would have been the better title!

I sincerely hope that nushell becomes popular enough for O'Reilly to publish a book with that title. :-)

Interesting PoC, but does it solve a real world problem?

If I had to reinvent the shell by breaking backward compatibility, I would do a statistical study using all the shell scripts publicly available on GitHub, a survey about the critics of the current shells in the literature, and then I would try to optimize a metric such as : words typed to complete a task, reproductivity, time-to-market, communication, mathematical soundness, etc. The result of the study might be that the current shell is enough and the cost of switching to another is too big (I don't know, it's an hypothesis)

People shared many thoughts and opinions on this thread, so that's already a win by itself

>Interesting PoC, but does it solve a real world problem?

Yes, the real world problem of making it combining shell commands easier and more interactive.

A solution doesn't need to solve a whole new problem. It's enough that it solves an existing, already solved problem, better.

The same way the travel luggage with 4 wheels doesn't solve anything the travel luggage with 2 wheels or no wheels solves, just makes so much more easy and awesome.

It also opens up a compelling use case for having a nice web interface for working with the shell (where you can button sort and export to Json on output) since the structure is common.

What real word problem does bash solve over sh? Or tcsh over csh? Or ksh? Fish is the newest shell I’ve been using and it’s leaps and bounds better than dealing with posix syntax for the sake of unnecessary compatibility (if I need posix, I can run a posix shell without trouble). That said, fish is explicit about how the features it includes feel more like they should have evolved in th 90s, not the 2010s.

Asking forgiveness is better than asking permission. Innovation rarely looks like a sure bet prior to it unfolding.

Looks cool, but when I see things like this I figure there's no point in trying to learn something that won't exist on 99% of the systems I need to work on. Bash sucks, but it's almost always there. Embedded systems, desktops, big iron... Like many of the established Unix tools, bash has market penetration and staying power.

You might be in the "late majority" phase of tech adoption, and that's ok. Some of us early adopters bang our heads on new stuff and pass the results of our tests on to slightly less foolhardy souls :) Eventually, established tools arise out of the mess.


I like the concept a lot, especially the focus on outputting data in an organised way.

What I feel could be improved is the presentation of the tabular data. Some ideas below:

- The screenshots show two horizontal lines consisting of dashes, one at the start of the table, one at the end. To me they seem redundant and could be replaced with spaces (whitespace). I did a quick check and PostgreSQL's psql tables also don't have lines like this.

- How about using Unicode lines instead of dashes and plus sign in the header? This way there would be fewer gaps there. I'm not sure though how this would affect compatibility.

Ideally, presentation of the data would be handled by themes. Hopefully they build theming support.

I'd immediately say it would be nice with configuration that could opt-out of the relative and fuzzy dates.

Relative like "today" would be fine, if it includes the exact time today too.

Yes, this is needed for sure. Some output didn't appear on my terminal because it is hard coded to appear as black. The author must have a light colored terminal that differs from mine.

Yes, I think Unicode lines would be much nicer. "+", "-", and "|" are used for many other purposes.

I love the idea of treating output as tabular data.

I see you already have a where operator. What would fit this very well would be in-built sql-like syntax. Since tables are first class, you should have an in-built way of handling them.

A languiage similar to q/kdb - it has a great SQL dsl, but also works as a regular programming language. It's also very concise and expressive, you can write oneliners that do mind bogglingly complex things.

What's a dsl? I wasn't able to find an answer in my quick google search

Domain Specific Language

If you want to play around in the land of DSL's ever, I recommend Racket, a Lisp dialect.

Whoa. It has been a while since I've seen something that truly feels like it can revolutionise my workflow. Amazing job to everyone involved in this project!

Really excited to try this, surprised that there is no pre-built binaries. Are these planned?

Yes, we'll hopefully have those up soon.

i feel like structured data in a shell will phase out. The posix pipes allow streaming any data. So if programs want to stream structured data, just build programs who know how to communicate with each other using structured data. The older shells withstand the test of time with generic interfaces.

For instance, because of the influx of JSON the past few years I have learned to use jq very well. It does everything I need and I didn't need to upgrade my shell to use it. What if everything becomes protobuf down the road?

   i feel like structured data in a shell will phase out. 
   The posix pipes allow streaming any data.
There's a huge mismatch in coreutils.

Nearly everything is built around a line-based philosophy. However, nearly any coreutil you care to name also likes to cram multiple bits of information on a single line.

That means you often have to do a bunch of ugly manipulation with awk, sed, cut, whatever to anything programmatic with the output of each command.

    For instance, because of the influx of JSON the past few 
    years I have learned to use jq very well. It does everything I need
Yeah, jq is awesome and covers a lot of needs. But coreutils don't really emit json, so... yeah. There's a lot that it can't work with, right?

    What if everything becomes protobuf down the road? 
I know you're being facetious, I think protobuf is clearly overkill for very many needs.

Nushell seems to build upon what we've learned in the last 30-40 years. Pipes and simple interchange formats work well and are very useful for a lot of use cases. However, experience also shows that very few programs output lines of text that are atomic in nature. Each line represents multiple things and we frequently need to parse them out. That seems like a very common use case that our shell can handle for us.

I like this a lot but why does this have to be a completely new shell? Couldn't this also just be a set of tools that I can use in an existing shell and get the best of both worlds?

The idea of Mario is to use Python objects in your native shell (bash/zsh/fish/etc)


Xonsh seems like a more popular, maintained and viable approach to that:


Xonsh cant handle backgrounding commands. For example `sleep 10` followed by a ctrl-z just hangs the shell. It has been this way for a long while.

Xonsh is a whole shell environment though, which isn't what parent wanted. As for popularity, yes, Mario is relatively new. :-)

The GitHub repo looks kinda weird on first glance (>1k commits but <100 stars, virtually no contributors or issues, empty LICENSE file) but yeah it is more of what I thought.

Personally, I'm not very accustomed to Python though, so I find the syntax kinda weird. I'd prefer something that feels more like a standard Un*x tool (like nushell).

This was my take away, as well. Bourne and Bash are everywhere and zsh will be included in macOS updates. I definitely see value in a separate set of text processing tools.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact