
Introducing nushell - bradleybuda
http://www.jonathanturner.org/2019/08/introducing-nushell.html
======
toyg
The Powershell idea to push structured data through the pipe, which this
project fundamentally replicates with a relational structure, has its merits;
but it will always displease someone, and carries the burden of having to
rewrite millions of utilities in order to unlock the real potential.

What if, instead, we pushed the approach “upstream”, asking systems to have an
additional stream that all shells can access? We have stdout and stderr, we
could add “stddata” - a pipe that expects and produces structured data. Then
it would become trivial to add support around the userland, without
displeasing anyone - because it’s just an additional interface, not a
replacement. The pipe should support two or three formats to start with (csv,
json, and maybe xml, so you can handle both tabular and child-parent relations
with a varying degree of precision) and shells could have a special character
for piping stddata (I like ¥) so everything else about them would stay the
same.

~~~
StreamBright
I am at the stage that I am more than happy to rewrite whatever it takes to
get better experience. This includes the OS. I think we have sacrificed too
much on the altar of backward compatibility. I guess I am just tired of
breakage because a file has a space in its name.

~~~
alxlaz
> I am at the stage that I am more than happy to rewrite whatever it takes to
> get better experience. This includes the OS

Ever tried to rewrite one (and the software on top of it, since we're
"including" the OS) :)?

~~~
pmarreck
Shout-out to anyone who remembers being blown away by BeOS when it was brand-
new and extremely promising

~~~
justforyou
I now have the Virtual Void song stuck in my head.

------
pixelmonkey
The compelling idea here is that they convert the output of common shell
commands into tabular data that can be manipulated using common operators, so
that you don't have the remember sorting/filtering/grouping flags that may be
different for every different shell command. So, imagine being able to use the
same sorting/filtering logic on the output of `ls` as you might on the output
of `ps`, and without relying on hacky solutions like pipelines to `grep`,
`cut`, and `sort`.

It also means shell command output can be easily transposed into JSON and CSV.
Actually pretty clever!

~~~
entangledqubit
I've been delving into bash scripting a bit more than I'd like as of late and
the lack of universally available consistent structured output for the CLI
really got to me. Most of the script contents end up being these obfuscating
and brittle "hacky solutions" that never should have been necessary. When I
thought about pursuing fixing this I felt like the task was a bit
overwhelming. I'm delighted that these developers are working on this!

~~~
bigtrakzapzap
The key flaw of UNIX philosophy is destructuring deserialization and
reserialization based on lines, necessitating all manner of argument escaping
and field delimiters, when pipelines should be streams of typed messages that
encapsulate data in typed fields. Logs especially (logging to files is a
terrible idea, because it creates log rotation headaches and each program
requires a log parser because of the loss of structured information) also.
Line-oriented pipeline processing is fundamentally _too simple._ Settling on a
common data simple/universal format robust enough for all purposes, including
very large data sets, to exchange between programs and is just complicated
enough to eliminate escapement and delimiter headaches without throwing away
flexibility (by a new/refined set of processing tools/commands) is key.

~~~
whatshisface
It's not so much a flaw as a tradeoff. Because it's so easy to produce lines
of text, virtually every program supports it, even programs written by hackers
who aren't interested in following OS communication standards.

~~~
bigtrakzapzap
No, that's your opinion, not mine. It's a flaw because it creates escaping
magic, delimiters that are in-band (spaces) and every program has to know how
to parse the output of every other program, wasting human time and processing
time. Structured communication, say like an open standard for a schema-less,
self-describing, flexible de/serialization like protobufs/msgpack, would be
far superior, more usable, more efficient and simple but no too simple to
process streams of data with structure and programmability already there.

Being able to dump structured information out of an exception directly into a
log, and then from a log into a database, without any loss of information or
extraneous log parsing, is a clear win. Or from a command as simple as listing
files ("ls") into a database or into any other tool or program. Outputing only
line-oriented strings is just throwing away type information, and creates more
work for everyone else, even more so than continuing to stay with lines
processing tools.

~~~
sk5t
So, if you're not a fan of the UNIX philosophy, maybe check out Powershell. Or
take a look at WMI and DCOM in Windows. Eschew shell scripts in favor of
strongly-typed programs that process strongly-typed binary formats, or XML, or
whatever. The alternatives are out there.

"Worse is better" isn't for everyone.

~~~
JohnBooty
Nushell does not seem like a violation of the unix philosophy, or at least the
version of it that I like best.

"Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a universal
interface."

Perhaps I'm wrong, isn't nushell simply adding one more dimension?

Instead of a one-dimensional array of lines, the standard is now... a two-
dimensional array of tabular data. Perhaps it is not strictly a "text stream"
but this does not seem to me to be a violation of the spirit of the message.

Simple line-based text streams are _clearly_ inadequate IMO. 99.99% of all
Unix programs output multiple data fields on one line, and to do anything
useful at all with them in a programmatic fashion you wind up needing to hack
together some way of parsing out those multiple bits of data on every line.

    
    
        maybe check out Powershell
    

I left Windows right around the time PS became popular, so I never really
worked with it.

It seems like overkill for most/all things I'd ever want to do. Powershell
objects seem very very powerful, but they seemed like too much.

Nushell seems like a nice compromise. Avoids the overkill functionality of
Powershell.

~~~
pmarreck
Semantically, a one-dimensional list/array of “things” can be trivially
transposed into a series of lines with a linebreak separator, but I don’t
think the same holds true for a “list of lists” (2-dimensional data) or a
“list of lists of lists” (3D data) etc. At least without a standard begin-end
list delimiter that allows nesting.

Just thinking about a way that perhaps an old tool can be “wrapped” to be
tricked into working with 2+-dimensional data by somehow divvying up the 2+
dimension input data into concurrent 1-dimensional streams, but this seems to
require a way to represent more than 1 dimension of data without breaking
existing utilities (unless there was, like, a wrapper/unwrapper layer that
handled this...)

------
mamcx
This is almost exactly what I have dreamed with a shell. I'm building a
relational language ([http://tablam.org](http://tablam.org)) with the plan of
something alike.

Some extra ideas:

\- Tables are amazing. But I think we need at least support for this "shapes"
of data: List, Tables, Trees and maybe graphs

\- I think will be nice to have an equivalent of Request/responses alike http.
And then a way to introspect what a command do, like auto-swagger.

\- Having widgets like table-browser, graphs, etc AS COMPONENTS.

In FoxPro you could say "browse" and it display a DataTable grid with full
editing capabilities on the current table (or it ask for a table to open). In
this case:

    
    
        ls | ui_browse
    

The key here is that the widgets work stand-alone, so is not necesary to fully
build a terminal app just to see nicely data. Look like
[https://goaccess.io](https://goaccess.io) and more like this idea:
[https://github.com/sqshq/sampler](https://github.com/sqshq/sampler)

This mean we could get a jupyter-like experience here!

\- Some way to do curryng and maybe environment alias: You could setup an
environment for the system, the user, this directory. In the environment is a
.toml with vars and stuff.

The other thing where I think shells fail... is that are still old-style
terminal. Having text output is cool, but having more rich widget system will
unlock better the potential

~~~
felixfbecker
Like PowerShell's different output formats (table, list, wide, ...)?

~~~
mamcx
No output. Declared the input/output is in the shape table, list, tree.

Like when a http response say is json, text, csv, etc. This information unlock
certain capabilities and will allow to provide efficient execution of queries.

If psql (postgresql terminal cli) declare their data is in the "tables" shape
then it could take the |where name = "" query itself and execute it instead of
output all the rows...

------
roryrjb
There's a lot of comments here already so I don't know if this has been
addressed already, but I don't get this shell. It seems to be fixing a problem
that doesn't exist, and a lot of comments are echoing it. Current shells are
not deficient, as Doug McIlroy originally stated, the universal interface is
text and tools should take input and generate output based on this. You
obviously can then pipe things together with these tools not necessarily
knowing in advance how they will be used in a pipeline. You can get very far
with sed, awk, etc, you just have to spend a little bit of time learning them,
but they are not complicated. You can of course then introduce a command like
jq for specialised purposes. But hte point is the shell isn't providing this,
they are separate tools that you orchestrate using the shell. There is not a
flaw with how it works at the moment, I truly believe and this isn't a dig at
this project or anyone in particular, that people do not spend enough time
learning these tools that have been there for decades. The re-inventions that
appear on GitHub are not a product of the deficiency of the existing tools, I
don't know what they are but it's not that. Unless I am missing something.

~~~
roryrjb
One downvote and counting. This is just my opinion. Let's crank this up a
notch. Those advocating for more advanced interfaces are missing the point.
Text is all you need, no graphics or objects or widgets, just text. I have no
problem with this, I enjoy it like this, we don't need an improvement. There I
said it, any downvotes will be worth it, because I get to voice my opinion. In
the end I will still be happily using my POSIX shell scripts, awk, sed and
enjoy my life.

~~~
Siyo
"Goto is all you need, no ifs or loops or dynamic dispatch, just jumps. I have
no problem with this, I enjoy it like this, we don't need an improvement.
There I said it, any downvotes will be worth it, because I get to voice my
opinion. In the end I will still be happily using my assembler and enjoy my
life."

I'm sorry, I couldn't help myself. Your comment reminds me of an anecdote I
heard from the early times of structured programming. When structured
programming was just gaining its feet, there was a certain class of
programmers who just could not understand why people would want to write
structured code. You can do everything in assembly they said, you have much
more control over performance, etc. They looked down on structured programming
as not "real programming".

There's a lot of benefits to adding some structure to text. I don't think that
Nushell's approach is the best one, but to say that there are no problems and
we shouldn't look to improve things is just backwards. We should always look
to improve our tools and our craft, otherwise we would still be stuck writing
assembly.

~~~
chongli
_There 's a lot of benefits to adding some structure to text._

There are benefits, true, but there are also potentially serious drawbacks.
Chief among them, I would hazard, is the risk that we get locked into a format
that didn't anticipate a (completely unknown) future need and we have to go
back and rewrite everything again.

The beauty of text's lack of structure is that people are free to interpret it
any way they please.

~~~
ziftface
I keep seeing comments like this one, but just as you can parse textual data
into something more structured, like the examples in the post, you can also
print the structured data as text. So the tools that use and generate
structured data are no less compatible because it's easy to go back and forth
between the formats at will. You're not losing anything with this approach.

~~~
sverige
> You're not losing anything with this approach.

True, but you are adding something. And if you add things you don't really
need, it becomes plain old cruft.

Like the others above, I understand the urge to make some things simpler by
adding complexity, but for the long haul I'm convinced it's better to keep
tools simple and add any needed complexity to the code you're writing (whether
it's shell scripts or anything else). And then document the complexity so
future you or some poor stranger can understand why the hell you did that in
the first place.

But I don't look forward to reading comments like "X was deprecated so we're
converting these tables back to text streams so they can be converted back to
Y properly."

------
rvz
A brand new shell, especially in Rust is a great thing to see and also how it
tackles the problems that exist in other shells.

However, its interesting to see so many dependencies that are required to
build it (73 crates last time I checked) and as shells have always been
portable across many operating systems, I wonder how portable nushell would be
since it wishes to replace my current shell.

~~~
smt88
I think the JavaScript/npm world has trained a generation of devs that high
dependency counts are a code smell. This _can_ be true, of course, but that's
largely because of external factors.

In a perfect world, a language's ecosystem would have a tiny stdlib and allow
importing of libraries for everything else. The Linux philosophy would be
followed strictly.

The problem is just that the overhead in securing, maintaining, and organizing
all those libraries is pretty large, as we've seen by npm's repeated failures.
Of course the *nix community seems to have largely solved the problem, but
there's also a massive culture of volunteerism benefiting them.

~~~
wwright
I think there’s definitely room for mirrors of package repos that focus on a
stablized and audited subset of packages, with frequent security and bug
backports and medium-length cycles for new features — like what Fedora and
Debian do, but for language ecosystems.

~~~
Qerub
Agreed. ActiveState has had that as a commercial offering for quite some time
([https://www.activestate.com/products/](https://www.activestate.com/products/))
and Red Hat is getting into it with (OpenShift) Application Runtimes. Given
that providing such a thing is hard not-so-fun work I have a hard time seeing
it getting available for free. (ASP).NET Core might be alone in being well-
financed and free, something I think will lure (enterprise software)
developers over long-term.

------
felixfbecker
This looks great, but what exactly are the benefits over PowerShell? The
article mentions PowerShell, then says:

> What if we could take the ideas of a structured shell and make it more
> functional? What if it worked on Windows, Linux, and macOS? What if it had
> great error messages?

\- Any shell with piping is already very functional, and PowerShell has script
blocks (higher-order functions). Or does this mean "functional" in the literal
sense? What functionality is missing? \- PowerShell works on macOS, Linux and
Windows \- The examples of error messages further down look almost exactly
like PowerShells error messages (underlining the command in the pipeline that
failed)

It is not clear to me what exactly the authors sought out to do different

~~~
kjksf
They clearly know about PowerShell. They acknowledge it as an inspiration and
the core concept is the same.

That doesn't mean there aren't ways to improve upon PowerShell.

As a light user of PowerShell I consider it a poor implementation of a great
idea.

Commands are way too long, control structures have poor syntax etc.

To make this very concrete, take this example from nu:

    
    
        ls | where size > 4kb
    

Now give me an equivalent in PowerShell and then let's talk about how
different Nu is from PowerShell.

~~~
dragonwriter
The equivalent, btw, is:

    
    
      ls | where length -gt 4kb
    

The only difference is powershell uses “>” and “<” for redirection, and so
uses mnemonic operators for comparisons.

Nu doesn't seem to support redirection (or even have anything other than one
input and one output stream, though metadata might serve the purposes
otherwise served by additional streams; it's not how a program would return
something Nu would treat as metadata, though.)

I guess that with a structured-data shell, error type of info could just be
returned in-band under a different tag, in any case.

------
massung
Pretty cool idea and project. Hope you take it further and it turns into
something truly spectacular!

My only comment is on the article. This line:

> Nu would not have been possible without Rust.

That's just patently false.

Perhaps you wouldn't have chosen to code it in another language. Or perhaps
Rust had one or two benefits that assisted in the development of Nu. But you
don't word it that way or go into any details as to what made Rust
special/better than X, Y or Z language (not that you need to, but if you're
going to make a wild statement, you should be prepared to back it up). The
only 3 points made following that were about features that have widely existed
in other languages well before Rust came along: async/await, async streams,
and serde.

Rust is cool and is still exploring some really great ideas in making programs
safer and more maintainable. But, it's not the greatest language ever (at
least not yet), nor does it make the impossible possible. It can merely make
some things easier at the cost of something else.

I'm glad you were able to use it effectively to create a tool you - and others
- find useful.

~~~
moogly
Yeah I agree. That was intellectually dishonest.

I really like the idea and look of this project. To me it feels like
PowerShell done better (and implemented by non-aliens).

~~~
lokedhs
I was about to post the same thing.

I could put this together in Common Lisp. I know precisely what CL
technologies I'd use to do it, and perhaps after finishing the work I'd say
"this wouldn't have been possible without Lisp", which would be true in some
respect, but just as dishonest as the original article.

That doesn't mean I don't respect the work done. It's an interesting project
and most definitely something that deserves attention. The language used to
develop it is the least interesting aspect of this.

~~~
johnisgood
Yeah, a program written in X would not have been possible without the
existence of programming language X.

------
Legogris
Interesting! I am also looking forward to setting aside time to play around
with [https://elv.sh/](https://elv.sh/), which takes a completely different
approach with a similar philosophy.

~~~
xiaq
Elvish author here, thanks for the plug :)

------
arendtio
I pretty much like the idea, but my biggest problem with current day shells is
error handling.

Errexit (aka `set -e`) is a great idea badly executed. There are many things
which can be improved (e.g. printing a stack trace by default), but at its
core, there are two things which break it.

1\. Inconsistent behavior: [https://www.in-
ulm.de/~mascheck/various/set-e/](https://www.in-
ulm.de/~mascheck/various/set-e/)

2\. You can't rely on it.

Inconsistent behavior should be no problem if a new cross-platform
implementation is coming, but not being able to rely on it sucks. To give a
quick demo:

    
    
      #!/bin/bash
      
      foo() {
        set -e
        false
        echo "Sucks"
      }
      
      printf 'Normal: %s\n' "$(foo)"
      printf 'Condition: %s\n'  "$(foo || true)"
    

Output:

    
    
      Normal: 
      Condition: Sucks
    

So if someone is about to bring a new shell, please fix the error handling
along the way. But hey, Yehuda Katz is part of the project. So I am looking
forward to what is coming next :-)

~~~
sigotirandolas
After running into this and many other issues in sh/bash scripts (such as the
pitfalls of handling newlines and spaces correctly with the splitting rules,
the awkwardness of basic features such as lists or arithmetic, and the
generally inconsistent and obscure syntax), I've moved all my shell scripts to
Python3 and I have no regrets about it. Calling external programs is still
very easy with the subprocess module. And it's installed pretty much
everywhere you have sh/bash on.

To be fair, I am not even sure if creating a new shell scripting language is
something worthwhile in this day and age. There's some value in the
convenience of instantly turning a sequence of commands you just manually
typed into a script, but there's a conflict in that the shell has to provide
you a way to hack a sequence of actions together with as few characters as
possible, while a script has to be comprehensible, maintainable and
extensible. And the inertia you would have to overcome to successfully
introduce a new programming language in addition to a new shell makes it
intractable.

------
djsumdog
I really like this conceptually. I feel like with Powershell, you need a bunch
of commands that actually pass objects around or act upon them; and something
similar in Linux would require writing a lot of extensions first.

I've been using fish shell for years and glad I started down that route. It's
been incredibly useful.

I really want to give this a shot too. I feel like too often I'll open up
Python, read a file and then manipulate it when I need to something more
complex than simple grep/sed/cut/wc etc... This feels like it really fills in
that gap. It starts with something really simple: get your unstructured output
in a structure, and then build from it. I could see people writing more custom
extensions and objects later on if this gets popular.

~~~
j88439h84
Python has a similar idea with Mario. But because it's not a shell itself, it
requires less of a commitment.

[https://github.com/python-mario/mario](https://github.com/python-mario/mario)

~~~
sciurus
Or even more similar to the project being discussed,
[https://xon.sh/](https://xon.sh/)

~~~
agalunar
If you like xonsh, I might recommend the tako shell (by a former xonsh dev).

[https://takoshell.org/](https://takoshell.org/)

~~~
Skunkleton
Tako seems to handle background tasks better than xonsh. The link to the code
isn't working though, so I am not sure why.

~~~
agalunar
Can you tell me which link is broken? The author's a friend, and I can poke
him to update it.

~~~
Skunkleton
[https://hz.mit.edu/git/tako/tako](https://hz.mit.edu/git/tako/tako)

------
burlesona
I read this, and read all about it, and the whole time I thought the name was
“nutshell,” until I saw they abbreviated it “nu.”

Now I’m really disappointed that it’s not called “nutshell.”

But this is still very cool.

~~~
MeteorMarc
Nushell in a nutshell would have been the better title!

~~~
doodpants
I sincerely hope that nushell becomes popular enough for O'Reilly to publish a
book with that title. :-)

------
antpls
Interesting PoC, but does it solve a real world problem?

If I had to reinvent the shell by breaking backward compatibility, I would do
a statistical study using all the shell scripts publicly available on GitHub,
a survey about the critics of the current shells in the literature, and then I
would try to optimize a metric such as : words typed to complete a task,
reproductivity, time-to-market, communication, mathematical soundness, etc.
The result of the study might be that the current shell is enough and the cost
of switching to another is too big (I don't know, it's an hypothesis)

People shared many thoughts and opinions on this thread, so that's already a
win by itself

~~~
coldtea
> _Interesting PoC, but does it solve a real world problem?_

Yes, the real world problem of making it combining shell commands easier and
more interactive.

A solution doesn't need to solve a whole new problem. It's enough that it
solves an existing, already solved problem, better.

The same way the travel luggage with 4 wheels doesn't solve anything the
travel luggage with 2 wheels or no wheels solves, just makes so much more easy
and awesome.

~~~
tgtweak
It also opens up a compelling use case for having a nice web interface for
working with the shell (where you can button sort and export to Json on
output) since the structure is common.

------
01100011
Looks cool, but when I see things like this I figure there's no point in
trying to learn something that won't exist on 99% of the systems I need to
work on. Bash sucks, but it's almost always there. Embedded systems, desktops,
big iron... Like many of the established Unix tools, bash has market
penetration and staying power.

~~~
canadaduane
You might be in the "late majority" phase of tech adoption, and that's ok.
Some of us early adopters bang our heads on new stuff and pass the results of
our tests on to slightly less foolhardy souls :) Eventually, established tools
arise out of the mess.

[https://ondigitalmarketing.com/learn/odm/foundations/5-custo...](https://ondigitalmarketing.com/learn/odm/foundations/5-customer-
segments-technology-adoption/)

------
thih9
I like the concept a lot, especially the focus on outputting data in an
organised way.

What I feel could be improved is the presentation of the tabular data. Some
ideas below:

\- The screenshots show two horizontal lines consisting of dashes, one at the
start of the table, one at the end. To me they seem redundant and could be
replaced with spaces (whitespace). I did a quick check and PostgreSQL's psql
tables also don't have lines like this.

\- How about using Unicode lines instead of dashes and plus sign in the
header? This way there would be fewer gaps there. I'm not sure though how this
would affect compatibility.

~~~
nwah1
Ideally, presentation of the data would be handled by themes. Hopefully they
build theming support.

~~~
kzrdude
I'd immediately say it would be nice with configuration that could opt-out of
the relative and fuzzy dates.

Relative like "today" would be fine, if it includes the exact time today too.

------
maest
I love the idea of treating output as tabular data.

I see you already have a where operator. What would fit this very well would
be in-built sql-like syntax. Since tables are first class, you should have an
in-built way of handling them.

A languiage similar to q/kdb - it has a great SQL dsl, but also works as a
regular programming language. It's also very concise and expressive, you can
write oneliners that do mind bogglingly complex things.

~~~
smashthepants
What's a dsl? I wasn't able to find an answer in my quick google search

~~~
lvturner
Domain Specific Language

------
dom96
Whoa. It has been a while since I've seen something that truly feels like it
can revolutionise my workflow. Amazing job to everyone involved in this
project!

Really excited to try this, surprised that there is no pre-built binaries. Are
these planned?

~~~
jntrnr1
Yes, we'll hopefully have those up soon.

------
sigmonsays
i feel like structured data in a shell will phase out. The posix pipes allow
streaming _any_ data. So if programs want to stream structured data, just
build programs who know how to communicate with each other using structured
data. The older shells withstand the test of time with generic interfaces.

For instance, because of the influx of JSON the past few years I have learned
to use jq very well. It does everything I need and I didn't need to upgrade my
shell to use it. What if everything becomes protobuf down the road?

~~~
JohnBooty

       i feel like structured data in a shell will phase out. 
       The posix pipes allow streaming any data.
    

There's a huge mismatch in coreutils.

Nearly everything is built around a line-based philosophy. However, nearly any
coreutil you care to name also likes to cram multiple bits of information on a
single line.

That means you often have to do a bunch of ugly manipulation with awk, sed,
cut, whatever to anything programmatic with the output of each command.

    
    
        For instance, because of the influx of JSON the past few 
        years I have learned to use jq very well. It does everything I need
    

Yeah, jq is awesome and covers a lot of needs. But coreutils don't really emit
json, so... yeah. There's a lot that it can't work with, right?

    
    
        What if everything becomes protobuf down the road? 
    

I know you're being facetious, I think protobuf is clearly overkill for very
many needs.

Nushell seems to build upon what we've learned in the last 30-40 years. Pipes
and simple interchange formats work well and are very useful for a lot of use
cases. However, experience also shows that very few programs output lines of
text that are atomic in nature. Each line represents multiple things and we
frequently need to parse them out. That seems like a very common use case that
our shell can handle for us.

------
t0astbread
I like this a lot but why does this have to be a completely new shell?
Couldn't this also just be a set of tools that I can use in an existing shell
and get the best of both worlds?

~~~
j88439h84
The idea of Mario is to use Python objects in your native shell
(bash/zsh/fish/etc)

[https://github.com/python-mario/mario](https://github.com/python-mario/mario)

~~~
scrollaway
Xonsh seems like a more popular, maintained and viable approach to that:

[https://github.com/xonsh/xonsh/](https://github.com/xonsh/xonsh/)

~~~
j88439h84
Xonsh is a whole shell environment though, which isn't what parent wanted. As
for popularity, yes, Mario is relatively new. :-)

~~~
t0astbread
The GitHub repo looks kinda weird on first glance (>1k commits but <100 stars,
virtually no contributors or issues, empty LICENSE file) but yeah it is more
of what I thought.

Personally, I'm not very accustomed to Python though, so I find the syntax
kinda weird. I'd prefer something that feels more like a standard Un*x tool
(like nushell).

------
orf
Looks amazing! Unfortunately a `cargo install` on stable fails due to a
dependency "futures-async-stream v0.1.0-alpha.4" with the "rawkey,clipboard"
features.

The source is pretty interesting. Here's the PS command:
[https://github.com/nushell/nushell/blob/master/src/commands/...](https://github.com/nushell/nushell/blob/master/src/commands/ps.rs#L27)

And the sort-by:
[https://github.com/nushell/nushell/blob/master/src/commands/...](https://github.com/nushell/nushell/blob/master/src/commands/sort_by.rs#L29)

~~~
digitalinfinity
Yup- the doc at [https://book.nushell.sh/en/installation#installing-
rust](https://book.nushell.sh/en/installation#installing-rust) does call out
that it currently needs the rust nightly toolchain.

------
zellyn
How do I do this in nushell? It's an alias I created recently to ripgrep
through java and go files and tell me which have both of two words:

    
    
      function both {
          comm -1 -2 <(rg -i -l -t java -t go $1 | sort) <(rg -i -l -t java -t go $2 | sort)
      }
    

Example usage: `both sharded jobqueue`

~~~
tux1968
Your question brings to light some of the deep flexibility current shells
allow with a very terse syntax. At least from Nu you can accept and process
the output of your script.

As an aside, your one liner is having to process every file twice, even ones
that have already been excluded by the other ripgrep... something like this
should be more performant:

rg -il $2 $(rg -il -t java -t go $1)

~~~
zellyn
Nice. Perhaps with an xargs thrown in to be safe...

------
entire-name
An object pipeline is certainly interesting, and I do use PowerShell from time
to time. However, there is a reason extremely simple concepts such as
"everything is a file" or "input/output/error are all text" survived for this
long. I will quote an answer provided in a vaguely related question [0]:

> There is no spoon^H^H^Hstructured data. There is only text.

[0]:
[https://unix.stackexchange.com/a/400](https://unix.stackexchange.com/a/400)

------
fouc
This nicely ties into the recent HN thread about a dream OS that would provide
databases at the OS level instead of forcing the re-implementation of
databases in many programs.

[https://news.ycombinator.com/item?id=20754592](https://news.ycombinator.com/item?id=20754592)

------
coldtea

      ^ls -la | lines | skip 1 | split-column " " perms files group user size month day time name
    

This is neat, but does nushell have the ability (should be trivial to add) to
save such a pipeline and reuse it (similar to an "alias")?

e.g.

    
    
      myls = ^ls -la | lines | skip 1 | split-column " " perms files group user size month day time name
    

and then:

    
    
      myls | where size > 1000
    

Also perhaps the built-in commands could be namespaced? E.g. nu.ls or nu-ls vs
ls. Then you wouldn't need escaping, and the names wouldn't shadow traditional
commands (but could allude to them).

~~~
okaram
About namespacing, they are doing something equivalent, but making theirs the
default. If they make theirs nu.ls, many (most) people would type ls ;)

~~~
coldtea
>* If they make theirs nu.ls, many (most) people would type ls ;)*

Sure, and they'll get "ls" (the standard one, from POSIX), which should be the
least surprise. From then on, they'll know to call nu.ls explicitly when they
need it. Sounds like a better option.

------
chrissoundz
I really enjoy seeing new ideas around interfaces. But I do wonder, rather
than creating a new shell, couldn't we have a 'json/structured' mode for cli
programs - and maintain compatibility with existing shells? For example if it
was toggled with a SHELL_FORMAT environment variable?

What issues could be faced with this approach?

~~~
munchbunny
PowerShell has an IMO good answer to this: if you run a classic CLI program,
its output is interpreted as an array of strings (lines), so if you pipe the
output, you have a way to process it into something object-like.

Having a toggle creates the problem that you have to write two implementations
for your tools, or you end up having to memorize which commands can run in
which modes. It's easier to have a consistent pattern and a way to up-convert
existing CLI programs.

------
dustfinger
This project has goals similar to GNU eshell [1]. Great ideas, thank you for
sharing.

[1]
[https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...](https://www.gnu.org/software/emacs/manual/html_mono/eshell.html#Introduction)

------
m3at
This is great, we have a tendency to be content with existing tools we're
familiar with but be blind to their limitations. I'm looking forward to how
this project evolve!

However an important point for practical usage is the capability to run
existing scripts, for example as oilshell [1] does. I skimmed through the
README and doc but couldn't find the current status nor future plans.

[1]
[http://www.oilshell.org/blog/2019/06/17.html](http://www.oilshell.org/blog/2019/06/17.html)

------
mstratman
Any time the first line of a project description reads "This is NewTool,
written in <language>" it's an immediate red flag that makes it hard for me to
continue reading.

Obviously it doesn't necessarily mean the project has no merits, but language
choice is one of the less relevant details and it signals that the author
wanted to make a toy in a language new to them. Good for them, but most of the
time the rest of the world doesn't care.

If the project has merits, and this one might, it should talk about them in
the intro.

~~~
crispinb
> most of the time the rest of the world doesn't care

Much of the time, HN readers however will. There's the odd software developer
hanging about here, some of whom are interested in programming languages, or
so I believe.

------
skywhopper
This is really cool. I've always liked the ideas behind Powershell's approach,
and so I'm glad to see someone is working on Linux-friendly shell that has the
same sort of features.

I have to say that I don't care for the snark about "having to remember"
arguments to various pipeline-munging commands. I get that this is a big
challenge for new folks working with typical unix shells, but I'm not sure I
see how this actually solves it. In order to avoid all those flags, a system
like this requires more verbs overall, and those verbs have their own argument
structure, eg `split-column " | " firstname lastname job`, so now instead of
being able to label the arguments and order them how I want, I now have to
remember the order they come in, and what they mean, and that `split-column "
"` apparently eats extra whitespace (does `split-column " | "`?). Yes, the
standard unix tools have these issues, as well. That's my point.

So yes, I love this idea, please keep pursuing it, and I look forward to
trying it out. But focus on the benefits of working with structured data
rather than pushing lines that sound good but aren't actually true.

------
savolai
The idea is inspiring!

I'm finding it hard to get over them using pipes and dashes for table borders.
Is this the 70's? It's genuinely hard to read these tables. See:
[https://www.interaction-
design.org/literature/topics/gestalt...](https://www.interaction-
design.org/literature/topics/gestalt-principles)

------
nikisweeting
Hmm I like this but there's no way it's replacing fish for me, I'd rather use
it as standalone command that provides the same functionality.

e.g. `ls | nu where size > 10` or `nu open Cargo.toml`

~~~
IggleSniggle
I finally decided to go look at fish. Why didn’t I do this sooner??

------
nailer
As a long time Unix person who also discovered pwsh, like Yehuda did in the
article:

Can I make nushell cmdlets (or equivalent) in something other than rust? pwsh
limits me to .net languages, but I wanna use node.

------
gigatexal
This looks really cool but isn’t Powershell cross platform now? It looks to
shadow much of the features of Powershell which opens scripting to the whole
of the .Net library

~~~
juststeve
The assumption with powershell is that your admin tasks should be coded in
.net, and their dependencies should be imported from .net. Not required, but
preferred.

~~~
gigatexal
Yeah and since .Net isn’t my cup of tea this shell could be pretty cool

~~~
juststeve
yes hopefully this project can implement a double pipe operator or something,
so one pipe means pipe structured data (like powershell) and the other is pipe
stdin/stdout output via memory so it is super fast.

------
naasking
Not sure if they're aware of Shcaml, which seems even more powerful since it
supports anonymous records in lines:

[https://www.ccs.neu.edu/home/tov/code/shcaml/doc/](https://www.ccs.neu.edu/home/tov/code/shcaml/doc/)

Edit: features look similar in fact, since you can name columns and refer to
them by name. Neat!

------
Klasiaster
I find it good in general but don't like the idea of reusing the names like
`ls` and so on which is confusing with the option to invoke the real `ls`.
Since they have single-character names already, a simple `l` would be one way
to solve this.

I'm interested to see a solution to error handling (within multiple pipes!)
and subprocess management as well but couldn't find much in the docs.

------
j88439h84
Mario has a similar feel to this but in Python, and lets you use arbitrary
code and construct your own functions in pure Python.

[https://github.com/python-mario/mario](https://github.com/python-mario/mario)

    
    
        $ mario read-csv-dicts map 'x["name"]' < hackers.csv
        Alice
        Bob
        Carol

~~~
mkl
You've made 9 comments in this thread, all promoting Mario. Are you the
author?

------
jonahss
Love it! Thanks. I've always thought powershell was a great concept, badly
executed (those long names with capital letters, confusing errors, and
insanely long time to return when a command is missing).

Is anyone interested in a shell with a GUI? As in, data still passing as text,
but having a shell which can render textual data into widgets when displayed
in the shell?

~~~
felixfbecker
PowerShell has this with out-gridview, check out:
[https://devblogs.microsoft.com/powershell/out-gridview-
retur...](https://devblogs.microsoft.com/powershell/out-gridview-returns/)

My favourite feature is that you can select objects in the GUI to pass on to
the next pipeline step

------
perspective1
Very neat, love it. Few things-- 1) Recommend this install line (for those of
us on mainline/standard who've never used a different toolchain) cargo
+nightly install nu 2) Things like ls | where readonly don't work as
expected-- readonly should be a bool, shouldn't it? 3) How do you paginate? ls
| less doesn't work

------
rajangdavis
This seems like it could potentially be an awesome tool for data
cleaning/munging in the terminal.

Really interested to see where this goes..

------
cwmoore
Within the boundaries of the demo, it feels like all my time reading man pages
is a landline network, and this is cellular.

------
gitgud
Interesting, there's definitely a niche for this. But I'm not sure it could
replace a normal shell in my opinion. I'm confused at what other commands
would do in this context? Will it tabularize; echo "Hello World"?

There's an elegant simplicity of treating all piped output as a string rather
than objects...

------
aerovistae
Feature suggestion: detect timing differences between program outputs to
determine where an individual output ended and the next one began, so people
can automatically scroll to the top of a stack trace for instance by double
clicking it or something because the shell will recognize all those lines as a
single message.

------
Gehinnn
Is the processed data typed? Can you offer auto completion of data fields
based on these types?

I also thought quite a couple of times about implementing a data centric,
typed shell language. My primary goal was to have a shell where one could
discover every feature by oneself through the magic of types.

------
maest
People are really excited about this. Here's a question:

Would you pay for this?

My prior is that no, people would not pay for a new shell - they'd prefer
sticking to bash because A. it's "standard" B. it's free.

I have to wonder how developing new tolls like nushell can be sustained over
the long run.

------
lostmsu
I feel open parsing files is misguided. Downright wrong if it profilerates
into scripts, because suddenly simple scripts can't work on JSON, because
opening JSON returns something different from a stream of lines.

~~~
dragonwriter
> I feel open parsing files is misguided. Downright wrong if it profilerates
> into scripts, because suddenly simple scripts can't work on JSON, because
> opening JSON returns something different from a stream of lines.

Doesn't “open --raw” address that?

[https://book.nushell.sh/en/loading_data#opening-in-raw-
mode](https://book.nushell.sh/en/loading_data#opening-in-raw-mode)

Also, I suspect most simple scripts that want to work on JSON Will _want_ to
parse it, so the default makes them more easily able to work with it, not
less.

~~~
lostmsu
I am having a problem with --raw not being default. Defaults must be secure
and stable.

~~~
mkl
Can you explain why parsing JSON can't be secure or stable?

~~~
lostmsu
The problem is not with JSON parsing. It is with the fact, that people will
use open without --raw to process, for example, Python files, because it is
easy and works now. And some day later shell will learn to parse them into AST
with open, and today's scripts will no longer work. Or worse start doing
something very wrong.

------
sriram_malhar
This looks very interesting.

Unfortunately, I'm having trouble building it on Mac OS X 10.14.5 and with the
latest rust update installed.

    
    
        cargo:warning=libgit2/src/streams/stransport.c:12:10: fatal 
           error: 'CoreFoundation/CoreFoundation.h' file not found
    
        cargo:warning=#include <CoreFoundation/CoreFoundation.h>
    

Can anyone clue me in? An Xcode update perhaps? (I'm doing it, but it is
taking forever). I have a CoreFoundation.h on my system under MacOSX10.14.sdk,
but it is not under a CoreFoundation folder.

------
magoon
Bash is ubiquitous - the best selling point for knowing it and sticking to it.

I desire a fancy new shell but don’t want it to become a crutch, limiting my
usefulness and familiarity when I’m on a system lacking it.

------
tarruda
The idea of programs that work on structured data is quite old. Anyone ever
read "structural regular expressions" by Rob Pike? Highly recommended:
[http://doc.cat-v.org/bell_labs/structural_regexps/](http://doc.cat-v.org/bell_labs/structural_regexps/)

Also, using structural regexps for efficient interactive text editing:
[https://github.com/martanne/vis](https://github.com/martanne/vis)

------
kerng
This is very cool! The giant drawback of Poweshell is that it's so focused on
Windows - this projects seems to provide some of that power cross platform.
Very neat idea!

~~~
felixfbecker
PowerShell has been focused on cross platform since 2016 (3 years now)

------
qwerty456127
I'we always felt like there should be no file system and no shell but a
relational database with a particularly suitable and extensible SQL dialect
REPL.

~~~
joshspankit
I’m digging this, and up you a graph database, But how would you handle
“small-file” cruft? Still have optional “folder” rows?

~~~
qwerty456127
I can't really tell how do we apply make this idea compatible with the common
workflow/model of computing. In my imaginary world everything down to the
level of a statement of what is a code file in the real world is a separate
record in a relational database, there are no file names and no folders (only
tags, commentaries and other metadata).

In the real world folder hierarchy is probably inevitable so yes, it has to be
stored in a way (in a separate table perhaps so we can have many-to-many
relationships sort of what hardlinks do).

~~~
scoobyyabbadoo
> In my imaginary world everything down to the level of a statement of what is
> a code file in the real world is a separate record in a relational database,

I think you'd like prolog [https://swish.swi-
prolog.org/example/examples.swinb](https://swish.swi-
prolog.org/example/examples.swinb)

~~~
qwerty456127
It's cool, indeed. Nevertheless, besides files which contain sets of logical
statements/data, there are pictures, audio and video recordings which are
binary by nature. Does Prolog support this kind of data types?

~~~
scoobyyabbadoo
Yes, and you can see a few hundred libraries here [https://www.swi-
prolog.org/pack/list](https://www.swi-prolog.org/pack/list)

------
weyman
Yes! I’ve always preferred PowerShell for scripting, Bash for an interactive
shell. This seems like the best of both worlds, can’t wait to try it out.

------
grandinj
how long before it starts implementing SQL over the input data?

They are already tables after all :-)

~~~
chubot
Yeah for [http://oilshell.org/](http://oilshell.org/) I am planning to use the
model of "data frames", which I compare to SQL here:

 _What Is a Data Frame? (In Python, R, and SQL)_

[http://www.oilshell.org/blog/2018/11/30.html](http://www.oilshell.org/blog/2018/11/30.html)

I think starting with either SQL or data frames would be a good choice. They
are similar except data frames also focus more on mathematical modeling, and
data frames usually don't have indices. Otherwise they both have
selection/projection, sorting, various kinds of joining, etc. Data frames also
have vectorized arithmetic.

I am trying to reuse the designs that have proven to work over time (R, SQL,
bash) rather than inventing new variants with potentially needless
differences.

------
SneakyPit
In case you are interested in more sophisticated ls or find, try fselect (
[https://github.com/jhspetersson/fselect](https://github.com/jhspetersson/fselect)
). That tool supports JSON as output as well.

Fselect has few really cool features, esp. when you got to deal with complex
queries. Although not unix way at all.

------
dgudkov
> Everything is data

We use a similar approach in our data prep tool [1] for non-technical users.
Lists of files, lists of emails, contents of text files, spreadsheets or
database tables - piping it all together allows mixing ETL and process
automation in a similar fashion.

[1] [https://easymorph.com](https://easymorph.com)

Disclaimer: I'm the founder.

------
nixpulvis
I too have been off and on writing a Rust shell, though I want it to support
`chsh` POSIX compatibility for my Arch setup. It's no where near complete
though.

[https://github.com/nixpulvis/oursh](https://github.com/nixpulvis/oursh)

`nu` seems very cool (and I love the name ;), but isn't building atm.

------
GordonS
I like how powerful Powershell is, but I absolutely loathe it's verbose syntax
and Pascal-Kebab-Case naming (at least for idiomatic Powershell).

This seems to have a lot of the power of Powershell, but with a terse,
familiar syntax - I really like the look of this!

Being cross-platfotm is a huge seller for me too - I mainly work on Windows,
but also on Linux and MacOS.

~~~
gilrain
PowerShell is cross platform these days, FYI.

~~~
GordonS
Yes, I should have mentioned that I've used Powershell Core too.

------
pvg
Wonder if the temptation wasn't there to just call it 'nutshell' and if so,
how it was resisted.

~~~
frereubu
That's in fact how I read it in the title before reading your comment...

------
hestefisk
I like this. Very neat with tables; easier to memorise than PowerShells very
verbose command names. It feels like a pseudo SQL interface to the OS. Only
area where I think it oversteps the Unix philosophy is the integrated terminal
multiplexer. I think the shell should be terminal agnostic.

------
frankus
This seems to address a lot of the shortcomings presented here:
[https://mkremins.github.io/blog/unix-not-acceptable-
unix/](https://mkremins.github.io/blog/unix-not-acceptable-unix/)

------
ilaksh
This looks great. Is the autocomplete as good as fish? If so I may switch from
fish to nu. The autocomplete matters a lot. With fish I often repeat commands
during development and I only have to type the first letter or two.

~~~
felixfbecker
I find PowerShell autocompletion amazing. It has Ctrl+Space autocompletion
similar to an IDE where you can select options with up and down arrow. Also
ctrl+r for history search like fish (not always active like fish
unfortunately)

------
guylepage3
Lately I've been paying more attention to alternative CLI tooling, extensions,
etc. Our open source community uses git-flow-avh for example and it has been
extremely helpful. I can't wait to try this out.

------
thesuperbigfrog
Nu looks very promising. Are there any prebuilt binaries available?

~~~
jntrnr1
Not yet, but we'll try to post some soon.

------
quickthrower2
The next step this begs for is for the shell to be like lisp. The results are
s expressions and you can write macros to slice dice and pipeline the data to
your hearts content.

~~~
Sorrop
You may find closh interesting then
[https://github.com/dundalek/closh](https://github.com/dundalek/closh)

Previous HN entry about it:
[https://news.ycombinator.com/item?id=15600928](https://news.ycombinator.com/item?id=15600928)

------
amelius
Do shells exist which don't have a fixed maximum argument size, so you'd never
need "xargs"? Or is the problem innate to Unix?

~~~
codetrotter
> ARG_MAX, maximum length of arguments for a new process

> or why do you get

> command: arg list too long

> [...]

> It's only the exec() system call and its direct variants, which will yield
> this error. They return the corresponding error condition E2BIG
> (<sys/errno.h>). The shell is not to blame, it just delivers this error to
> you.

[https://www.in-ulm.de/~mascheck/various/argmax/](https://www.in-
ulm.de/~mascheck/various/argmax/)

Additionally, see also
[https://unix.stackexchange.com/a/120842](https://unix.stackexchange.com/a/120842)
which speaks a bit about Linux specifics, and how Linux behaves a bit
differently from a lot of the other operating systems in the Unix family with
regards to limits on the length of arguments.

------
djabatt
I just like seeing the fresh hacking in the Unix Shell space. I miss my old
friends sed, awk, and xargs. This is compelling and fun too.

------
rabidrat
This is great! It's like VisiData as a shell. I wonder how easy it would be to
use the VisiData interface with the shell output?

------
simonblack
Why did I keep getting the impression on reading the article that I was seeing
mysql 'select' readouts?

------
farx
What a missed opportunity not naming the post "nushell in a nutshell"

------
ulugh
Cool project! I like the intuitive syntax and clean doc. Will give it a try.

------
KhoomeiK
Am I the only one who first read the title as "nutshell"?

------
techntoke
In the example you have, `ps` takes way too long.

------
lemoncucumber
Could someone explain this to me in a nushell?

------
bigtrakzapzap
Wow. Impressive usability OOTB. Great job!

------
tus88
Job management?

~~~
jntrnr1
Not yet, though we're talking about a few possible approaches.

~~~
OldHand2018
How about a test framework? PowerShell has a unit test framework called Pester
which is installed by default in Windows 10 and Server 2012+.

Anything you can script in PowerShell can have unit tests and code coverage
and it integrates into CI systems.

I don't really like using PowerShell, but built-in unit testing is something I
wish other shells offered. It is a huge plus.

------
boobePhuu7iet7i
Looks stupid

~~~
dang
Would you mind reviewing the site guidelines and sticking to the intended
spirit when posting here? We'd be grateful.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

------
bradknowles
It looks to me like they took the concept of “everything as sql” to the
extreme.

Do you want to live your entire life and everything you do in sql? I don’t.

I think I will pass.

