
Import – A Simple and Fast Module System for Bash and Other Unix Shells - pcr910303
https://import.pw/
======
ary

      # The URL is downloaded once, cached forever, and then sourced
      import "https://git.io/fAWiz"
    

Call me old and cranky, but I thought this naive fad of piping the unchecked
contents of URLs to the shell would die off eventually once people got clued
in to how dangerous it is. At a bare minimum the contents of any shell script
coming from a remote source should be manually reviewed the first time, and
then verified to have not changed by comparing against a locally stored
cryptographic hash.

This is no better than curl with some automatic caching. At the end of the day
why not store your dependencies along side your scripts and then not worry
about it changing unless you change it yourself?

~~~
treve
Piping a url into bash is not more or less risky than downloading a windows
binary, installing a snap, installing a npm/composer/whatever package with
post-install hooks, downloading a binary for your Mac.

It's impossible to read every line of code we execute (many of which are
closed source anyway).

Downloading a large script and reading it before running it in bash is also
hardly good security measure.

It's much more practical to teach people how to recognize trustworthy sources,
and how HTTPS works. Or you can switch to a platform like iOS where there's
complete control, sandboxes and reviews.

~~~
ary
The risks involved with executing code are legion, and wanting to promote
cultural change in the direction of better safety isn't invalidated by
pointing out failings in other areas.

> Downloading a large script and reading it before running it in bash is also
> hardly good security measure.

> It's much more practical to teach people how to recognize trustworthy
> sources, and how HTTPS works. Or you can switch to a platform like iOS where
> there's complete control, sandboxes and reviews.

Downloading bash scripts and inspecting them before running _is definitely_ a
good security measure, as is running untrusted software in a VM or container
where possible, and generally promoting caution when running anything acquired
from the Internet. Even if the source is considered trustworthy there is still
the possibility that it has been compromised. npm, pypi, and even the Linux
kernel source tree are all examples of this.

In this particular situation the use case is extremely weak. The problem is
solved by incorporating dependencies locally, as I pointed out above.

~~~
treve
This is all fair, it just sometimes feel people are a little too focused on
running bash scripts, when something like `npm i create-react-app` downloads
200K lines of code, all of which can autorun. 'Read everything you download'
is just not incredibly helpful. From all the code we trust and not read is
just a drop in the ocean.

~~~
_frkl
I totally agree, and never understood the focus on curly bash script. People
seem to be ok to 'git clone; make; sudo make install' something, but hesitate
to pipe a remote bash script that comes from the same author, and often is
hosted in the exact same repo... At least, nobody ever specifically pointed
out how dangerous the make scenario is.

There are a few minor things one should do when writing such a curly bash
script, but overall I, personally, don't worry too much about it when I
encounter them, I just do my usual security assessment and risk mitigation
(which usually does not include reading the code -- that's just not
practical).

~~~
craftinator
I think it's the programmer's equivalent of "get off my lawn you kids" or "if
you make that face for too long, it'll get stuck like that forever".

------
kazinator

       # The URL is downloaded once, cached forever, and then sourced
       import "https://git.io/fAWiz"
    

Where is this program declaring the above content's expected SHA256 to check
that the expected content is being received, and not some altered version?

Think about it. You write the program with the above #import, and at the time,
it fetches a certain piece of code. You see in your cache that it's the right
one.

Then you deploy the script to a fellow user. They run it, and the import
fetches something different from the same URL. The script breaks. You're
scratching your head: hey, it works on my machine, why not yours? Or worse:
the replacement code is _malicious_ not simply _breaking_. The user's files
are gone or whatever.

~~~
Spivak
Don't get me wrong, that's would be a nice feature, but I think it's a little
optimistic that people would actually use it and ask for more security outside
of TLS. I mean the most popular (n)vim plug-in managers just clone arbitrary
git repos with no other verification.

Code signing or hashing is vanishingly rare and a huge burden on the end user
who simply, unfortunately, not care in the slightest. The "good enough" root
of trust is that the server hosting the code hasn't been compromised.

~~~
kazinator
Far from this being rare, every major GNU/Linux distro checks the hashes of
downloaded material it depends on. Failing to do this threatens
integrity/stability of the system and provides an open door for malware.

Vim package managers pulling git repos can easily check out a specific hash;
if they are just blindly following branch heads by name, that's just dumb and
nobody should emulate their behavior.

TLS is neither here nor there because there isn't much of a privacy concern.
Plaintext HTTP works fine for fetching some piece of FOSS, if you check the
digest to verify that you're getting what you're expecting to be getting.

On the contrary, the assurance that the HTTPS host key has not changed is not
relevant. It does not amount to (is something other than) the relevant
assurance that the URL content has not changed.

Moreover, there may be numerous hosts which provide that exact same content.
If you have the digest, and the original host stops providing the content
reliably, you can confidently switch to a different mirror, which could be a
HTTPS host you have never contacted, or even a HTTP one.

The trust that the server hosting the code has not been compromised is
misplaced and foolish. The person in charge of that content could be someone
who thinks it is a good idea to change the content of a software release (to
sneak in one more bugfix) without changing anything in the URL. Thus two of
your own users who think they are using the exact same version of your
software will fetch the same foolib-1.2.tar.gz URL a day apart, and get
different files, which breaks the integrity of your own release. Without
hashes, this goes undetected.

There is no "huge burden" here. All the module tool has to do is to make it
mandatory for the fetching commmand to have a digest argument. If the digest
doesn't match the content, all it has to do is fail that command, and issue a
diagnostic. The diagnostic can include the actual hash that would work, and
point to the file and line number where the failing import is located.

The user relies on hundreds of imports that they didn't write themselves, and
which require no attention at all in the happy case that they fetch what they
are supposed to fetch.

~~~
Spivak
I don't think this detracts from your point but despite being technically
true, it doesn't imply what you think. Linux package managers really verify
the signature of the hash which is downloaded from the same server and then
compare it against the package. Code signing is the real root of trust here,
not hashing.

You can't escape the total insecurity of git cloning because a malicious actor
will just create a branch with the name of the desired hash and every client
will happily clone it. You are pretty much always trusting the server unless
you clone and _then_ verify the content. Plus pretty much all "language"
package managers and brew do the "dumb" thing since you only usually specify
name and maybe version in your dep file.

I'm not saying that this isn't a good idea, but there's a long long list of
examples where extremely popular code managers/downloaders that simply don't
care about this.

~~~
kazinator
> _Linux package managers really verify the signature of the hash_

> _Code signing is the real root of trust here, not hashing._

Hashing is the root of that specific trust which says "this is the same thing
it was 17 months ago when we first fetched it".

(It is not deductively implied; it is predicated on the difficulty of
producing a rogue file that has the same hash.)

Maybe it was a fake then, loaded with malware; that's a related, though
different concern.

The fact is that I can inspect something, like a module of shell script code,
to my satisfaction to be free of problems, and debug all my stuff with it. The
hash then gives a lot of assurance that it's the same thing I validated
before.

> _You can 't escape the total insecurity of git cloning because a malicious
> actor will just create a branch with the name of the desired hash and every
> client will happily clone it_

If a branch or tag can be created that has a name indistinguishable from the
syntax of a hash, that's a disturbing ambiguity in git that should be arguably
patched away. It is tangential to this topic.

I believe, perhaps mistakenly, that after the repository is cloned and checked
out (by any means: hash or branch/release tag), a validation can be made that
HEAD points to the expected commit, e.g. using

    
    
       git rev-parse --verify HEAD^{commit}
    

That should defeat the ruse.

Language managers are based on a vested interest in popularizing a language,
by maximizing the amount of activity occurring in its "ecosystem". If every
dependency were pinned down by a hash, such that the upstream developers have
to beg everyone to update to a new hash every time they want to publish some
change, that would put a huge a damper on the party. Language package managers
are not motivated by stability, reliability and security. I'm not in favor of
language package managers, and none of my work or side projects depend on
them.

------
remix2000
Impressive…

How people manage to poison even old-school shell scripts with useless bloat.

If you want to source remote scripts that badly (despite this being against
policies of almost every company), better keep it simple:

    
    
        telesource() { f=~/.cache/${1/\//} ; [ -f "$f" ] || curl -SsLo "$f" "$1" && source "$f" ; }
    

So you can invoke it like that:

    
    
        # The URL is downloaded once, cached forever, and then sourced
        telesource https://git.io/fAWiz
    

What does the name ``import`` even mean? Does it anyhow relate to sourcing? Or
even shell scripting at all?

Now I'm waiting for a full JIT ECMAScript VM in pure portable Bash. Impress me
even more ;)

Nonetheless, that's what it looks like to me. I'm just not fond of spilling
idioms from one technology to everything else its users touch. That's much
like our IT equivalent of Midas touch.

~~~
emiliovesprini
> What does the name ``import`` even mean?

 _import_ name-clashes with ImageMagick's import, which comes pre installed in
lots of Linux distros.

Issue (by me):
[https://github.com/importpw/import/issues/35](https://github.com/importpw/import/issues/35)

------
edw
Color me old, but wtf, the JS script kiddie obsession of never writing a line
of code that can be packaged in a module has now come to shell scripting?!

~~~
choeger
Hello old, nice to meet you. I am skeptical.

------
varbhat
I think that this is bad idea and it increases non-working scripts(what if url
stops working? ) . Also, it increases malwares.(what if url pointed to is
changed to malware script?)

Golang's package management is very good, but this isn't.

Why can't they just paste the required script in main script itself? What's
the necessity of import here?

------
meddlepal
Never change below your feet unless your machines are ephemeral, constantly
changing, and thus do not have a permanent cache.

Do not use this in production folks.

~~~
SrslyJosh
The bare minimum for saying "never change" involves a cryptographically secure
checksum of the content, and even then it's not guaranteed.

------
alganet
I did a similar thingy a while back:

[https://github.com/alganet/coral/blob/master/doc/spec/requir...](https://github.com/alganet/coral/blob/master/doc/spec/require.md)

It also does references (`require `something from inside modules) and some
stuff. It is missing the download component though.

I've also worked on
[https://github.com/Mosai/workshop](https://github.com/Mosai/workshop), a
previous attempt at shell tooling (please do not use it!)

Also, there was a popular
[https://github.com/pixelomer/bashpm](https://github.com/pixelomer/bashpm) at
some point.

Most stable thing would probably be: [https://github.com/shellfire-
dev](https://github.com/shellfire-dev)

Here are my thoughts working in this "hobby" for the last 6 years:

\- External tools are bad for shell portability and performance. It costs more
to invoke "cut" than to do a loop and a couple of parameter expansions. Tools
like cut, sed and similar are not suited for small strings, they are for large
files.

\- ksh93 and ksh2020 support is possible, see:
[https://gist.github.com/bk2204/9dd24df0e4499a02a300578ebdca4...](https://gist.github.com/bk2204/9dd24df0e4499a02a300578ebdca4728)
These kshs are __fast __, I want them as supported engines.

\- Pipes are bad for performance. Local variable buffer is faster in all
shells. There is a proof here somewhere: [https://github.com/alganet/shell-
proto](https://github.com/alganet/shell-proto)

\- We need to create a shell "dialect". Leave a lot of stuff out to simplify
its syntax, so we can write a shell parser written in pure shell. This is a
crucial point towards a better shell ecosystem.

I have some research related to these issues, I'd love to collaborate in
something. Standards, maybe? I think the shell community needs to talk as much
as the early JS community did in the past.

------
risson
I hate the way a lot of tools nowadays assume it's okay to just create a new
directory in the user's home. For instance, this tools create ~/.import-cache.
Why doesn't it use a subdirectory in ~/.cache? What's the next step?
~/.import-config? And then ~/import-tmp? This is just getting out of hand.
There are places where your tool can store config and cache data, use them.

~~~
augustk
Or even better: $XDG_CACHE_HOME

[https://specifications.freedesktop.org/basedir-
spec/basedir-...](https://specifications.freedesktop.org/basedir-spec/basedir-
spec-latest.html)

------
price
Exciting idea!

For me the application of this that I'm most interested in personally is for
dotfiles, configuring the interactive shell. I have a fair amount of useful
stuff in my own bashrc that I've developed over the years, which people who
see me work often want to use too (and I happily share). I've wished for a
good way to share each logically-independent customization in its own separate
chunk -- so that people can pick and choose, and also combine what they like
from me with their own existing config and chunks they've picked up elsewhere.

For zsh, oh-my-zsh seems to have been extremely successful, and IIUC this
describes a key technical contribution at the core of how it's worked so well.

So I'm hoping that this thing can play part of the role of oh-my-zsh, for
bash! (And for other shells too, sure.)

I'll definitely be interested to read more of the details of how it works, and
to play around with it.

~~~
price
(Before using this for real, I'd certainly want to be able to track the exact
version I'm getting of a dependency, with a hash, in a "lockfile" sort of file
that I can keep in version control. That's not mentioned in the demo on the
front page - but if it doesn't already have that feature, I hope it will grow
it.)

~~~
JoshTriplett
If you're pinning a shell script to a specific hash, why not just keep the
script itself in version control?

~~~
price
I'm thinking especially of the use case of sharing for people who have not
been accumulating dotfiles forever and haven't set up some kind of way to
manage their dotfiles in version control. (Heck, I don't myself have a system
that I _love_ for that, though it meets my basic requirements in that I can
always ask Git exactly what changed.)

So "add this one line to get my prompt" is then a much lower-friction
suggestion than "copy this whole little script".

------
aeyes
Nice!

Who says that you have to download from arbitrary URLs? If you have to use
public modules you are free to copy the source to some internal server first.
It's just a single file!

It looks like modules can't import other modules so this shouldn't get too out
of hand.

------
idclip
this makes me uncomfortable...

reasons:

1\. i dont believe package systems help bash. we already have ansible for
systems ... if this goes the way of npm, bash will get bloat and chaos.

2\. shells are a system process thats usually very privileged .. i see alot of
abuse potential since it interprets everything you source.

3\. see 1, and think performance on medium sized scripts, or if an expensive
"module" is loaded.

4\. versioning chaos, killing old scripts ... in the unix and linux space,
some decade olds scripts are still used.

this would expose weaknesses ... but also destabilise a lot.

edit: 3,4, also thank you regardless of my opinion, your code and execution
look and feels great.

------
JoshTriplett
Seems to be failing at the moment:
[https://imgur.com/a/UEbJBKA](https://imgur.com/a/UEbJBKA)

~~~
boogies
It was the same for me the first time I tried, but the "View on GitHub" link
at the bottom still worked (and now the main page works too).

------
coredog64
At my last employer (large US bank you’ve definitely heard of), they had
something like this internally. Tools/scripts were packaged up as modules and
you could/would “module load foo”. It knew what OS you were running, so
Windows users could do the same thing for cross-platform tools.

It was very handy and I’m looking forward to trying this as a substitute.

------
maddyboo
This would be perfect if it also included a bundler/linker to build single-
file executables.

I also find it amusing each time the argument against using URLs as import
specs comes up. Go already does this, we do it on the web with <script> tags,
and now Deno is doing it. It's proven to work as long as you can reasonably
trust the URL.

------
earthboundkid
Your scientists were so preoccupied with whether or not they could &c

------
lma21
what's wrong with

    
    
        source <file_path>

~~~
kasabali
Not webscale /s

