
Pigshell – Unix the Web - nmcfarl
http://pigshell.com/v/0.6.2/
======
tubelite
Author here.

\- Please use a desktop browser (Chrome preferred). While mobile browsers
kinda-sorta work if you're just clicking through the examples, keyboard input
doesn't work on mobiles (yet).

\- Pigshell has few users (in fact, using the plural is a slight marketing
exaggeration) so please bear with me for any teething issues, broken links,
inadequate documentation...

\- One of the practical use cases is copying files across Google accounts, and
backing it up to your desktop. (Details here:
[http://pigshell.com/v/0.6.2/doc/gdrive.html](http://pigshell.com/v/0.6.2/doc/gdrive.html))
The nice thing about a unix approach is that the exact same cp command can be
used to copy GDrive to Dropbox, or any other supported filesystem.

\- Some more D3 visualization examples here, using Mike Bostock's gist viewer:
[http://bl.ocks.org/ganeshv](http://bl.ocks.org/ganeshv) (Do use the "Open in
a new window" links, like this one:
[http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/](http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/)).

~~~
fsiefken
Hi tubelite, on mobile safari and chrome IOS7.04 with an ipad the keyboard
comes up after I clicked on this example command: pig:/$ cat
[http://pigshell.com/sample/photos/bchips.jpg](http://pigshell.com/sample/photos/bchips.jpg)
I can type in stuff and after enter the keyboard disappears again. A external
bluetooth keyboard might work.

~~~
tubelite
Yes, that happens, but it is awkward and unreliable. The CLI is a CodeMirror
instance (version 2.something), which has known issues with iOS. I need to
check if later versions work better on iOS.

------
tubelite
So if you are running psty, here is a fun sequence of commands to try out:

    
    
      cd http://dilbert.com/strips/comic/1995-06-24/
      ls
      cat 21021.strip.zoom.gif
    

Never really liked the colorization of Dilbert. It was so much nicer as a line
drawing. Best we can do is remove the colors.

    
    
      cat 21021.strip.zoom.gif | to canvas | pixastic -d
    

And to save your result,

    
    
      cat 21021.strip.zoom.gif | to canvas | pixastic -d >/home/dilbert-unix.png
    

Or to GDrive or Dropbox or even the local RamFS by changing the target
directory.

(Scott Adams is fine with personal usage. I checked. And you need psty because
it proxies HTTP requests for sites which don't have CORS enabled)

------
bespoke_engnr
Wow, this is a really interesting way to think about web resources. After
about two minutes of playing around with this, it gave me a more pleasant
experience of the web than I've had in a while.

~~~
anigbrowl
This is what it was like when I first got on the Internet (around 1992); there
was no web then, instead you FTPed everything and used resources like Gopher,
Archie, and Veronica to find documents.

I find PigShell interesting from a historic point of view but think it also
points to a flaw in how the web currently works, with far too much glue and
eye candy, which is a direct result of the ad business model.

~~~
bespoke_engnr
I completely agree. I actually deleted a bunch of my comment before posting
it, because I started getting carried away pondering about what the right
level of abstraction is for dealing with web resources.

At the risk of sounding like one of the semantic-web xml-standards people, it
really does make me wonder if all this horrible, incompatible, site-specific
frontend crap is just a fad that will give way to tools (like psty) that
ignore all that and just deal with the raw semantic content (and HTTP verbs)
at a higher level of abstraction.

Woah, man. We're like... _surfing_ the Matrix, man.

~~~
svalorzen
Things like this really make me miss the old Firefox experiment "Ubiquity". It
was truly a marvel for the time it was up, and I often wonder why they did
discontinue it.

You could search websites, send emails, look at maps and much more (you could
build your own commands) without ever actually opening a webpage. It had
200.000 users when they shut it down.

~~~
jacobwcarlson
Wouldn't you just be able to fork it and continue as a hobby project of your
own? Who knows, maybe you find other enthusiasts and it becomes a real project
again. Mozilla licenses are (I believe and could be wrong), quite lenient
about these things.

~~~
Dewie
This is what I found by looking at its Wikipedia article:

[https://bitbucket.org/satyr/ubiquity/downloads](https://bitbucket.org/satyr/ubiquity/downloads)

------
fiatjaf
How is the procedure for "installing" new programs? Adding a local directory
with javascript commands to the PATH will do it?

What about saving the state of things. I want, for example, to `mount` my
local directory with psty everytime I open pigshell, and I wanna have my PATH
set automatically. Will a `/local/.config/.pigrc` do this? (This raises
another question: file editing. Where is the file editing tool? Maybe a web-
based editor somewhere). Well, maybe a gist will do it.

~~~
tubelite
There are two kinds of commands: Javascript commands (like ls, cat) and
scripts (ps, kill, man, csv2js, table2js).

Currently, Javascript commands have to be built-in at "compilation" time. (I
need to add a load command which will evaluate a new JS command inside the
pigshell closure)

Scripts can be anywhere in PATH.

There is an rc mechanism for doing exactly what you want. At boot time, we run
/etc/profile, followed by /local/rc.sh - this is a file saved in the browser's
localStorage. Mine is very simple:

    
    
      HOME=/home
      mount http://localhost:50937/ $HOME
    

You can create it simply by

    
    
      echo "HOME=/home\nmount http://localhost:50937/ $HOME" >/local/rc.sh
    

or running "edit /local/rc.sh". Edit provides a simple inline CodeMirror-based
editor (actually even the CLI is a CodeMirror editor instance). I use vim on
the desktop to do any serious editing. If you run psty, /home is a desktop
directory visible inside pigshell, so edit on desktop/run in shell is a low-
friction process.

If you manage to mess up /local/rc.sh to the point where pigshell is not
"booting", then you can boot it in "safe mode" by appending #norc to the URL,
i.e. [http://pigshell.com/#norc](http://pigshell.com/#norc)

~~~
progman
I downloaded
[http://pigshell.com/v/0.6.2/psty.py](http://pigshell.com/v/0.6.2/psty.py),
started it with "python psty.py -a" and did "mount
[http://localhost:50937/](http://localhost:50937/) /dir" successfully.

Unfortunately only "ls /dir/file" works but neither "cat /dir/file" nor "cp
/dir/file ." although "file" was marked with "chmod a+r". I get the message
"Cross-origin request denied (check if psty is running)". What do I wrong?

~~~
tubelite
Argh! This is a bug. Has been fixed in master
([https://github.com/pigshell/pigshell/commit/f402ef445977015a...](https://github.com/pigshell/pigshell/commit/f402ef445977015afd0f6a9ea2b10e9facfa6a9e))

As a workaround in 0.6.2, please use a trailing slash in the name of the mount
point. ( /dir/ rather than /dir)

Alternately, use
[http://pigshell.com/v/0.6.3-pre4/](http://pigshell.com/v/0.6.3-pre4/) which
would have the fix.

~~~
progman
Unfortunately these suggestions don't work also. Could the problem be in
psty.py? Or is my Python 2.7 obsolete? There is a strange error message in
psty:

EXCEPT timestamp out of range for platform time_t Traceback (most recent call
last): File "psty.py", line 404, in send_head self.send_header("Last-
Modified", self.date_time_string(entry["mtime"])) File
"/usr/lib/python2.7/BaseHTTPServer.py", line 468, in date_time_string year,
month, day, hh, mm, ss, wd, y, z = time.gmtime(timestamp) ValueError:
timestamp out of range for platform time_t

~~~
tubelite
This is a new bug. I'm guessing it got exposed because you have a 32-bit
machine, and I've been testing so far on 64-bit. Edit line 404 of psty.py as
follows:

    
    
      -            self.send_header("Last-Modified", self.date_time_string(entry["mtime"]))
      +            self.send_header("Last-Modified", self.date_time_string(entry["mtime"] / 1000))
    

If it still doesn't work, maybe we can continue the conversation on the google
groups forum?

~~~
progman
I have a 64-bit machine but your fix works great :-)

Thanks for this, and thanks for sharing your amazing pigshell!

------
nmcfarl
More in this comment by the author:
[https://news.ycombinator.com/item?id=8436716](https://news.ycombinator.com/item?id=8436716)

------
Erwin
Interesting. I'm trying to really go out all the API for my project and was
thinking about something like a Mathemathica-like notebook for the local API.
E.g. say you want to execute an API call which normally does GET
/api/v1/x/y/z; in the "API shell" you type x/y/z and the JSON collection is
summarized as a small datatable (not showing all of the 500 rows / 40 columns
that might have, but an interesting summary) that you can later interact with.

I'm thinking that such an approach can give the user their own way to
explore/modify the data ,especially if you combine this with some kind of pipe
lines that let you analyze/mutate/visualize the data without having to write
some Python code to access the API (even though with a REST API plus e.g.
requests it's simple).

I like the psty idea to integrate with the local system. Sounds like it could
also be used to start e.g. emacs on a remote page. Say you have a site with an
API that lets you read/write files, you mount it and then can fire up emacs on
it via pigshell.

~~~
privong
Is this approach similar to something like flickrfs[0], to make one's flickr
account locally accessible? It always seemed like a cool idea (though I never
had much luck with flickrfs itself), and it is neat to see the broader utility
of psty.

[0] [https://launchpad.net/flickrfs](https://launchpad.net/flickrfs)

------
bilalhusain
The psty links in README file are broken. A working link is
[http://pigshell.com/v/0.6.2/psty.py](http://pigshell.com/v/0.6.2/psty.py)

As mentioned in the README, you need to run psty server locally and establish
websocket connection (mount in pigshell). This will enable you to harness
local filesystem from pigshell.

------
tomw1808
I just had a glance about the functionality, but for my understanding, this is
the best idea right behind ifttt. Pipelining all the web's things with well
designed interfaces on a shell. Even if the product itself would be unstable,
alone the idea is great enough for a gold-medal. Keep it up!

------
vezzy-fnord
So in many ways, this is like having a FUSE/Hurd translator implementation
directly from the browser, though it also provides a mini-Unix environment due
to needing to implement corollaries to some Unix utilities.

The scope of this project is definitely what makes it nifty.

------
diltonm
It's very cool, gets back to using the power of the simplicity of UNIX. I
found though, that X middle click paste doesn't work.

I made a directory at /mnt/mccomsoft then ran Firefox and didn't see it which
is good. :) Then I tried mounting that using your HttpFS with:

    
    
      mount http://www.mccomsoft.com/index.html /mnt/mccomsoft
    

It gave me:

    
    
      Cross-origin request denied (check if psty is running)
    

Which is also good. :) If I load Pigshell from my domain though then it should
mount fine.

One user commented below it's like the Internet was a shell and you've cracked
it open. I agree.

------
progman
Congratulations!

The way how you implemented "man" ("man ls" for instance) shows how a
convenient shell can look like. Pigshell could be a foundation for a
completely new kind of toolchain for web development. I am thinking of
seemless integration of JS editors, D3 graphics for pipe output, etc. ... Wait
- it's already working :-)

[http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/](http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/)

If it just were possible to access the bare bone hardware so that we wouldn't
be restricted to the JS world (in a closed environment, of course).

~~~
pmontra
The way the man page is formatted is great but I don't like the scrollbar
inside a scrollbar effect. It's been traditionally a worst practice. It's
better to have the man page take all the vertical space it needs in the main
frame and add a Close button to the end of the man.

By the way, from
[http://pigshell.com/v/0.6.2/doc/psty.html](http://pigshell.com/v/0.6.2/doc/psty.html)
"Websocket server: Every unix app on the desktop which uses stdin/stdout can
participate in a pigshell pipeline" so maybe your request has already been
implemented.

~~~
progman
> By the way, ...

Thanks, meanwhile I have realized that also :-)

------
aidos
Kudos, tubelite. This is really cool. I'm not sure I'd ever really use it but
it's been fun to play with. Definitely made me think that I'd like to use the
web in a more command based fashion.

------
dllthomas
Have you given thought to exposing this to the regular shell with FUSE?

~~~
tubelite
The concept of lazy-stream-of-JS-objects and HTML as a display engine underpin
pigshell.

I guess it would be possible to adapt the filesystem code to fuse4js. It won't
be a drop-in; unix FS ops differ from pigshell's, but it shouldn't be too
hard. Personally, I find psty
([http://pigshell.com/v/0.6.2/doc/psty.html](http://pigshell.com/v/0.6.2/doc/psty.html))
a better way to incorporate (a part of) the unix filesystem into the web
rather than fuse - which is the other way round.

~~~
clebio
Isn't this -- at least in spirit -- what IPython notebook tries to do? In
IPython, you can use shell commands with a bang, e.g. `!ls -l`. And `ipython
notebook` uses ZMQ to connect the server and client.

At any rate, awesome work. I've been thinking for a while, offhandedly, about
the idea of piping web URLs around. But I don't know how in fact it would work
under the hood. I write a lot of API interfaces in my work, and it's all in
the details -- every API is different in small but important ways.

~~~
tubelite
Yes, it is similar in spirit to IPython.

We have been developing pigshell off and on since ~2012, and I was unaware of
IPython notebooks until ~6 months ago. When I'd last used IPython, it didn't
have any of the notebook stuff. So it was a bit of a jar to see that they had
been there, done that, got the T-shirt, published the book, etc. etc. :)

Pigshell can run in the browser with zero installation, and has a low barrier
for casual usage. "Notebooks" aka gists can be shared without requiring a
backend. The focus of pigshell is more around providing file adapters for
web/cloud data stores.

That said - more power to IPython! I love the conversational, exploratory, CLI
style of interacting with a computing environment. The more ecosystems of this
sort, the better.

------
dullcrisp
I do love this idea, but it bothers me from a security standpoint. Is there a
way to do something like this without giving the author explicit control of
all of my online data?

~~~
tubelite
The author has NO control whatsoever of your data.

\- Pigshell is a 100% client-side Javascript app. There is no server side.
pigshell.com is a dumb static html/js/css server.

\- All data (username/password as well as user data like photos and files)
travels directly between the user's browser and the provider (e.g. google.com,
facebook.com). pigshell.com is not involved at all.

\- Pigshell is free software - you are free to download the source code,
examine, modify and run it locally yourself.
([https://github.com/pigshell/pigshell](https://github.com/pigshell/pigshell))

\- Our privacy policy is minimal because we know NOTHING about you - not your
name, email address, facebook id, nothing. Only your IP address is recorded in
apache logs, which is the case with every website you visit.

What is more, _we don 't want to know_. Pigshell has been deliberately
designed so as to afford maximum privacy and freedom to the end user.

~~~
dullcrisp
I don't really doubt you or your intentions, but I guess I might just not
understand how authentication works for something like this.

When I try to sign in with Google Drive, it tells me that PigShell (developer
email: xxxx@gmail.com) would like access to my Google Drive, as well as my
photos and videos. Does that give your API key access to my account, or does
only code that I run that's hosted on pigshell.com have access? What do the
permissions get tied to? And how would it work if I were to host it locally?

~~~
tubelite
We use OAuth2 client-side flow. Many pure client-side apps use the same
technique - pigshell is not blazing any new trails here.

Only the code you run that's hosted on pigshell.com has access. The
permissions are tied to the app-id which is embedded in the code. The access
token is persisted in your browser either as a cookie or explicitly in
localStorage.

We don't and won't support OAuth 1 (Twitter, Flickr etc), which lacks a pure
client-side flow, just to avoid the issue of users having to trust the
pigshell.com server to generate (and not leak or misuse) the access tokens.

Here is a rough guide to local setup:

\- Check out the git sources, run "make" (some more details here, but reading
the Makefile should help)

\- set up apache to serve the virtual host pigshell.com (if you want to use
Dropbox, you need to create a self-signed SSL certificate and set up https as
well)

\- modify /etc/hosts and set 127.0.0.1 to point to pigshell.com

This way, static assets as well as redirect URLs from the OAuth2 server will
hit your local server rather than pigshell.com

------
talles
> Error loading root filesystem at
> [http://pigshell.com/v/0.6.2/root.tar](http://pigshell.com/v/0.6.2/root.tar):
> Forbidden

:(

------
crater
I like the idea. If I'm playing with some facebook data, right now I use
whatever language I'm confortable with and use its HTTP request functionality,
and than manipulate the data how I want in that language. To achieve the same
thing in pigshell I have to write a shell script that manipulates the data.
What can pigshell do that I can't just do in another language?

~~~
tubelite
In theory, the availability of APIs should be enough for the technical user to
write their own data extraction routines, connect them to other APIs and so
on, using their preferred language and toolchain. In practice, reading up the
documentation, writing and debugging code sets up enough of a barrier to entry
that it discourages casual usage without a powerful incentive.

The availability of a file abstraction covering many popular cloud services,
web resources etc. enables casual, low cost experimentation _across_ these
domains, without having to read up each one of the APIs. If something clicks,
then it may make sense to write a better "native" implementation.

------
jayseeg
With some practice, or even better, some support libraries, this looks like a
great way to quickly explore viz prototypes. Great work!

------
roywiggins
Attempting to copy photos out of Facebook albums into GDrive gives me
"Exception: SecurityError: The operation is insecure." I'm not running psty
locally.

This is super cool, though. There's an undocumented template engine built in
it seems? (see /bin/mostliked) which is neat.

------
TimWolla
Does not work in Internet Explorer 11: Exception: The property "slice" of an
undefined or null referance cannot be accessed (Exception: Die Eigenschaft
"slice" eines undefinierten oder Nullverweises kann nicht abgerufen werden.)

~~~
tubelite
Sorry, Internet Explorer is not supported currently. Please try Chrome or
Firefox, in that order.

------
ConSeannery
This is awesome. It reminds me of a talk I saw recently, linked from
hackernews at some point in the last year or so. Something about re-thinking
internet infrastructure into a "directory-tree" like model. Does anyone
remember it?

~~~
oftenwrong
I don't, but I am interested if you find it. We already have a hierarchy in a
way.

The domain tree goes from top to bottom this way <\---

The resource tree goes from top to bottom this way --->

So, using numbering for order, you could look at a URI like:

0002.0001.0000/0003/0004/0005

Meanwhile, in an alternate reality...

    
    
        $ ls /net | shuf -n 16
        coffee
        net
        рф
        한국
        org
        dating
        pw
        中國
        institute
        இலங்கை
        ninja
        com
        みんな
        su
        政务

------
itazula
Wow, this was like the nutshell cracked and the universe spilled out ...

------
linvin
Ah! Nice feeling that whole of internet as a single (unix) machine.

~~~
serf
reminds me of Plan9 sorta.

------
omouse
I like the sentiment, however I think it'd be cooler if you could get the
Facebook, Google, and other APIs to behave in the "everything is a file"
tradition of UNIX.

------
shykes
Looks a lot like webshell ([http://webshell.io](http://webshell.io)). I wonder
if there are other similar tools out there?

------
fiatjaf
I'm curious about how are people using this. Is it just a visualization tool
for data or what else? Is anyone using it as their primary navigation tool?

~~~
basantrajan
perhaps stating the obvious here, but with the abstraction pigshell provides
of resources on the web, it should be reasonably straight forward to build
arbitrary "internet applications" too.

with deferred pipes, split and joins (that pigshell already provides/supports)
it should be possible to implement arbitrary work/data flow definitions one
desires.

coupled with psty, one could build/harness special purpose "transformers" on
the web

pushing it further, one could conceive of setting up pipes between
transformers on the web.

of course getting there will require significant work on a security model as
well.

[and then someone will figure out a way to interject ads]

~~~
progman
> of course getting there will require significant work on a security model as
> well.

It could probably be easier than you imagine. The security model could be
implemented in psty completely by granting/denying requested resources whose
permissions are stored in a database on the psty server, and managed by using
long security IDs in the URLs. This way the security model could even be fine-
grained. In other words, pigshell itself could be embedded into a secure
environment, covered by the permissions of psty.

------
damian2000
My connection is shaped at 128 kbps at the moment - am grateful that this
loads really quickly. Awesome stuff. HN itself is really low bandwidth too.

------
kolev
This is beautiful!

------
drivingmenuts
I have no idea what I would use this for, but I really wish I did.

Very cool.

------
altrego99
How do I know what are all the fields available on a file?

~~~
tubelite
Here is one way:

    
    
      ls -d /path/to/file_or_dir | printf
    

This command prints the stack of file objects. The last one printed is at the
"top" of the stack and tells you the fields available. You can then print a
field

    
    
      ls -d /gdrive/user@gmail.com/somedoc | printf "%(raw.ownerNames)s"
    

Sorry, a lot more documentation is needed. Working on it.

------
codehero
I typed in `oink` but the command was not found...

------
kybernetyk

        rm -r facebook
    

ah, that feels good.

------
mgrassotti
holy crap this is awesome

------
joseph8th
A new toy! Thanks!

