
File Transfer with SSH, Tee, and Base64 - susam
https://susam.in/blog/file-transfer-with-ssh-tee-and-base64/
======
rsync
This is a problem I used to solve with 'zmodem' which you can install on both
ends and use to perform text transfers, solely over the interactive channel.

I think the binaries were named 'zs' and 'zr'. It's been a _long time_ since I
needed to do this anywhere...

~~~
EvanAnderson
You're close on the names of the binaries. They're 'sz' and 'rz'.

I used the crap out of them dialing-in to systems back in the day. I had
TELNET and FTP access to the Internet out of a dial-in to a shell on a Unix
host in the mid 90s (no SLIP, PPP, or SLiRP from the provider at the time. It
was great to be able to ftp down files to my home directory then 'sz' them
back home.

~~~
wahern
I wish terminal emulators still supported zmodem natively like some of the
better emulators many years ago. It's not too difficult to configure iTerm2 to
invoke rz when detecting the escape sequence[1], but it's still too much
friction. sz/rz is something I would be using everyday, but I prefer sticking
to Terminal.app on macOS--I prefer zero-config environments because of all the
various systems, local and remote, I interact with on a daily basis.

scp and sftp are great, but the nice thing about in-band file transfers is
that your shell session already has the current context for simplifying the
transfer. For example, you often want to send to or receive from the current
working directory, or something nearby, and interactive file globbing is just
a single tab key away, without having to switch sessions, login all over
again, copy paths, etc.

[1] I don't miss automatic printing without confirmation, though. The prank of
`cat /vmlinuz | write user` wasn't that funny when the service desk hands you
an entire ream of garbage along with a reprimand.

------
silviot
My solution in these cases:

    
    
        ssh hostname tar cvjf - /path/to/folder | tar xjf -
    

Basically I ask ssh to execute tar on the remote host to create a compressed
archive. ssh will output the archive contents on the local host; this data is
then passed on to a local tar for extraction.

~~~
susam
I do mention in my blog post that the SSH gateway in between forbids execution
of remote commands without a login shell, so this rules this out as the
solution. Otherwise, yes, this would have been a fine solution.

~~~
tssva
Would this make it through the SSH gateway? It creates a pseudo TTY and
executes bash as a login shell within it.

    
    
      ssh hostname -qt 'bash -lc "tar czf - /path/to/folder 2>/dev/null | base64"' | base64 -id | tar xvzf -

~~~
wahern
Probably want to prepend a prefix so you can extract only those lines and
filter banners, motd's, etc, like:

    
    
      "... base64 | sed -e 's/^/~/'" | sed -ne 's/^~//p' | base64 ...`
    

BTW, base64 isn't a standard command (nonexistent on OpenBSD), and the options
can vary based on implementation even when it exists (decode is -d for GNU
coreutils base64, but -D for the macOS command). You can whip up an
alternative using od(1) and printf(1):

    
    
      #!/bin/sh
      # encode as backslash-escaped octal (\0NNN)
      export LC_ALL=C
      od -An -to1 -v \
      | sed -ne 's/\([01234567][01234567]*\)/\\0\1/gp' \
      | tr -cd '\\01234567\n'
    
      #!/bin/sh
      # decode backslash-escaped octal
      export LC_ALL=C
      while read -r L; do
        printf "%b" "$L"
      done
    

It might not be strictly POSIX compliant, but I don't think there are any
environments where printf isn't 8-bit clean in the C locale. I've tested
something similar (see below) on various versions of bash and d?ash, as well
as /bin/sh on AIX, FreeBSD, macOS, OpenBSD, NetBSD, and Solaris.

You can implement base64 as a shell script using the above techniques, and do
so without relying on awk, which isn't always available. If od(1) isn't
available you can use dd(1) to extract stdin byte-by-byte and convert with
printf(1), but that gets really slow. I've written an implementation that only
needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either
dd or od. It doesn't need tr, but it's even slower without it. It just
converts the bytes to decimal and uses plain shell arithmetic for the
primitive encoding and decoding, similar to how it would be done in any
language.

~~~
tssva
It certainly isn't a complete cross platform solution but to address the
inconsistent options problem and some of the availability issues openssl could
be used instead.

    
    
      ... | openssl enc -base64" | openssl -base64 -d | ...

~~~
tssva
Self reply. Noticed I left out the enc option from the 2nd openssl command.
Should be:

    
    
      ... | openssl enc -base64" | openssl enc -base64 -d | ...

------
djent
This seems like a CTF trick. If I were faced with this problem, I would do an
SSH local port forward, and then just wget the file, hosted locally with
python -m SimpleHTTPServer

~~~
livre
I think we can assume that if they are restricting things as the article
describes they also disabled port forwarding.

~~~
susam
That's right. See
[https://news.ycombinator.com/item?id=24507146](https://news.ycombinator.com/item?id=24507146)
for example.

------
nrclark
The instructions in this article should work, but they're more complicated
than necessary maybe. If you don't have scp, you can usually just do something
like this:

    
    
      ssh user@host cat remote_file > local_file
    

to copy the contents of "remote_file" on the server to "local_file" on your
local machine.

~~~
Fnoord
> If you don't have scp [...]

scp is deprecated [1]; use sftp instead. At the very least, sftp should be
your preference over scp.

[1] [https://unix.stackexchange.com/questions/571293/is-scp-
unsaf...](https://unix.stackexchange.com/questions/571293/is-scp-unsafe-
should-it-be-replaced-with-sftp)

~~~
fulafel
Your link doesn't say it's deprecated...

the sftp command has lousy ux, it's unlikely people will switch to it en
masse.

~~~
Fnoord
> The scp protocol is outdated, inflexible and not readily fixed. We recommend
> the use of more modern protocols like sftp and rsync for file transfer
> instead. [1]

> From The Jargon File (version 4.4.7, 29 Dec 2003) [jargon]:

> deprecated adj.

> Said of a program or feature that is considered obsolescent and in the
> process of being phased out, usually in favor of a specified replacement.
> [..]

[1]
[https://www.openssh.com/txt/release-8.0](https://www.openssh.com/txt/release-8.0)

~~~
fulafel
But your cited definition agrees with me? It's not being phased out, hence not
being deprecated.

~~~
V6HBGNQHU
> The scp protocol is outdated, inflexible and not readily fixed. We recommend
> the use of more modern protocols like sftp and rsync for file transfer
> instead.

~~~
fulafel
I'm not sure what the misunderstanding is here.. The above text says there are
better protocols and recommends their use over scp, but doesn't say scp is
being phased out or discontinued in the future.

On *ix things stay around for pretty long. You still have ed and ftp on Ubuntu
20.04!

------
jxy
If the ssh gateway OP mentions is just a ssh server, a ProxyCommand should be
used, and the gateway should be transparent after that.

    
    
        Host rcpu-tun
          User YOURLOGIN
          IdentityFile ~/.ssh/YOURSECRET
          ProxyCommand ssh -q -W HOSTBEHINDFW:22 -l PROXYLOGIN SSHGATEWAY
          HostName HOSTBEHINDFW

~~~
wahern
ssh -W requires that the gateway support port forwarding, which is probably
disabled if its enforcing interactive logins. (-W also implicitly sets -T,
though I don't know if that would fail the login or not.) Likewise, -J is
shorthand for a ProxyCommand that runs nc on the gateway, which also won't
work if the gateway requires interactive (i.e. TTY interposed) logins. (EDIT:
Looking at the OpenSSH source code, -J is shorthand for generating a
ProxyCommand using ssh -W. Either it was refactored to use -W after -W was
added, or I'm misremembering how -J used to work.)

------
waterhouse
If you can SSH directly into both systems, then you don't need to print
anything directly to the terminal. You can go:

    
    
      ssh system1 'cat payload' > temp
      cat temp | ssh system2 'cat > payload'
    

Or cut out the middleman, as long as getting two password prompts isn't too
confusing:

    
    
      ssh system1 'cat payload' | ssh system2 'cat > payload'
    

I've found the base64/copy-paste trick useful when I _can 't_ do a
conventional SSH. Usually this is when one of the machines is accessed via a
serial console.

In that scenario, it can help to compress the data before base64-ing it:

    
    
      machine1$ cat payload | gzip | base64
      <output>
      machine2$ base64 -d | gzip -d > payload
      <paste output>

~~~
vlovich123
They mention that you can’t run commands without an interactive shell. I
believe that knocks out your proposal because then rsync+ssh would work
without all the complexity you have in your post with all the same benefits.
Could be wrong though.

~~~
waterhouse
I see, I did miss that.

I do wonder if there's a workaround for that, though. "bash" has the "-l"
option for "Make bash act as if it had been invoked as a login shell", and
also "-i" forces interactive mode. For ssh, "-t" is "Force pseudo-terminal
allocation"; would that work?

~~~
jwilk
I don't follow. How would any of these option help?

------
anderiv
I just confirmed this can also work using AWS Session Manager (a "secure" and
easily-auditable way to provide shell access to EC2 instances):

    
    
        $ aws ssm start-session  --target i-XXXXXX | tee base64.txt

------
Johnny555
I do something similar to move small files via copy-and-paste to/from hosts
that I have to reach through an intermediate jump host.

Rather than copying the file to the jump host, then to the remote host, I'll
just gzip and base64 encode it, then copy and paste it to the remote host.

~~~
EvanAnderson
I've done this to get files into VMs that have poor or no network
connectivity, too. If I can get a console session w/ the ability to paste
keystrokes I'm golden... >smile<

~~~
johnisgood
Heh, thought I was not alone! Although I do not use SSH, I just use netcat.
Shared directories are a hassle. :D

~~~
EvanAnderson
That's a litmus test for me when working with other technicians. If they need
to transfer a file to me across a shared LAN and they offer to pipe the file
over Netcat (and they just expect that I know what they mean) then they
"rate". If I suggest Netcat and they understand they "rate". If they have no
idea what I'm talking about or, worse, suggest a "cloud" service, then I file
them appropriately.

Edit: Just to be clear, I'm talking about transient situations-- like when I'm
working with another contractor on a day-long gig, etc. Obviously, for an
established LAN piping files to each other over isn't a sensible or secure
file transfer method.

~~~
johnisgood
I do not know why you are being down-voted. I would expect people to know what
netcat is, and what it is used for. If all someone can come up with is related
to cloud, then I would dismiss them, too. These "problems" have been solved a
very long time ago, and we have great alternatives.

For myself I have written two shell functions; _nc_send_ and _nc_recv_. It
uses Nmap's ncat with SSL. It uses GPG and to verify data integrity, I use
_b2sum_. Simple, yet perfect for the task. I use it often besides _scp_ , but
of course sometimes _netcat_ is all I need.

~~~
EvanAnderson
I wouldn't necessarily "dismiss" somebody, but knowing about simple tools is a
proxy in my mind for experience level. I can tailor my communication
appropriately when I have some idea of experience level. If I know I can't say
to somebody plugged-in to my laptop directly w/ a patch cable "Give me the
last two octets of your APIPA address and start listening on port 31338 piped
out to a file" then I know that there are a lot of other tasks that I'll need
to break down in minute detail.

~~~
johnisgood
Okay, you are right. I would not dismiss them either for this alone, I would
first perhaps ask them why they think that it is the best option, or what
alternatives there are, and so forth.

> Give me the last two octets of your APIPA address

Is not APIPA Windows-only, by the way?

I can definitely give you two octets of an IP address through using awk, sed,
or using C and bitwise operations. Do I pass? :D

~~~
EvanAnderson
APIPA was standardized as IPv4LL in RFC 3927[1]. I never remember the acronym
IPv4LL-- I always remember APIPA instead. It was a Microsoft construct, but it
has fairly broad OS support now, to my knowledge.

[1] [https://tools.ietf.org/html/rfc3927](https://tools.ietf.org/html/rfc3927)

------
jwilk
> sed '1,/$ base64/d;/$ exit/,$d' ssh.txt

Huh. I'm surpised this works, because $ is not escaped.

~~~
formerly_proven
Single quotes.

~~~
jwilk
No, I mean $ usually means end-of-string (or end-of-line) in regexps. But TIL
in BREs, $ means literal character in the middle of an expression:

[https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd...](https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap09.html#tag_09_03_08)

~~~
formerly_proven
Oh I see now, I never thought about $ being usable anywhere except at the end
as an anchor!

------
katari_v
scp is so slow, does anyone want to improve this?

