
Show HN: Station307 – Stream files between cURL, Wget and/or browser - hakvroot
https://www.station307.com
======
hakvroot
Author here! This is a little side project I created because I found myself
using scp + email to send a file from a server to someone sitting right next
to me more than once. I have a different project which does relaying over HTTP
so I figured it might be a good idea to reuse a part of the codebase for a
simple file streaming service. I finished it yesterday and it already proved
useful for myself, and I hope it also proves to be useful for you.

If there are any (more) questions on the what or how, I'm happy to answer
them. The day here is however coming to an end so it might take me a couple of
hours. While I'm gone, please be aware that the service is running on a
$5/month VM ;).

~~~
maxpert
Hey really nice to hear from you. Would be really nice if you share source
code with the community. I really like the HTTP status code 100 trick :D

~~~
hakvroot
Thanks! I made some comments regarding to open sourcing it in reply to vortico
(this thread) - current status is contemplating.

------
prohor
Very nice. I've bookmarked it.

By the way - there was another similar service recently, but stores files:
[https://transfer.sh/](https://transfer.sh/) ,
[https://news.ycombinator.com/item?id=11322007](https://news.ycombinator.com/item?id=11322007)

~~~
hakvroot
Thank you! And thanks for linking to the thread as well; I did already come
across it but it definitely contains some good discussion on general
feasibility, DCMA issues and other similar projects. It was also nice to learn
that transfer.sh is also made in Holland :).

------
vortico
I've recently wanted to write a streaming download service like this recently.
The great thing about your service is that although it's a proof of concept of
a handful of neat tricks, the core functionality works perfectly and could be
extremely useful. This makes it faster than ever to drop upload links in a
chat room and effectively solves the problem in
[https://xkcd.com/949/](https://xkcd.com/949/).

Is the backend licensed as free software? It would be fantastic for admins to
host their own version of this since it allows the concept to propagate
without worry of shutting down.

~~~
hakvroot
Currently I haven't licensed the backend as free software; the primary reason
for this is that I reuse quite a lot of components from a different project of
mine ([https://gemin.ee](https://gemin.ee)) and I haven't decided yet on
whether I want to release those components or remove them as dependencies.
Since I too am familiar with useful stuff suddenly disappearing I'll give you
my word that I will release the source if I have to shut down. If I decide to
share the source I'll post a link to it on the site itself, so you won't have
to look for it ;).

On another note, I'd encourage you to write a streaming download service.
Compared to file uploading - storing - downloading I found it fun to do but
also quite non-trivial to accomplish. I am planning to have another go at it
myself, but with a different language (Go, Pony or Elixir). When run alongside
of each other with the same UI/frontend/api it would make a nice benchmarking
mechanism :).

------
exhilaration
How does this work? I see the pending POST in the Chrome network tab, which
completes when someone downloads it, and then become pending again for the
next download. So are you proxying the uploaded data to the downloading
client?

~~~
hakvroot
That's exactly what's going on. In a bit more detail: when a POST arrives and
no receiver is available yet the server lets the sender know that it won't
handle any more data (TCP Zero Window). When the receiver connects to the
server the server asks the sender to continue writing and the data is relayed
(proxied) from sender to receiver. If multiple receivers connect they're put
into a queue and the server waits for the sender to POST the data again once
the ongoing transfer is finished.

An interesting difference is that in the browser the rePOSTing is done through
scripting whereas when using cURL or Wget (or httpie, or ...) the server will
issue a 307 Temporary Redirect allowing cURL/Wget to send the file again
without manual intervention. Hence the name! (although that doesn't really
account for the "Station" part yet)

~~~
darkf
Hi! Thanks a lot for sharing details about it, it's an ingenious app. :)

Could you please elaborate a bit on how you work with the TCP window size --
how might I set it to 0 and back? Do you simply use SO_RCVBUF or is there more
to it?

If you happen to go through with making more variants of this, it would be
amazing if you could make them open source as well (even just the backend).
It's a clever idea that solves a very real problem!

~~~
hakvroot
Thank you! And I'd be happy to elaborate a bit more, although you already
explained most of it yourself ;). Currently the server just stops reading from
the socket and the rest is being handled by the networking stack (i.e. buffer
fills up, window size is to 0). At this moment the buffer size is also fixed
(it sits somewhere between having good throughput and not consuming too much
memory for parked connections), but an improvement would be to change it
dynamically to improve behavior on both fronts.

For any other versions or source releases I'll post an update!

------
disaster01
This is very useful! How do they think to earn money with it?

~~~
hakvroot
Thanks! I don't have any plans to monetize this to be honest. I built this
because I had the need for this myself and, if people like it, it will be a
nice exercise in maintaining something that works well under stress :).

------
pkd
Looks great! What is the technology stack behind this, if I may ask?

~~~
hakvroot
Thank you, and of course you may.

The backend is written in Java(8) and it is built with the following
(noteworthy) components :

    
    
      Netty[1] - Extremely fast and stable Java networking framework,
                 serves static files and relays traffic 
                 (both tasks are custom implementations)
    
      Guice[2] - IoC container by Google, nice for keeping the code clean
    
      jOOQ[3] - I log some statistics in PostgreSQL 
                (file size, content type, transfer time - but no filenames). 
                If you're comfortable with SQL jOOQ is a real breeze to use.
    

The backend is hosted on the cheapest VM option at Digital Ocean (Amsterdam
region), (which seems to be holding up pretty well so far :). I do run
PostgreSQL on another $5 VM, so the Station307 server only has to worry about
serving static content and moving large chunks of bytes around. SSL
certificates are kindly provided by Let's Encrypt[4].

For the frontend, I know for a fact it uses left-pad (babel-core)... Written
in JavaScript (ES2015 + JSX) and built with :

    
    
      Mithril[5] - Fast, small and simple React-like framework, quite a joy to use.
    
      Bootstrap[6] - Because not having to worry about L&F is quite nice. 
                     Trimmed it down a bit by only importing the parts I needed (SCSS).
    

And I think that's about it. If there's anything else you'd like to know, let
me know!

    
    
      [1] http://netty.io
      [2] https://github.com/google/guice
      [3] http://www.jooq.org
      [4] https://letsencrypt.org
      [5] http://mithril.js.org
      [6] https://getbootstrap.com
    

_edit: formatting_ D:

------
Douchiz
WOW this is so cool

