
SFTPGo: A Full Featured SFTP Server in Go - ngaut
https://github.com/drakkan/sftpgo
======
nickcw
This looks like a very interesting project to make a lightweight sftp server,
though the dependency on an SQL db makes it a bit harder to install.

Go has a really good set of crypto primitives, an excellent ssh implementation
maintained by the core team, and a great sftp server and client library which
makes this project possible.

I recently added an sftp server to rclone:
[https://rclone.org/commands/rclone_serve_sftp/](https://rclone.org/commands/rclone_serve_sftp/)
\- this can serve any of the cloud providers rclone support as sftp (or local
disk). This runs on windows/macOS/linux too.

It was a joy to add this as the Go libraries are very well thought out and
easy to use.

~~~
edwinyzh
Not sure the case in this context, but usually the SQLite dependency can
almost be seen as no-dependency at all - it's just a file on the disk without
any external dependency.

~~~
rakoo
On the other hand it requires a C compiler, so it's definitely heavier than a
pure go application

~~~
jlubawy
Serious question, why hasn't anyone rewritten it in Go? Been awhile since I've
looked at it but C to Go doesn't seem that difficult to translate.

Edit: just looked at the amalgamation again, it's huge... Still surprised no
one has tried it yet that I know of. Pre-processor directives would probably
make it even more difficult though.

Edit2: and it already works so what's the point, other than an extremely
challenging problem

~~~
minxomat
Sqlite has been rewritten in both Go and C#.

It's really not that hard. The code is clearly documented and everything works
in roughly the same way. Easily achievable by a single developer in a few
months.

Sqlite code is emulating classes with structs and methods (they're even called
classes in the code). This it's pretty easy to translate into languages with
reference and struct support.

Memory allocation is separated into an allocation provider, which is easy to
implement in GC languages (just create the struct requested).

The parser and VM opcode file is tricky. The opcodes are parsed from comments
in one big file. I'm not a fan of that. But once generated for a particular
version, pretty easy to translate.

Testing is another can of worms. Translating the exact version of Tcl, too is
the best course of action. This is required to run and pass the (hundreds of
thousands of) base line tests. Other correctness tests (in sqlite because of
branch coverage requirements for aircraft software) are harder to translate.
The prerelease burn in test is also not publicly available.

Preprocessor directives are not used as macros. Rather they are used to enable
features. In a port, you can for example choose to exclude WAL or specific
optimizations. FTS can be left out if not needed. The different lock
implementations can be deferred to the stdlib.

If you interested in the perf overhead, here's a small benchmark of a C#
implementation I maintain:

[https://drive.google.com/file/d/11Bgfh1WgEreDenosoEdWkYAOb6u...](https://drive.google.com/file/d/11Bgfh1WgEreDenosoEdWkYAOb6u_fhMQ/view?usp=drivesdk)

~~~
lifty
Can you point to the pure Go re-implementation of Sqlite? I was looking for
such a thing but never found it. Instead I had to go with a k/v store like
Bolt.

~~~
8lall0
[https://godoc.org/modernc.org/ql](https://godoc.org/modernc.org/ql)

~~~
minxomat
That's not it. It was a GitLab project, but searching for that is pretty
difficult. I'll edit this comment if i find it.

Edit: Here it is:
[https://gitlab.com/cznic/sqlite](https://gitlab.com/cznic/sqlite)

------
colek42
A lot of the world runs on sftp. The flexible user management in this code
would have saved my team about a month of time implimenting our own solution.

------
linuxdude314
Part of the utility of SFTP to clients and admins is the fact there are no
additional requirements other than SSHD to function.

This seems to fly in the face of that.

If you are willing to install more software, why not use a more feature filled
file server?

~~~
pilif
OpenSSH is very much tied to system users, but sometimes you might want to
give access to external users that don’t need to exist as actual unix users.

 _because_ SSH and SFTP are so closely tied together, a configuration via PAM
is pretty hard and inconvenient because creating fake users via PAM for SFTP
will also create them for SSH and because there’s no easy way to map all such
virtual users to the same user-id. Also, because OpenSSH has zero support for
virtual users, aside of a PAM configuration, you also need an NSS
configuration and now all your virtual users in some database have suddenly
become system users on your box.

SFTP as a protocol on the other hand is very convenient over, say FTP over TLS
because it’s using a single TCP port and it has been created this century.

So having this self-contained project is useful when you need to allow third
parties access to files but you also don’t want to create system users for
them or risk f’ing something up with PAM

~~~
jasonjayr
I'd second this, I've longed struggled to come up with a easy-to-deploy
sshd/sftp/chroot configuration that permitted easy database-driven
configuration w/o extra shell access. You have to fight a lot of defaults to
get this just right.

Would the OpenSSH upstream accept patches for an unprivileged sshd/sftp-
subsystem to make this easier to use their battle tested code?

~~~
sandreas
If you just want to share files and not giving the option to upload files, you
could try graft...

[https://github.com/sandreas/graft](https://github.com/sandreas/graft)

graft serve ./*.txt

will serve every txt file in the current dir...

~~~
yjftsjthsd-h
For that use case, I just use `python3 -m http.server` (or the Python 2
equivalent), especially since Python is almost always already installed.

~~~
sandreas
Cool... is SSL support for secure file transfer?

------
andrewchambers
I did a project a while ago for fun using go to implement an sftp server for a
dropbox account. (Though I used openssh for the ssh server).

[https://github.com/andrewchambers/sftpplease](https://github.com/andrewchambers/sftpplease)

------
todd3834
Go seems like a really good fit for this kind of project

~~~
alpb
Yeah, I'm hoping to see use of Go routines in useful rewrites of projects like
rsync with parallel file uploads. It can lead to clean, extensible, and
maintainable code with still low overhead and high throughput.

~~~
meruru
It's in C again, but since you mentioned rewrites and rsync:
[https://github.com/kristapsdz/openrsync](https://github.com/kristapsdz/openrsync)

------
deedubaya
I've been avoiding starting a project myself which will require SFTP. Passing
text files around via SFTP is how a lot of the world works. It's still painful
to implement this, and makes the transfer of data slow (polling for new files
to process etc). This introduces a big delay in any data feedback cycles.
Managing users with system accounts is painful too.

This project looks to alleviate a lot of these problems.

It'd be great to be able to register webhooks so that it could send events to
external systems. Ideally I'd like to know when a file is created/deleted etc
without having to walk directories on the sftp server on a regular basis.

~~~
drakkan
Events are now supported, you can configure custom system commands and/or HTTP
notifications on SFTP upload, download, delete or rename

------
heinrichhartman
Can this project be used as a library and integrated into an existing program?

I am asking, b/c I am currently working on a little document management system
for home-use. It looks like the only sane way to integrate with document
scanners is (S)FTP upload or Email (SMPT). I have tried to use off-the-shelf
FTP servers for this and inotify to get notified of new uploads. The the
solutions work OK, but are hard to setup and rather brittle (and limited to
Linux).

Go seems to be the ideal language for this project, because its concurrency
model and the ease of distribution (as single binary files).

\- Has anyone here experience with building systems like this (integrating via
FTP/SMTP)?

\- Any recommendations for languages and libraries?

~~~
quickquestion42
The linked project seems to be a wrapper around

[https://github.com/pkg/sftp](https://github.com/pkg/sftp)

~~~
heinrichhartman
Ah Thx! Should have checked the code myself. The library looks good. Not too
much documentation
([https://github.com/pkg/sftp/issues/267](https://github.com/pkg/sftp/issues/267),
[https://github.com/pkg/sftp/issues/247](https://github.com/pkg/sftp/issues/247)),
but straight forward and battle tested.

------
maxekman
I’m currently architecting a transport administration system written fully in
Go which includes XMLEDI over SFTP; this project is perfect as we need to
integrate with the current IAM.

------
drondin
I was looking for something like this when I was asked to get an FTP working
on kubernetes...

Don't ask why...

------
microcolonel
Go is a very convenient option of you have an API agreement based on dropping
files on an SFTP.

------
jhoechtl
Using JSON for configuration: Fail

~~~
diroussel
What would your preferred config file formats be, and why?

~~~
jhoechtl
Any format which enables inline comments, so INI, YAML

~~~
drakkan
SFTPGo switched to viper for configuration so it now supports JSON, YAML,
TOML, env vars ecc..

The configuration format it's a your choice

