SethFTP

I recently had the need to transfer a large number of pictures from my phone to my server, but was unable to find an acceptable way to do it. Both MTP and Bluetooth are slow and require a close distance; SFTP clients are packed full of ads and are unable to saturate my bandwidth. I decided to write my own program to solve this problem and the result is SethFTP: a secure, powerful and efficient file transfer protocol. The protocol has been implemented on FreeBSD (server + client), Android (client) and Windows (client).

Details

Both SCP and SFTP tend to be chatty when lots of small files are being transferred, resulting in slow speeds over high-latency links. I decided to solve this problem out of the gate by designing SethFTP to be heavily pipelined. Whether you are transferring hundreds of small files or a few large ones, the protocol-level performance remains identical. This is accomplished by coalescing multiple operations together until the message reaches a reasonable size, at which point it is encrypted and sent over the wire. Think of it as an on-the-fly 'tar'ing and un'tar'ing process.

The protocol supports robust file resumption. This means that both the client and server will compute and compare a MAC over the existing part of the file to be sure they match first; contrast this against SFTP where no such comparison is done, which can result in frankenstein files. The protocol is mode-based; see below for a complete list of modes and commands.

I experimented with supporting metadata preservation (such as timestamps) but ultimately decided against it due to the fact that such data is non-portable. For example, not every filesystem supports setting the file birth time, nor do they all store the timestamps to the same precision. Not every filesystem has the concept of 'owner' and 'group' either. If such data must be preserved, then the solution is to wrap the files in a container format (such as a tar file) and then transmit that. This protocol is data-focused.

Demonstration

The main activity is pretty simple and contains connectivity information.

SethFTP

When 'download' is tapped, the client issues the dir command to get a directory listing. I wrote a custom RecyclerView adapter and layout to render the returned listing. It allows the user to navigate between directories and select multiple files.

SethFTP

Once files have been selected and 'download' is tapped, the client first issues an info command for files that already exist locally, in order to check if resumption is needed. It then requests the files from the server via the download command. The main activity is periodically updated to reflect download progress. The app also maintains a notification with some basic information about the transfer.

SethFTP SethFTP

When 'upload' is tapped, the client displays a file picker activity. In this example, I am selecting all of the files that I just downloaded from the server.

SethFTP

Once files have been selected and 'open' is tapped, the client first issues an info command for the selected files to check if resumption is needed. It then streams the data to the server via the upload command. The main activity is periodically updated to reflect upload progress as well as server acknowledgement. The app also maintains a notification with some basic information about the transfer.

SethFTP SethFTP

I wrote a simple console client as well that runs in FreeBSD and Windows. It is interactive and feels a lot like an SFTP session. In this example, I am performing a glob to locate files of interest, then downloading one of them from the server. I then upload the file back to the server.

SethFTP