Scaling and connection handling

Simple FTP Services run on auto-scaling clusters of multiple servers per region. Servers are regularly added, removed and replaced - during traffic changes and during blue/green deployments of new versions - and the process is designed so in-flight transfers are never cut off by a scaling event.

When a server is taken out of the cluster it goes through a drain sequence:

1. The server is removed from DNS
2. We wait for DNS propagation and the DNS TTL to expire
3. The server stops accepting new connections
4. Existing connections are given time to finish their work
5. The server shuts down

So an upload or download that is in progress when a scale-down starts is allowed to complete before the server hosting it goes away.

Two practical consequences of running on a cluster:

• The IP address behind the hostname changes between connections. If your SSH client warns about this, see SSH host keys. If your firewall needs fixed addresses, use the static IP hostnames.
• Very long-lived idle sessions may occasionally need to reconnect after a deployment. Transfers in progress are drained; idle control connections are the first thing to go.