Truncated files on SFTP and FTP: when the poller gets there first

A partner reports that last night’s order file was short. You check: 4,812 rows instead of 5,000, and the last row stops halfway through a postcode. You check the transfer log, and it says the transfer succeeded. You check their side, and their log says the upload succeeded too. Both are telling the truth.

This is partial-file pickup, and it is the most common way an SFTP or FTP integration silently corrupts data. It is not a bug in the server, the client, or the network. It is a gap in the protocol: neither SFTP nor FTP has any way to say “this file is finished”. Every fix for it is a convention layered on top, and the conventions are not all equally good.

The directory is not a queue

The mental model that causes the problem is treating a drop directory like a message queue: the sender puts a file in, the receiver takes a file out. But a queue has a commit point — a message is not visible until it is whole. A directory has no such thing.

When a client uploads a file, the server creates the directory entry when the transfer starts, not when it ends. From that instant the filename is in the listing, its size grows as bytes arrive, and any reader can open it and read to end-of-file — which simply means “as far as the writer has got”. The reader has no way to distinguish that from the real end of the file.

The window is as long as the upload takes, which makes the risk a function of file size and link speed rather than anything you control. A 40 KB file over a fast link is exposed for milliseconds and you may never see the bug in years of running. The same integration with a 2 GB nightly export over a congested VPN is exposed for twenty minutes, and it will bite you within the week.

This is why partial pickup has a signature failure pattern: it appears when volumes grow, in an integration that has “worked fine for years”. Nothing changed except the size of the window.

Why nothing in your logs says “truncated”

The reason this is so expensive to diagnose is that truncation is invisible at every layer that could report it:

• The server answered every command correctly. It listed a file that existed and served the bytes that existed.
• The transfer completed. FTP returns 226 Transfer complete, SFTP closes the handle cleanly, and the byte count in the log is the byte count that was actually moved.
• The sender finished its upload afterwards and logged success, because it did succeed — the bytes it wrote went somewhere.
• Your integrity check, if you have one, usually compares the bytes downloaded against the size the server reported at download time. Both numbers describe the same partial file, so they agree.

Whether anyone notices then depends entirely on the file format — which is a dangerous thing to depend on:

The formats most used for bulk business data are exactly the ones that fail silently. That is why the bug is normally discovered downstream, days later, as a reconciliation discrepancy rather than as an alert.

The rule that actually fixes it

A name the receiver is allowed to collect must never exist while it holds an incomplete file.

Every real fix is an application of that one sentence, and every weak fix is an attempt to guess at completion instead of guaranteeing it. Notice what the rule does not say: it says nothing about waiting, timing, or how long an upload takes. Anything that depends on a duration is a probability, not a guarantee.

Fix 1 — upload to a name nobody collects, then rename

This is the standard answer and it is the correct one. The sender uploads to a temporary name, and once the last byte is written, renames it to the name the receiver is watching for. The receiver only ever collects files matching a pattern that the temporary name does not match.

The strength of this is that rename is atomic. On a POSIX filesystem rename(2) either has happened or has not; there is no instant at which another process sees a half-renamed file. The receiver’s view goes directly from “no matching file” to “a complete matching file”, with nothing in between to race against.

Concretely, on each protocol:

# SFTP
put orders.csv orders.csv.part
rename orders.csv.part orders.csv

# FTP
STOR orders.csv.part
RNFR orders.csv.part
RNTO orders.csv

Four things that quietly break it

• Renaming across filesystems. Atomicity is a property of rename(2), which only works within one filesystem. If the temporary file is on a different mount to the target, the operation becomes a copy followed by a delete — and the copy is exposed for its whole duration, putting you back where you started. Keep the temporary file in the same directory as its final name; that is the simplest way to be sure.
• A receiver pattern that is too loose. The rename only helps if the temporary name is genuinely excluded. A receiver collecting *, or matching orders anywhere in the name, will happily collect orders.csv.part and you have gained nothing. The pattern and the temporary name are one design, not two.
• SFTP’s rename is not always an overwrite. In SFTP version 3, which is what most servers and clients actually speak, the behaviour when the target already exists is unspecified, and many servers fail the request. If your process re-sends a file under the same name, the rename can fail and leave a .part file lying around forever. OpenSSH servers offer the posix-rename@openssh.com extension, which does a true POSIX rename with overwrite; SFTP version 5 added explicit overwrite flags. If neither is available you must delete the target first — which reopens a small window, so prefer unique names per file over re-using one.
• Servers backed by object storage. An SFTP or FTP front end over S3, Azure Blob or GCS does not have a filesystem underneath it, and “rename” is normally implemented as a server-side copy and delete — not atomic, and potentially slow and costly for large objects. Some of these servers only publish the object when the handle is closed, which makes them safe by construction; others expose a growing key. The behaviour varies by product, so test yours rather than assuming either way.

Fix 2 — a staging directory

Mechanically the same guarantee, in a shape that is often easier to get a partner to implement: upload into /staging/, and move the finished file to /ready/. The receiver only ever looks at /ready/.

Because a move within one filesystem is a rename, this inherits the same atomicity and the same caveat — the two directories must be on the same filesystem, or the move degrades into a copy. Its practical advantage is diagnostic: a file sitting in /staging/ is a visibly stuck or abandoned upload, whereas an orphaned .part file in a busy directory tends to go unnoticed for months.

Fix 3 — marker files, when rename is not available

Some partners run software that can upload and nothing else — no rename, no second directory. The fallback is a marker file: upload orders.csv, then upload a tiny orders.csv.done. The receiver watches for markers, and when it sees one it collects the matching data file, then removes both.

This works, and it is a real improvement, but it is weaker than a rename in ways worth knowing before you rely on it:

• It depends on ordering. If the sender uploads several files in parallel, or retries a failed data upload after its marker has landed, the marker can be present while the data file is not what you think it is.
• It doubles the operations and therefore the failure modes: a marker whose data file never arrives is a new kind of stuck state you have to handle and alert on.
• It only protects the data file. The marker itself is subject to exactly the same race, which is why markers should be small enough to land in a single write.

When you control neither side: age and stability

Now the honest part. Often you cannot change the sender at all — it is a bank, a carrier, a dialer, a piece of vendor software from 2004 — and the file arrives written in place under its final name. What then?

You are reduced to guessing at completion from the outside, and it is worth being clear-eyed that this is mitigation, not a fix. There are two signals available, and both are heuristics:

Minimum age. Skip any file whose modification time is more recent than some threshold — say sixty seconds. The assumption is that a file untouched for a minute is finished. That assumption is wrong exactly when the network stalls, which is also when uploads are slowest and files are largest.

Size stability. Record each file’s size, and only collect it once two separate observations agree. The assumption is that a file whose size stopped changing is finished. Same weakness, plus several more:

• A stalled or throttled transfer can pause for longer than your interval and look perfectly stable while more bytes are still coming.
• A resumed transfer — FTP’s REST, or a client retrying after a dropped connection — leaves the file at a fixed size for however long the retry backoff lasts.
• Some clients pre-allocate the full size up front, so the size is stable and correct from the first instant while the contents are still zeros.
• Some servers only update the size they report on flush or close, so a growing file can look static.
• Checking “twice” a few seconds apart is barely a check at all. Two observations two seconds apart during a twenty-minute upload prove nothing; the interval between them is what carries the evidence, not the fact that there were two.

There is also a subtlety that catches many implementations: timestamp granularity varies by protocol. FTP’s LIST output typically gives minutes, not seconds — and for older files, the year instead of a time at all. So a file listed as modified at 09:04 was modified somewhere in a sixty-second range, and a naive “is it 60 seconds old?” check can pass a file that is actually one second old.MLSD and MDTM give you seconds; SFTP gives seconds and, from version 4, better. If you build an age rule, it has to reason about the precision of the clock it is reading, not just the number.

Done properly, an age-and-stability gate is a reasonable defence, and it is frequently the only one available. Just size it honestly: pick a threshold comfortably longer than your slowest realistic upload, accept the latency that costs you, and alarm on files that never stabilise rather than letting them sit silently forever.

Verify what you received

Everything so far tries to avoid collecting a bad file. This last layer assumes one got through anyway, which over a long enough period it will.

• A manifest or checksum. If the sender can publish an expected byte count, row count or SHA-256 — in a sidecar file, in a header row, in a trailer record — the receiver can refuse anything that does not match. This is the only control here that catches corruption as well as truncation.
• A trailer record. Many established formats already carry one (EDI’s segment counts, for instance). If yours does, check it; a truncated file cannot have a valid trailer.
• Sanity bounds. If a daily file is always within 10% of 5,000 rows, a 4,812-row file is worth an alert even without a manifest. Crude, but it catches the case that matters and costs nothing.
• Do not delete until you have verified. If the receiver removes the source as soon as it has the bytes, a bad pickup is unrecoverable and the sender has no reason to re-send. Moving the file to an archive directory instead of deleting it turns a data-loss incident into an afternoon of replaying files.

If you are the one writing the file

Everything above has a mirror image. When you push files to a partner’s SFTP or FTP server, their poller is racing your upload, and their integration probably has none of these protections. Two things follow.

First, upload to a temporary name and rename, as a matter of course, on every push endpoint you operate — unless the partner has specifically told you their system expects something else. It costs one extra command and it removes the entire class of problem for them.

Second, understand what a failed upload leaves behind. If a data connection drops mid-transfer, the server keeps the bytes it already wrote. Your client will report the failure and retry, correctly — but in the gap between the failure and the retry, there is a partial file sitting in the partner’s directory under its final name, completely indistinguishable from a good one. A retry loop with no staging is therefore not just exposed during the upload; it is exposed for the whole backoff too.

One adjacent trap while you are there: FTP’s ASCII mode (TYPE A) rewrites line endings in transit. On a binary file — a zip, an image, a compressed export — that corrupts the contents and changes the length, and it presents to whoever receives it as “the file is the wrong size”, which sends everybody hunting for a truncation bug that is not there. Transfer in binary mode (TYPE I) unless you have a specific reason not to.

How DocEvent handles it

DocEvent’s Channels can collect files from an FTP or SFTP server you nominate, by connecting out to it on a schedule — which puts us squarely in the position described above, polling a directory somebody else writes into. So the controls in this article are the ones we had to build.

An FTP-pull or SFTP-pull receive endpoint gives you:

• A filename pattern. Set it to *.csv and a .part or .tmp file is simply never a candidate. This is the setting that makes a sender’s rename actually protect you, and it is the first one to reach for.
• “Ignore files newer than” — the minimum-age gate, with the timestamp-precision problem handled for you. We compute a file’s provable age from the granularity of the clock the server actually reported, so a minute-granular FTP listing is not mistaken for a second-accurate one; a file is collected as soon as it is provably old enough, and not before.
• “Require a stable size across two polls” for servers whose timestamps cannot be trusted at all. The important detail is that the two observations must come from genuinely separate poll cycles with real elapsed time between them — not two listings a second apart within one cycle, which is the version of this check that proves nothing.
• A verification step before the irreversible one. Before deleting or moving a collected file, the endpoint can re-check that the file on the server is still the same name, size and modification time it fetched. If the sender replaced it in between, we do not delete somebody else’s data.
• Move instead of delete. The completion action can move each collected file to another directory rather than removing it, so there is something to go back to.

None of which changes the underlying truth, and we would rather say so plainly: if the sender writes in place under the final name, every one of these is a narrowing of the window, not a closing of it. The filename pattern is the only setting in that list that is a guarantee, and it only becomes one when the sender does its half. If you can get a partner to add a rename, that is worth more than every other control here combined.

The checklist

• Can the sender rename after upload? If yes, do that, and gate the receiver on a pattern that excludes the temporary name. Stop here; the problem is solved.
• If not, can they use a staging directory and move? Same guarantee, sometimes easier to sell.
• If not, can they write a marker file after the data file? Weaker, but real.
• If none of the above, apply a minimum age sized against your slowest realistic upload, and a size-stability check across genuinely separated polls.
• Independently of all of it: verify what you got — manifest, trailer, or at minimum a sanity bound on size or row count.
• Archive rather than delete, so a bad pickup is recoverable.
• Alarm on files that never stabilise and on files that arrive outside their expected size range.
• When you are the sender, upload to a temporary name and rename — and remember a failed transfer leaves a partial file behind until the retry.