Truncated files on SFTP and FTP: when the poller gets there first

A partner reports that last night’s order file was short. You check: 4,812 rows instead of 5,000, and the last row stops halfway through a postcode. You check the transfer log, and it says the transfer succeeded. You check their side, and their log says the upload succeeded too. Both are telling the truth.

This is partial-file pickup, and it is a common and particularly deceptive way for an SFTP or FTP integration to silently corrupt data. It is not a bug in the server, the client, or the network. It is a gap in what the protocols expose: neither SFTP nor FTP exposes a standard “committed and ready for another client” state in a directory listing. The uploader knows when its own transfer has completed - the server tells it so, with FTP’s 226 reply (RFC 959) or a clean SFTP close - but an independent poller sees only a filename, a size and timestamps. Every fix is a convention layered on top, and the conventions are not all equally good. In this article: the race itself, why every log says success, the rule that fixes it, the three fixes, the heuristics when you control neither side, verification, the sender’s mirror-image and how DocEvent’s pull endpoints apply it.

The directory is not a queue

The mental model that causes the problem is treating a drop directory like a message queue: the sender puts a file in, the receiver takes a file out. But a queue has a commit point - a message is not visible until it is whole. A directory has no such thing.

On many conventional FTP and SFTP servers - anything backed by a normal filesystem - the directory entry becomes visible when the transfer starts, not when it ends. From that instant the filename is in the listing, its size grows as bytes arrive, and any reader can open it and read to end-of-file - which simply means “as far as the writer has got”. The reader has no way to distinguish that from the real end of the file. (Some servers and object-storage backends only publish a file once its upload completes - a behaviour worth verifying rather than assuming, either way; more on that below.)

The window is as long as the upload takes, which makes the risk a function of file size and link speed rather than anything you control. A 40 KB file over a fast link is exposed for milliseconds; a 2 GB nightly export over a congested VPN is exposed for twenty minutes, and will bite you within the week. Hence the signature failure pattern: partial pickup appears when volumes grow, in an integration that has “worked fine for years”. Nothing changed except the size of the window.

Why nothing in your logs says “truncated”

The reason this is so expensive to diagnose is that truncation is invisible at every layer that could report it:

• The server answered every command correctly - it listed a file that existed and served the bytes that existed.
• The transfer completed: 226 Transfer complete, or a clean SFTP close, and the logged byte count is the count actually moved.
• The sender finished its upload afterwards and logged success, because it did succeed.
• Your integrity check, if you have one, compares the bytes downloaded against the size the server reported at download time - two numbers describing the same partial file, so they agree.

Whether anyone notices then depends entirely on the file format - which is a dangerous thing to depend on:

The formats most used for bulk business data are exactly the ones that fail silently. That is why the bug is normally discovered downstream, days later, as a reconciliation discrepancy rather than as an alert.

The rule that actually fixes it

A name the receiver is allowed to collect must never exist while it holds an incomplete file.

Every real fix is an application of that one sentence, and every weak fix is an attempt to guess at completion instead of guaranteeing it. Notice what the rule does not say: it says nothing about waiting or timing. Anything that depends on a duration is a probability, not a guarantee.

Fix 1 - upload to a name nobody collects, then rename

This is the standard answer and it is the correct one: upload to a temporary name and, once the last byte is written, rename it to the name the receiver watches for - a pattern the temporary name does not match.

The strength of this is that on a conventional server backed by a single POSIX filesystem, a same-directory rename is normally atomic (POSIX rename()): observers see either the temporary name or the final name, with no state in between to race against. Note what that sentence leans on. The FTP and SFTP rename commands - RNFR/RNTO in RFC 959, SSH_FXP_RENAME in the SFTP v3 specification - do not themselves guarantee that every server backend implements the operation this way. A conventional Unix server does; an object-backed or virtual server must be tested.

Concretely, on each protocol:

# SFTP
put orders.csv orders.csv.part
rename orders.csv.part orders.csv

# FTP
STOR orders.csv.part
RNFR orders.csv.part
RNTO orders.csv

Four things that quietly break it

• Renaming across filesystems. Atomicity is a property of a rename within one filesystem. Across filesystems, POSIX rename() normally fails (EXDEV); some tools and virtual-storage servers then emulate the move as a copy followed by a delete - a fallback that is not atomic and exposes the destination name for the whole duration of the copy. Keep the temporary file in the same directory as its final name; that is the simplest way to be sure.
• A receiver pattern that is too loose. A receiver collecting *, or matching orders anywhere in the name, will happily collect orders.csv.part and you have gained nothing. The pattern and the temporary name are one design, not two.
• SFTP’s rename is not always an overwrite. In SFTP version 3 - what most servers and clients actually speak - the standard rename request is expected to fail when the target already exists (spec, §6.5), although real implementations have varied. OpenSSH’s posix-rename@openssh.com extension does a true POSIX rename with overwrite, and later SFTP versions added explicit overwrite flags. If neither is available, prefer unique final names per file over re-using one - deleting the target first reopens a small window.
• Servers backed by object storage. An SFTP or FTP front end over S3, Azure Blob or GCS has no filesystem underneath, and “rename” is normally a server-side copy and delete - not atomic. Some of these servers only publish the object when the handle is closed, which makes them safe by construction; others expose a growing key. Test yours rather than assuming either way.

Fix 2 - a staging directory

Mechanically the same guarantee, in a shape that is often easier to get a partner to implement: upload into /staging/, and move the finished file to /ready/. The receiver only ever looks at /ready/.

A move within one filesystem is a rename, so this inherits the same atomicity - and the same caveat: across filesystems the rename fails and tools fall back to copy-and-delete, which exposes the file for the whole copy, so keep both directories on one filesystem. Its practical advantage is diagnostic: a file sitting in /staging/ is a visibly stuck or abandoned upload, whereas an orphaned .part file in a busy directory goes unnoticed for months.

Fix 3 - marker files, when rename is not available

Some partners run software that can upload and nothing else - no rename, no second directory. The fallback is a marker file: upload orders.csv, then upload a tiny orders.csv.done. The receiver acts only on markers, collecting the matching data file. It works, and it is a real improvement, but it is weaker than a rename in ways worth knowing:

• It depends on ordering and identity. If files are uploaded in parallel, retried, or re-sent under the same name, a marker can end up describing a different version of the data file than the one sitting there. Publish it only after the data upload is confirmed, bound to the exact file it announces - name the version in the marker, or use unique per-delivery filenames.
• A stale marker is a new failure mode. A marker left behind by an earlier attempt - its data file never arrived, or a cleanup missed it - triggers a collection the next time a file lands under that name. Unique marker names per delivery and deliberate cleanup matter more than anything about the marker itself.
• A marker with contents is itself a file being uploaded. If it carries meaning - a checksum, a row count - it can be read incomplete, exactly like the data file. Give it the same protection: upload it under a temporary name and rename it into place. If it is a bare existence signal, its contents never matter - its lifecycle does.

When you control neither side: age and stability

Now the honest part. Often you cannot change the sender at all - it is a bank, a carrier, a dialer, a piece of vendor software from 2004 - and the file arrives written in place under its final name. You are reduced to guessing at completion from the outside, and it is worth being clear-eyed that this is mitigation, not a fix.

Minimum age. Skip any file modified more recently than some threshold - say sixty seconds. The assumption - a file untouched for a minute is finished - is wrong exactly when the network stalls, which is also when uploads are slowest and files largest.

Size stability. Record each file’s size, and only collect it once two separate observations agree. Same weakness, plus several more:

• A stalled, throttled or dropped-and-retrying transfer can leave the size fixed for longer than your interval - through a congestion pause, or an FTP REST resume’s whole retry backoff - and look perfectly stable while more bytes are still coming.
• Some clients pre-allocate the full size up front (stable and correct while the contents are still zeros), and some servers only update the reported size on flush or close (a growing file looks static).
• The interval between the two observations is what carries the evidence, not the fact that there were two. Two listings seconds apart during a twenty-minute upload prove nothing.

There is also a subtlety that catches many implementations: timestamp precision varies by protocol and server. FTP’s LIST reply has no standardized machine-readable format at all (RFC 959) - the minute-level timestamps in Unix-style listings are an implementation convention. A file listed as modified at 09:04 was modified somewhere in a sixty-second range, so a naive “is it 60 seconds old?” check can pass a file that is one second old. Where the server supports them, MLSD and MDTM (RFC 3659) return standardized timestamps with second precision, and SFTP gives seconds. An age rule has to reason about the precision of the clock it is reading, not just the number.

Done properly, an age-and-stability gate is a reasonable defence, and frequently the only one available. Size the threshold against your slowest realistic upload, accept the latency, and alarm on files that never stabilise rather than letting them sit silently forever.

Verify what you received

Everything so far tries to avoid collecting a bad file. This last layer assumes one got through anyway, which over a long enough period it will.

• A manifest or checksum. If the sender can publish an expected byte count, row count or SHA-256, the receiver can refuse anything that does not match - the only control here that catches corruption as well as truncation.
• A trailer record. Many established formats already carry one (EDI’s segment counts, for instance). A truncated file cannot have a valid trailer.
• Sanity bounds. If a daily file is always within 10% of 5,000 rows, a 4,812-row file is worth an alert even without a manifest. Crude, but free.
• Do not delete until you have verified. If the receiver removes the source as soon as it has the bytes, a bad pickup is unrecoverable. Moving the file to an archive directory instead turns a data-loss incident into an afternoon of replaying files.

If you are the one writing the file

Everything above has a mirror image. When you push files to a partner’s SFTP or FTP server, their poller is racing your upload, and their integration probably has none of these protections. So upload to a temporary name and rename, as a matter of course, on every push endpoint you operate - unless the partner expects something else. It costs one extra command and removes the entire class of problem for them.

Also understand what a failed upload leaves behind. If a data connection drops mid-transfer, the server keeps the bytes it already wrote. Your client reports the failure and retries, correctly - but between the failure and the retry there is a partial file sitting in the partner’s directory under its final name, indistinguishable from a good one. A retry loop with no staging is exposed for the whole backoff, not just the upload.

One adjacent trap while you are there: FTP’s ASCII mode (TYPE A) rewrites line endings in transit. On a binary file that corrupts the contents and changes the length, and it presents to the receiver as “the file is the wrong size” - sending everybody hunting for a truncation bug that is not there. Transfer in binary mode (TYPE I) unless you have a specific reason not to.

How DocEvent handles it

DocEvent’s Channels can collect files from an FTP or SFTP server you nominate, by connecting out to it on a schedule - which puts us squarely in the position described above, polling a directory somebody else writes into. So the controls in this article are the ones we had to build. An FTP-pull or SFTP-pull receive endpoint gives you (capabilities current as of August 2026):

• A filename pattern. Set it to *.csv and a .part or .tmp file is simply never a candidate. This is the setting that makes a sender’s rename actually protect you, and it is the first one to reach for.
• “Ignore files newer than” - the minimum-age gate, with the timestamp-precision problem handled for you: we compute a file’s provable age from the granularity of the clock the server actually reported, so a minute-granular FTP listing is never mistaken for a second-accurate one.
• “Require a stable size across two polls” for servers whose timestamps cannot be trusted at all - two observations from genuinely separate poll cycles, with real elapsed time between them.
• A verification step before the irreversible one. Before deleting or moving a collected file, the endpoint can re-check that the file on the server is still the same name, size and modification time it fetched - if the sender replaced it in between, we do not delete somebody else’s data.
• Move instead of delete. The completion action can move each collected file to another directory rather than removing it, so there is something to go back to.

None of which changes the underlying truth, and we would rather say so plainly: if the sender writes in place under the final name, every one of these is a narrowing of the window, not a closing of it. The filename pattern is the only setting in that list that is a guarantee, and it only becomes one when the sender does its half. If you can get a partner to add a rename, that is worth more than every other control here combined.

The checklist

• Can the sender rename after upload? If yes, do that, gate the receiver on a pattern that excludes the temporary name, and stop - the problem is solved.
• If not: a staging directory and move gives the same guarantee; failing that, a marker file - published only after the data upload is confirmed, bound to the exact file version, cleaned up deliberately.
• When you control only the receiver: a minimum age sized against your slowest realistic upload, plus size stability across genuinely separated polls - and alarm on files that never stabilise.
• Independently: verify what you got - manifest, trailer, or a sanity bound - and archive rather than delete, so a bad pickup is recoverable.
• When you are the sender: upload to a temporary name and rename on every push - and remember a failed transfer leaves a partial file behind until the retry.

References

• RFC 959 - File Transfer Protocol: STOR, RNFR/RNTO, the 226 completion reply; LIST output has no standardized machine-readable format.
• RFC 3659 - FTP extensions: MLST/MLSD and MDTM, standardized machine-readable timestamps.
• SFTP version 3 specification (draft-ietf-secsh-filexfer-03) - SSH_FXP_RENAME; §6.5: rename is expected to fail when the target exists.
• POSIX rename() - atomicity within a filesystem; EXDEV for cross-filesystem renames.