File transfer audit trails: how to prove delivery

The email arrives at 9:12 on a Monday. “We didn’t receive Friday’s order file. Can you confirm it was sent?”

So you check. The scheduled job ran - the scheduler says so. The script exited zero - the log says so, insofar as there is a log. And that is where the trail ends. Which files were in Friday’s batch, whether this partner’s was one of them, what the receiving server said, whether anything failed and was retried - none of it was written down, because nothing was ever asked to write it down. The transfer has run unattended for four years. The proving part never existed.

This is the moment most managed file transfer purchases actually begin. Not with a transfer failing - with a question nobody can answer. In this article: why job logs can’t answer it, what a per-file record must contain, sizing retention, what counts as proof of delivery, failure handling, questions for vendors and how DocEvent does it.

You probably don’t have a transfer problem

When a team starts evaluating “MFT software”, the file-moving part of their setup is usually fine - OpenSSH, WinSCP, curl and cron have moved bytes with boring reliability for decades. What failed on Monday morning was the record. Strip the acronym down and most of what an MFT product actually sells is evidence and aftermath: a per-file history, retries you can watch, a place failed files go instead of vanishing, and a screen a person under pressure can read. So the right way to evaluate one - or to judge the scripts you already have - is not “can it do SFTP”; everything can do SFTP. It is: when a partner disputes a file, what exactly can you put in front of them?

The unit of record is the run, not the file

Schedulers and scripts record runs. The log line that exists says the 02:00 job started and finished; the exit code is the run’s one-word summary of itself. But every question that ever arrives is about a file: this invoice, that partner’s order export, the 21st’s statement. A wildcard upload - put /outbound/*.csv - moves two hundred files and produces one record. There is no row for the file the question is about, and no amount of grepping creates one after the fact.

The exit code makes it worse by flattening every failure into one bit. A DNS blip, an expired password, a full disk on the far side, and one file out of 214 failing all become “exit 1” - or, if the loop does not propagate errors carefully, “exit 0”. Whether a single failed file inside a batch surfaces at all depends on how defensively the script was written, which is to say: on the habits of an author who may have left the company.

Six questions a per-file record has to answer

If the unit of record is the file, what does its record need to contain? Six things - worth using as an inspection checklist against whatever you run today:

• Arrival. When the file entered your system, from where and from whom - which endpoint, which credential - plus its name and size at that moment. Disputes are two-sided: “you never received it” needs the same quality of answer as “you never sent it”.
• The decision. What the system decided to do with it: which rule matched, which destinations it was bound for. When a file goes somewhere surprising, “why did it go there” is the question, and it is the one most logging forgets entirely.
• Outcome, per destination. One file frequently goes to several places, and partial success is a normal Tuesday, not an edge case. “Delivered” as a single flag on the file is a lie waiting to happen; it has to be a status per destination, each with its own timestamp and attempt count.
• The failure, verbatim. When a delivery fails, the record needs what the server actually said - 530 Login incorrect is a diagnosis, “transfer failed” is a shrug. The difference decides whether the fix takes four minutes or a conference call.
• The aftermath. How many retries, when, with what result - and if the file is still undelivered, where it physically is right now. “Where is the file?” must have a location for an answer, not a silence.
• Durability and reach. All of the above, answerable weeks later, by someone who is not the person who built the pipeline, searchable by filename, partner and date.

Notice that none of these six is about moving bytes. That is the point of the whole exercise: they are the product.

The question arrives after the log dies

The sixth requirement - durability - deserves its own section, because it fails silently even in setups that log reasonably well. Transfer logs, where they exist, get their retention from a logrotate default: seven or fourteen daily rotations, then gone. That number was chosen by a package maintainer thinking about disk space. The questions, meanwhile, get their timing from business cycles: a partner notices a missing file at their month-end reconciliation; a finance team finds a gap when the books are closed; an auditor asks in quarters. Almost every question worth answering arrives after day fourteen.

So sizing retention is not a storage decision. The rule: the record must outlive the slowest question that will be asked of it - thirty days is the floor if disputes surface at month-end, and a hundred and twenty or more if you answer to quarterly audits or regulated record-keeping. And the record has to be reachable: ninety days of gzipped rotations on a box one engineer can ssh into fails the Monday-morning test just as surely as deleted logs, because the person who receives the partner’s email is usually in support or operations. An audit trail is only real if the person who gets the question can open it and search it by filename.

What counts as proof of delivery

Now the claim itself: “we sent it”. Evidence for that statement comes in grades, and most integrations rest on the weakest one without anyone having decided that on purpose.

An exit code proves your script did not notice a failure. That is the entire content of the claim.

Protocol completion - FTP’s 226 Transfer complete, a clean SFTP handle close, byte counts that match - proves the destination server accepted the bytes, and is worth capturing per file. But it says nothing about what happened next - a post-upload script that moves the file, a virus scanner that quarantines it - and it lives in a log only your side keeps.

Post-delivery verification is the strongest evidence available on a plain SFTP or FTP endpoint: after uploading, list the file on the destination server and compare name and size. Now you are no longer reporting your own claim - you observed their system holding the file. It costs one extra round-trip and upgrades “we sent it” to “it was there, at this size, at this time”. (Its limits: servers that whisk files away on close can defeat the re-list, and a size match is not a checksum. Evidence, not attestation.)

A signed receipt is where the other side starts producing the evidence. This is what AS2 was designed for: the receiving system computes a digest of exactly the bytes it received - the MIC, message integrity check - signs it, and returns it as an MDN, a message disposition notification (RFC 4130). Once its signature is verified and its MIC matches the digest you retained for the message, a signed MDN is cryptographic evidence, produced by the partner’s own software, that it received exactly what you sent - which is why the receipt and your digest are both kept. Walmart famously required AS2 of its suppliers back in 2002, and many large retail - and some healthcare - trading relationships still specify it. The flip side matters as much: an AS2 sender treats a missing receipt as a delivery problem - “transmitted but unconfirmed” is a distinct state, not a success.

A business acknowledgment sits above even that - but be precise about which kind. An EDI 997 or 999 functional acknowledgment confirms the interchange was received and passed structural checks; X12 is explicit that it is not application-level validation. Proof that the order, claim or invoice was actually accepted needs the corresponding business response - an 855 purchase-order acknowledgment, a 277CA claim status, a return file, an API callback. Where the workflow supports one, that is the gold standard.

An honest note about the top rungs: most partners are not an AS2 station. They are a directory on an SFTP server, and you cannot get a signed receipt from a directory. When that is the situation, your realistic ceiling is protocol completion plus post-delivery verification - recorded, per file, in a trail you keep. Which is precisely why the trail matters so much: when the other side cannot attest, your own evidence has to be complete and contemporaneous, because it is all there is.

Failure is part of the record

Everything above concerns files that arrive. The audit trail earns its keep on the ones that do not - specifically, at 2 a.m., after the last retry fails, when no human is watching. There are three common shapes for that moment, and two of them are bad:

• Silent drop. The message is discarded, the loop moves on. The file is simply gone, and the first anyone hears of it is the partner’s email. This is the worst outcome an integration can produce, and it is the default behaviour of a retry loop that runs out of retries without a plan.
• Retry forever. Nothing is lost, but a file that can never succeed - a deleted account, a renamed directory - blocks the queue behind it or burns quota indefinitely, and the failure still is not visible anywhere.
• Park it, with its reason. Bounded retries; then the file moves to a holding area - a dead-letter queue, a quarantine directory - along with the recorded reason and timestamps, where it can be re-sent once the cause is fixed. The parked file itself becomes the answer to “where is it now?”.

Two refinements separate a good implementation of the third shape from a token one. First, the retrying should be visible while it happens - “attempt 2 of 3, last tried 02:01” is a state someone can act on, not a forensic discovery. Second, distinguish transient failures from permanent ones: a network timeout deserves its retries; a rejected password will fail identically every time, and retrying it only delays the parking. Either way, what lands in the record must be the server’s actual response, because the parked file’s “why” is what turns a re-send from an experiment into a fix.

Monitoring is not the audit trail - you need both

One clarification before the shopping questions. The audit trail answers questions about the past: what happened to this file. Monitoring answers a question about now: is something currently wrong. Teams shopping for MFT tend to say “monitoring” while describing audit problems - the partner’s Monday email is an audit failure, not a monitoring one.

Failures are the easy half of monitoring: a failed delivery is an event, and an event can be surfaced. The genuinely hard half is absence. The file that never arrived produces no event at all - there is nothing to alert on unless the system holds an expectation: Acme’s order file lands every business day by 06:00. Most home-grown setups have no expectation model, and plenty of commercial ones are weak here too - a fair and revealing question to put to a vendor. The honest budget version is a daily reconciliation habit: yesterday’s expected count against yesterday’s actual, per partner, reviewed by a person.

The questions to ask a vendor - or your own scripts

If the argument so far is right, evaluating an MFT product is mostly evaluating its record-keeping - live, in a demo, with questions that have yes-or-no answers. They also work as an audit of whatever you run today, and remember who they are for: the audit trail is only as good as the least technical person who needs it under pressure.

• “Show me one file, end to end.” Arrival, decision, and outcome per destination, with timestamps, on one screen - including the verbatim server response on any failed delivery. If the vendor assembles the story from three tools, so will you, at 9:12 on a Monday.
• “One file in a batch of two hundred fails - what does that look like?” Partial success is the everyday case. If the answer renders as a single red X on the batch, the per-file story is a facade.
• “What happens after the final retry?” Where does the file go, who can see it, who can re-send it, and how many clicks is that? “It goes to the error log” is the wrong answer.
• “How long is history kept, and what can I search by?” Filename, partner, date range at minimum - retention measured against your question cycle, not their disk budget - and answerable by support without shell access.
• “If a partner requires AS2 - where do the receipts land, and what happens when one doesn’t arrive?” The second half is the revealing half: a missing receipt should become a visible, held state, not a quiet success.

How DocEvent handles it

DocEvent’s Channels move files between partners and systems for a living, so the record-keeping described above is not a feature we bolted on - it is most of what the product is. Held against the six questions (capabilities current as of August 2026):

• Every file gets its own record. Each channel has a Routing log; every file in it opens into a “File journey” view: when it arrived and through which endpoint (with protocol and source IP for endpoints you own), how it was routed, then one delivery lane per destination - queued, submitted, delivered, each with timestamps and the time elapsed since receipt.
• Failures carry the server’s words. A failed lane keeps its full attempt history, and each attempt records what the destination actually said - the real 530, the real HTTP response - not a summary. While retries remain, the lane says so, visibly: “retrying automatically - attempt 2 of 3”.
• Partial success is rendered as partial. A file bound for three destinations that reached two shows “Partial - 2 of 3”, not a global green tick; a file with no activity for 45 minutes is flagged as stalled rather than left looking in-flight.
• Failed files park; nothing is dropped. After the final retry - or immediately, for failures that retrying cannot fix - the file moves to Stuck files: name, destination, when it failed, and the recorded reason, with one-click re-send for a file, a selection, or everything, once the cause is fixed. The re-sendable copy is held for 24 hours; the record of what happened is not so short-lived - see below.
• Retention is sized to the question cycle. Routing history stays searchable for 120 days - by filename or file reference, over any date range in that window - and the same 120 days applies to the per-operation logs on the SFS side (every login, upload, download and delete, with a CSV export).
• Receipts, where the partner speaks AS2. AS2 endpoints record the receipt state on the same lane as the delivery - awaiting, received, rejected, integrity mismatch - with the receipt evidence kept alongside the file. A file whose receipt never arrives is parked as “transmitted but unconfirmed” rather than assumed delivered, and a late-arriving receipt flips the state to “your partner acknowledged this file - don’t resend it”.
• Shared channels audit without leaking. A channel’s members can be different companies, so the record is redacted by ownership: you see your own endpoints and lanes in full; other members’ destinations appear only as an anonymous marker. Each party gets a complete audit of its own side and no view into anyone else’s infrastructure.
• And one small thing that prevents big things: a Test connection button on endpoints - a side-effect-free check that connects and authenticates against the real destination without sending a file. Most 2 a.m. delivery failures are credentials and firewalls; this catches them at configuration time.

And in the spirit of the evidence ladder, the honest limits: DocEvent does not yet notify you when a delivery fails or a file parks - the trail and the stuck queue are in the console, and checking them is on you today. There is no expected-file/absence alerting either. And the deep per-attempt service logs behind a delivery are kept for about a week - the 120 days applies to the routing record itself. If those gaps are your dealbreakers, you now know the exact questions to ask anyone else, too.

The checklist

• Judge your setup by the Monday-morning test: a partner disputes a file - what can you put in front of them, and how fast?
• Make the file the unit of record: arrival, decision, per-destination outcome, verbatim failure, aftermath. A per-run log cannot be grepped into a per-file answer.
• Size retention to the slowest question - month-end disputes and quarterly audits, not logrotate defaults - and keep the record reachable by the people who actually get the questions.
• Know each partner’s rung on the evidence ladder. Use AS2 receipts where available - verified and retained - and treat a missing receipt as a failure; where the partner is a bare SFTP directory, record protocol completion and verify after delivery.
• Decide what happens after the last retry before it happens at 2 a.m.: bounded retries, visible while running, then park the file with its reason, replayable.

References

• RFC 4130 - MIME-based secure peer-to-peer business data interchange using HTTP (AS2): signed MDN receipts, MIC calculation and verification.
• X12 RFI 2099 - a 999 implementation acknowledgment confirms receipt and syntactic/implementation validity, not application-level acceptance.