File upload features connect an attacker-controlled byte stream to storage, parsers, background workers, and eventually other users. A check such as “the filename ends in .pdf” protects none of those boundaries. The body may be larger than declared, the apparent image may contain a different format, a valid archive may expand without bound, and a clean file can still become public under the wrong tenant.
The secure design is a pipeline with an explicit trust transition. Bytes begin untrusted, enter bounded quarantine, pass format-specific validation and policy checks, and become available only through a separate publication step. No individual scanner or MIME check proves a file safe. The pipeline instead limits what each failure can affect, preserves enough evidence to investigate it, and keeps authorization attached to the object throughout its lifecycle.
This article develops that model for a customer-support service that accepts images and PDF attachments. Users may upload directly through the API or through a short-lived object-storage reservation. In either path, the same state machine and policy decide whether an attachment can be referenced by a ticket or downloaded by an agent.
Start with Threats and an Explicit Lifecycle
Write the upload policy before implementing multipart parsing. It should define accepted media families, maximum encoded and decoded sizes, maximum item count, retention, who may create and read an object, whether active content is allowed, and what downstream transformations occur. “Any document” is not a policy because every parser adds a different attack surface.
The support service accepts JPEG, PNG, and PDF; rejects encrypted PDFs because scanners cannot inspect them; rejects archives entirely; removes image metadata during re-encoding; and never serves an original file inline. Those are product choices, not universal rules. A legal-document system may need original signatures and encrypted files, in which case it needs different isolation and review controls.
Represent the lifecycle durably:
Approved means the stored candidate passed a particular policy and scanner version. Published means an authorized business record references an approved derivative. Keeping those states separate prevents a scanner callback from deciding who may see a file. A later malware-signature update can move an approved object back into quarantine or revoke its derivative without rewriting history.
Assign a server-generated upload ID and opaque storage key. Preserve the user-supplied filename only as bounded display metadata after removing path components and control characters. Never use it as a filesystem path, object key, response header without safe encoding, or uniqueness mechanism. Two users can upload invoice.pdf; that coincidence must have no storage or authorization meaning.
Authorize Upload Intent Before Accepting Bytes
Authentication establishes an actor, while upload authorization asks whether that actor may add this kind of attachment to this tenant and prospective ticket. Check quotas, account state, allowed media policy, and object count when creating an upload reservation. Do not postpone every decision until after expensive bytes have arrived.
A reservation can contain:
{
"uploadId": "upl_01JY3F4KQ9W2",
"tenantId": "tenant_17",
"actorId": "user_204",
"purpose": "support-attachment",
"maxBytes": 10485760,
"allowedTypes": ["image/jpeg", "image/png", "application/pdf"],
"expiresAt": "2026-06-18T12:15:00Z"
}
The server owns these values. A client-supplied tenantId, claimed MIME type, or ticket ID is only input to validate against authenticated context. Bind a reservation to one opaque object key and one use. If direct object-store upload is supported, issue credentials limited to that key, method, encoded-size range where the storage platform can enforce it, required checksum fields, and short expiry. Do not grant a prefix that lets the client overwrite another user’s quarantine object.
Reservation expiry bounds abandoned work but does not guarantee a late transfer stopped. Finalization must compare the committed object’s key, generation, size, checksum, and reservation state. An expired reservation cannot be revived merely because bytes appeared. Cleanup should delete only the exact unclaimed object generation to avoid racing a replacement.
Rate limits need several dimensions. Limit reservation creation, concurrent receiving streams, bytes per actor and tenant, and retained quarantine storage. Request-count limits alone allow a few huge bodies to exhaust bandwidth or disk. Reject early when declared length exceeds policy, but still count actual streamed bytes because Content-Length may be absent, wrong, or hidden by chunked transfer.
Cross-site request protections depend on the API’s credential model. Cookie-authenticated upload endpoints need the same origin and anti-forgery controls as other state-changing routes. Bearer credentials must not appear in upload URLs or object names. Log actor and reservation IDs, not authorization headers or raw file contents.
Stream into Quarantine with Hard Bounds
Buffering an entire file in application memory turns the configured maximum into multiplied process pressure: concurrent uploads, parser copies, and retries can exceed the heap before validation runs. Stream the body to quarantine while counting bytes and computing a cryptographic digest. Apply backpressure from the destination rather than reading faster than storage can accept.
The ingress path should enforce, in order, a request deadline, header limits, multipart part count, per-part size, total body size, and allowed field names. A library parser is preferable to hand-written boundary parsing, but its limits must be configured and tested. Stop reading and abort the destination as soon as a limit is crossed. Draining or closing the connection is an HTTP-server decision; continuing to persist bytes is not.
import { createHash } from 'node:crypto';
import { Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
class ByteLimit extends Transform {
#seen = 0;
constructor(private readonly maximum: number) {
super();
}
override _transform(
chunk: Buffer,
_encoding: BufferEncoding,
callback: (error?: Error | null, data?: Buffer) => void,
): void {
this.#seen += chunk.length;
if (this.#seen > this.maximum) {
callback(new Error('UPLOAD_TOO_LARGE'));
return;
}
callback(null, chunk);
}
}
async function receiveToQuarantine(input: {
body: NodeJS.ReadableStream;
destination: NodeJS.WritableStream;
maxBytes: number;
}): Promise<{ bytes: number; sha256: string }> {
const digest = createHash('sha256');
let bytes = 0;
const observe = new Transform({
transform(chunk: Buffer, _encoding, callback) {
bytes += chunk.length;
digest.update(chunk);
callback(null, chunk);
},
});
await pipeline(input.body, new ByteLimit(input.maxBytes), observe, input.destination);
return { bytes, sha256: digest.digest('hex') };
}
The sketch omits server-specific cancellation wiring, but production code must abort the object upload when the client disconnects or the deadline expires. A partial object must never be considered quarantined. Use the storage service’s complete/abort semantics, a temporary key plus atomic commit where supported, or an explicit metadata state that no reader treats as complete.
Store quarantine objects in a private bucket or account with no public route, no content execution, separate credentials, restrictive lifecycle rules, and preferably a distinct trust boundary from published derivatives. Encryption at rest is useful but does not change whether malicious bytes can reach a parser. Avoid mounting quarantine storage into web servers or employee desktops.
Identify Content Without Trusting One Signal
The client-provided MIME type and filename extension are hints for user experience. The pipeline should compare several independent signals: allowed extension, declared type, magic bytes, successful structural parsing, and format-specific properties. Disagreement is a reason to reject or review, not an invitation to choose the most permissive answer.
Magic-byte sniffing catches obvious substitutions but is not full validation. Many formats share container signatures, support leading data, or permit embedded content. A valid PDF parser should locate required structures within bounded work, reject encryption under this policy, cap page and object counts, and identify active features that the service forbids. An image decoder should enforce width, height, frame count, color-space, and decoded-pixel bounds before allocating output.
Compressed and nested formats need expansion budgets. A tiny archive can expand into enormous output, contain deeply nested entries, use duplicate names, or escape a target directory with ../ paths. The example service rejects archives, which removes that entire class. A product that accepts them must cap entry count, total expanded bytes, compression ratio, nesting, path depth, and processing time, and extract inside an isolated workspace without following links.
Polyglot files exploit the fact that different parsers recognize different portions of the same bytes. Renaming or checking only a prefix does not neutralize them. The safest publication path for supported images is decode and re-encode into a narrowly configured format, producing a new derivative and discarding metadata and trailing bytes. For PDFs that must remain PDFs, combine strict structural policy, malware scanning, safe download disposition, and parser isolation rather than claiming sanitization is perfect.
Treat parser errors as data, not as verbose responses to the uploader. Return a stable code such as unsupported_file or file_rejected; retain a normalized internal reason. Raw parser messages can disclose library versions, paths, or file contents. Likewise, do not use a browser’s ability to preview a file as validation.
Quarantine, Scan, and Transform Asynchronously
After the object is durably quarantined, enqueue inspection using the upload ID and exact object generation. The worker loads policy from the reservation, verifies recorded size and digest, sniffs and parses the format under resource limits, calls malware detection, and creates any safe derivative. Each step records its tool or policy version and a bounded result.
Scanning is one control, not an oracle. Signature scanners can miss new malware, return transient errors, or be bypassed by encrypted content. Decide whether each scanner outcome is pass, reject, retry, or manual review. “Scanner unavailable” must not default to approved. At the same time, retrying forever keeps user data and work stuck; set an inspection deadline and surface an unavailable outcome to operators.
Run risky parsers with least privilege in isolated processes or containers: no ambient cloud credentials, no unrestricted network, read-only access to the one input, bounded CPU and memory, a temporary output directory, and a hard deadline. A container is not a complete security boundary, but process separation and syscall or sandbox controls reduce what a parser exploit can reach. Keep parsing libraries patched and inventory which version inspected every object.
Derivatives get new opaque keys and checksums. Never overwrite the quarantined original and call it sanitized; immutable inputs make rescanning and incident analysis possible within retention policy. For an image, publish only the re-encoded derivative. For a PDF, the approved object may be a copied generation with metadata that binds it to the original digest and inspection decision.
Cache scan results carefully. A digest can avoid rescanning identical bytes under the same tenant policy, engine versions, and format rules, but a previous clean verdict under an older signature set is not timeless. Cross-tenant digest sharing can leak that another tenant uploaded the same sensitive document through timing or status. The simplest safe default is tenant-scoped caching with explicit scanner-version invalidation.
Publish One Attachment Through a Recoverable Workflow
Follow a user uploading invoice.pdf to ticket tic_558. The API authenticates user user_204, confirms they may edit that ticket in tenant_17, and creates reservation upl_01JY3F4KQ9W2. The client streams bytes to its dedicated quarantine key. The server observes 824,311 bytes, computes digest d1...9a, commits the object generation, and moves the record to quarantined.
An inspection worker verifies the generation and detects a PDF despite the declared application/pdf; it then parses the structure, confirms the document is unencrypted and within page/object limits, and receives a clean scanner result. It writes an approved copy with a content-disposition-safe display name and records the original digest, scanner versions, and policy version. Approval alone does not attach it.
The client now calls an attach operation. In one database transaction, the service rechecks that the actor may modify tic_558, that the ticket and upload share a tenant, that the upload purpose is support-attachment, that state is approved, that it is not expired or already consumed elsewhere, and that the attachment count remains below policy. It inserts the ticket attachment and marks the reservation consumed.
This second authorization check matters because permission may have changed while scanning, or a malicious client may try to attach the approved object to another tenant’s ticket. Binding authorization only at reservation time creates a time-of-check/time-of-use gap. Binding only at attachment time lets unauthorized users consume storage and scanners. Both gates have distinct jobs.
Suppose the attach response is lost after commit. Retrying with the same upload and ticket returns the existing attachment through a uniqueness constraint. Suppose the scanner callback arrives twice; its conditional transition from the exact inspecting lease and object generation makes the second result harmless. Suppose cleanup runs concurrently; it deletes only expired, unconsumed records and exact generations, so it cannot remove the newly published derivative.
The API can expose polling state without revealing internals: receiving, processing, ready, rejected, or expired, plus stable user-actionable codes. A callback is optional, but polling remains useful when the callback fails. Progress claims should reflect completed stages, not fabricated percentages for scanners that provide no measurable progress.
Serve Files Without Reopening the Boundary
Every read begins with authorization against the current attachment and tenant, not possession of an object key. Keep storage keys unguessable anyway, but do not treat obscurity as access control. The application may stream the object after checking policy or issue a short-lived signed download URL limited to one key and response disposition. Long-lived public URLs defeat revocation and leak through logs, referrers, and shared messages.
Set response headers from server-owned metadata. Use Content-Type from validated format, X-Content-Type-Options: nosniff, and Content-Disposition: attachment for content that should not execute inline. Encode display filenames using the relevant HTTP rules and include a conservative ASCII fallback. Never reflect CR, LF, quotes, or path separators directly from the original name.
Serving from a separate origin without application cookies reduces active-content risk. A strict content security policy can add defense, but attachment disposition and format policy remain primary. For image derivatives intended inline, ensure re-encoding really produced the allowed image format and that SVG or HTML never enters through a broad image/* rule.
Revocation should be a metadata operation that immediately blocks new authorization and signed-URL issuance. Existing signed URLs remain valid until expiry, so choose short durations based on usability and incident needs. If the storage platform supports token revocation or a gated CDN, use it for higher-risk documents. Deleting bytes is separate and may be delayed by retention, backups, or legal holds; do not claim deletion before those systems agree.
Audit reads and administrative downloads according to sensitivity. Avoid logging full object URLs with embedded signatures. Metadata APIs should reveal only original display name, safe type, size, state, and relevant timestamps to authorized actors, not quarantine keys or scanner diagnostics.
Test Failure Windows and Operate the Pipeline
Unit tests should exercise filename normalization, policy matching, signature or token scope, MIME disagreement, malformed containers, size boundaries, image dimension overflow, encrypted PDFs, and response-header encoding. Keep a curated corpus of benign edge cases and inert adversarial fixtures. Document how fixtures were produced; do not depend on live malware in ordinary developer environments.
Integration tests need streaming and state failures:
- send a declared-small body that crosses the actual byte limit;
- disconnect halfway through a multipart part and verify no complete object exists;
- finalize the wrong object generation or checksum;
- expire a reservation while a direct upload completes;
- terminate an inspector after parsing but before recording its verdict;
- deliver duplicate and stale scanner results;
- revoke ticket permission between reservation and attachment;
- race publication with cleanup and rescanning;
- request a derivative through another tenant’s attachment ID.
Use parser fuzzing and decompression tests in isolated CI jobs appropriate to those libraries. Assert resource ceilings and timeouts, not just rejection. A parser that eventually rejects after exhausting memory has failed the security objective. Test signed download expiry and header behavior in real browsers for supported inline formats.
Operational metrics should show reservations, received bytes, quarantine age, inspection duration by stage, scanner unavailable rates, rejection codes, approved-but-unattached objects, publication failures, cleanup backlog, and download denials. Keep upload IDs searchable in logs and traces while excluding content. Alert on growing oldest-quarantined age, storage nearing quota, unusual type mismatch, repeated parser crashes, and a jump in cross-tenant authorization denials.
Runbooks should cover disabling one format, pausing publication while allowing bounded ingestion, rotating direct-upload credentials, rescanning affected policy versions, revoking derivatives, and draining expired quarantine. Rehearse scanner outages and cleanup recovery. Periodically verify lifecycle deletion with inventory reports rather than assuming a configured rule ran.
The key invariant is simple: received bytes are never equivalent to an authorized attachment. A secure pipeline preserves that distinction through bounded streaming, private quarantine, independent type validation, isolated inspection, immutable metadata, explicit publication, and current read authorization. Each control is fallible; together they make failure contained, visible, and recoverable.