A multimodal model can accept an image beside a paragraph, but that does not make the surrounding application a coherent multimodal system. The application still has to decide which image is current, whether text came from a user or OCR, how a database row relates to the pictured object, and what to do when those sources disagree. Most consequential failures occur at those boundaries, not inside the fusion layer.

The engineering thesis is therefore simple: treat every modality as typed, versioned, and potentially incomplete evidence. Preserve provenance through preprocessing, choose fusion according to the decision being made, and make degraded behavior explicit. A strong vision-language model cannot repair an order record joined to the wrong customer or recover detail destroyed by an aggressive thumbnail operation.

This chapter develops that approach through a product-return triage example. The system receives a customer note, package photographs, and structured order data. It may summarize evidence and route the case, but deterministic services retain authority over identity, eligibility, and money.

Define the Decision Before Choosing a Model

“Understand this image and text” is not a testable requirement. Begin with a bounded decision and an output contract. In the return example, the system must classify a submission as ready_for_review, needs_more_evidence, or manual_review, then list the evidence supporting that route. It must not approve a refund or infer facts absent from the supplied material.

That contract determines which modalities matter. A photo may show visible package damage. A note may explain when it occurred. An order record establishes which product was purchased and whether the request refers to that order. These sources are complementary, but they are not interchangeable. Text saying “the screen is cracked” does not prove visible damage; a photograph of a cracked screen does not establish ownership.

Represent missingness as data rather than an empty string or blank image. The distinction between not_provided, unreadable, unsupported_format, and rejected_by_policy affects both user guidance and monitoring. Likewise, record whether a field is observed, extracted, retrieved, or inferred. An output can then say that a serial number was read by OCR from image 2 instead of presenting it as an unquestioned property of the order.

A useful request envelope contains:

  • a stable request and tenant identifier;
  • the authenticated subject allowed to access the case;
  • one manifest entry per input, including media type, byte length, hash, capture time when trusted, and source;
  • explicit relationships, such as the image uploaded for order item 3;
  • transformation records for rotation, resizing, OCR, redaction, or transcription;
  • the exact structured-data schema version;
  • the permitted output kinds and side effects.

This envelope is more than bookkeeping. It prevents a model-friendly blob of tokens from becoming the system of record. It also makes retries reproducible: the application can tell whether it processed the same bytes and database snapshot or a different submission under the same conversational label.

Preserve Meaning During Modality Preprocessing

Each modality needs its own ingestion boundary. The goal is not to make everything look like text as early as possible; it is to produce validated representations while retaining enough original evidence to audit later decisions.

For images, decode with a hardened media library, reject decompression bombs and unsupported encodings, apply orientation deliberately, and enforce pixel as well as byte limits. A 10 MB compressed image can expand far beyond a safe in-memory budget. Create derivatives for model input, but retain the original object hash and the transformation recipe. Cropping, contrast adjustment, and downscaling can remove small labels or make a faint mark look stronger than it is. Never overwrite the original with the derivative.

For text, preserve language, paragraph boundaries, and source offsets. Normalizing Unicode and line endings can be useful, while indiscriminately collapsing whitespace can destroy tables, forms, or code. If text came from OCR, keep tokens or lines linked to image regions and attach engine/version metadata. OCR confidence is a signal about recognition, not proof that the recognized statement is true.

Structured records should cross a schema validator before entering model context. Convert database-specific values into explicit domain values: currency requires an amount and currency code; timestamps require a zone or an instant; status fields require a closed enum. Fetch only fields authorized for this task. A model does not need a customer’s complete profile to compare a submitted product identifier with an order item.

Preprocessing should be deterministic where possible. Given the same source bytes and versioned configuration, it should produce the same derivatives. That property supports cache keys, regression tests, and incident replay. Stochastic captioning or summarization belongs in a later model stage and should never silently replace source evidence.

Choose Fusion at the Right Boundary

Fusion means combining information across modalities, but there is no universally best point at which to combine it. The practical choices form a spectrum.

Strategy Mechanism Strength Main risk
Early fusion Encode modalities into one shared model context Captures fine cross-modal relationships Higher context and compute cost; harder attribution
Late fusion Run specialist models, then combine typed results Independent testing and graceful degradation Early summaries may discard useful detail
Tool-mediated fusion Let an orchestrator request OCR, search, or image inspection as needed Pays for expensive work selectively More control flow, latency, and tool-boundary failures
Rule-assisted fusion Combine model evidence with deterministic joins and rules Keeps business invariants outside the model Rules can become brittle if domain ownership is unclear

Use early fusion when the answer depends on spatial and linguistic relationships, such as matching a question to a region in a diagram. Prefer late fusion when modalities support separable claims, such as transcribing a label and validating it against a catalog. Tool-mediated fusion fits sparse workloads where only some requests need detailed image inspection. Real systems often mix all three.

Do not force agreement. A fusion stage should preserve contradictions such as photo_ocr.serial != order_item.serial, not average them into a confidence score. Confidence values from different components are rarely calibrated on the same population and cannot be safely multiplied or compared without validation. The output schema should instead name claim, source, extraction method, and any conflict.

Modality dropout also affects architecture. If users frequently submit text before photographs finish uploading, a system that requires a single early-fusion call may be unusable. A staged design can acknowledge the case, validate order data, and wait for media without inventing a complete answer. The workflow, rather than the model, owns when enough evidence exists to proceed.

Work Through a Return-Triage Example

Consider a submission with two photographs, the note “outer box crushed; device will not power on,” and an authenticated order identifier. The first photo shows the package exterior; the second shows a device label. The system’s job is to prepare a review packet, not decide compensation.

flowchart LR A[Authenticated submission] --> B[Validate manifest and ownership] B --> C[Decode and derive images] B --> D[Load minimal order snapshot] C --> E[OCR labels with regions] C --> F[Visual damage description] A --> G[Normalize customer note] D --> H[Deterministic identifier comparison] E --> H F --> I[Evidence fusion] G --> I H --> I I --> J{Evidence state} J -->|complete, consistent| K[Ready for reviewer] J -->|missing view| L[Request specific photo] J -->|conflict or uncertainty| M[Manual review]

The order service returns item SKU K-410, color black, and a stored serial suffix 82Q. OCR proposes serial suffix 82Q from a polygon in image 2. Application code performs the equality check because it is exact, cheap, and explainable. The visual model describes “compression and tearing along the upper-left package edge” and links that claim to image 1. It does not conclude that shipping caused an internal device failure.

A compact evidence object might look like this:

json
{
  "route": "ready_for_review",
  "claims": [
    {
      "kind": "visible_package_condition",
      "value": "compression and tearing on upper-left edge",
      "source": "image:1",
      "region": [0.04, 0.08, 0.46, 0.51]
    },
    {
      "kind": "serial_match",
      "value": true,
      "sources": ["image:2/ocr:serial", "order:item-3/serial-suffix"],
      "method": "deterministic_equal"
    },
    {
      "kind": "reported_symptom",
      "value": "device will not power on",
      "source": "customer:note"
    }
  ],
  "limitations": ["No evidence establishes the cause of the power failure"]
}

If OCR instead reads B2Q, the system should not ask the vision model to decide which identifier “looks right.” It can retry OCR with the original-resolution crop, ask the user for a clearer label photograph, or send the case to a reviewer. If image 1 is missing, it can request the exact exterior view rather than returning a generic failure. These are workflow decisions grounded in evidence state.

Keep OCR, Models, and Tools in Their Proper Roles

General multimodal models are attractive because one call can apparently replace several components. That convenience can erase useful boundaries. A dedicated barcode decoder provides check digits and a clear failure. OCR can return text with coordinates. A database query can enforce tenant filters. A vision-language model is better used for relationships and descriptions that those deterministic tools cannot express.

Choose the narrowest competent component for each claim. Use a barcode library before asking a model to read a barcode visually. Use geometry to determine whether an OCR region overlaps a required form field. Use model reasoning to associate a natural-language question with an area of a complex diagram. The orchestrator should record which component produced each result and should validate tool output against a schema before fusion.

Tool boundaries also protect availability. OCR may time out while order retrieval succeeds. The workflow can retry the idempotent extraction stage without repeating the database snapshot or model call. Cache immutable derivatives by source hash, transformation version, and tool version. Avoid caching authorization decisions or mutable business data under the media hash.

Content within an image or document is data, even when it resembles an instruction. The multimodal model should receive a clear task boundary, and tools should be exposed through application-controlled capabilities rather than credentials embedded in context. This is only one part of defending model applications; the important multimodal point is that pixels, OCR text, filenames, and document metadata all cross the same untrusted-content boundary.

Design Explicit Failure Modes and Fallbacks

Multimodal pipelines fail partially. Images arrive late, cameras blur labels, OCR supports the wrong script, a structured record changes during processing, or one model region becomes unavailable. A robust interface distinguishes degradation from success.

Define a capability matrix for each route. ready_for_review may require an authenticated order snapshot, at least one usable condition photo, and either a deterministic item match or explicit reviewer handling. A text-only submission may still be accepted as needs_more_evidence; it must not be presented as fully assessed. Show users which input is missing and allow replacement without restarting the case.

Set stage budgets inside an end-to-end deadline. Decode and validation happen before expensive inference. Run independent OCR and record retrieval concurrently when their resource limits permit it. Downscale for an initial pass, then inspect original-resolution crops only when needed. These choices reduce latency and cost, but aggressive staging can miss small details, so evaluate the routing policy as well as each model.

Fallbacks must preserve semantics. Replacing a failed visual model with OCR-only processing is acceptable for identifier extraction but not for damage description. A smaller model may support common image formats but fail on dense diagrams. State the reduced capability in the result and route uncertain cases safely. “Best effort” should never mean silently dropping a modality and returning the same confidence label.

Common failure modes include stale joins, transformations that erase evidence, duplicate uploads treated as independent corroboration, model summaries that outlive corrected source data, and disagreement hidden by a single aggregate score. Version every derived artifact, deduplicate by content hash while retaining upload events, and invalidate downstream results when a source or transformation changes.

Protect Privacy Across the Full Evidence Lifecycle

Images often contain more sensitive information than the task requires: faces, addresses, reflections, location metadata, neighboring documents, or a computer screen in the background. OCR turns some of that latent information into searchable text. Structured data adds stable identifiers that make accidental disclosure easier to connect to a person.

Apply authorization before retrieval and before inference, not only before showing the final answer. Minimize the crop and fields sent to each component. Strip unneeded metadata, redact regions when the task permits it, and keep originals in a separately controlled store with a defined retention period. Derivatives need retention rules too; a thumbnail and OCR transcript can remain sensitive after the source is deleted.

Know where every processor runs, what it retains, whether inputs may be used for provider training, and which region receives the data. Encrypt transport and storage, isolate tenants in caches and object keys, and avoid placing raw media or extracted personal data in ordinary logs. Trace identifiers and hashes are usually more useful operationally than copied payloads.

Human review is another data boundary. Reviewers should see only the evidence required for the decision, with access recorded and time-limited. Feedback exports must preserve that minimization. A convenient archive of reviewed screenshots can become an uncontrolled training dataset unless consent, purpose, deletion, and ownership are explicit.

Verify Components, Fusion, and Operations

Testing only complete happy-path examples leaves the dangerous joins unexamined. Build a matrix across modality presence, quality, language, format, device, and business slice. Include missing images, rotated photos, duplicate files, unsupported encodings, empty notes, stale records, conflicting identifiers, and multiple products in one frame. Keep a set of consented or purpose-built cases whose expected evidence and route are reviewed by domain experts.

Component checks should verify deterministic preprocessing, schema rejection, OCR region mapping, authorization filters, and cache invalidation. Fusion checks should assert source attribution and contradiction preservation. End-to-end checks should evaluate whether the system requests the right missing evidence and whether fallback behavior is visibly degraded. A correct route with an unsupported explanation is still a failure.

Metamorphic tests are especially valuable when exact prose varies. Rotating an image while preserving orientation metadata should not change the identified item. Reordering independent uploads should not change the route. Removing a source must remove claims supported only by that source. Replacing order data with another tenant’s record must be denied before model execution. Compressing an image within the supported quality range should not invent a new defect.

In operation, trace each stage with source and derivative hashes, component versions, queue time, inference time, token or media units, validation outcomes, fallback reason, and final route. Sample payloads only under a privacy-reviewed policy. Monitor unreadable-input rates, modality dropout, conflict rates, retries, manual-review routing, latency, and cost by device and workflow. A sudden rise in OCR conflicts may indicate a camera-app change, a new label design, or a preprocessing regression rather than model drift.

When an incident occurs, disable the narrow capability first: stop automated routing, force manual review, or bypass a faulty derivative while preserving submissions. Replay authorized, minimized cases against the previous version and inspect where provenance diverged. The defining property of a dependable multimodal system is not that every modality always works. It is that the system knows what evidence it has, what was done to it, and which decisions remain justified when one part fails.