The Sensors Stopped Speaking Separate Languages

MULTIMODAL LEARNING

The same place can arrive as optical reflectance, radar backscatter, elevation, temperature, rainfall, a land-cover label and a paragraph written by a field team. For decades these were processed in separate pipelines because they are not merely different file formats. They describe different physical properties in different mathematical spaces.

Multimodal learning tries to build a shared internal space in which those observations can inform one another. The model is not asked to pretend radar is optical imagery or that a sentence is a temperature field. It is asked to learn which structures correspond across them, which relationships are useful, and when one source should change the interpretation of another.

This is a more difficult problem than adding extra channels to an image. A red band and a near-infrared band are different measurements from the same instrument family. Radar and optical sensors have different imaging physics. Elevation is nearly static. Weather changes hourly. Text may refer to objects and causes rather than measurements. A shared model has to preserve those differences while still discovering common structure.

ALIGNMENT

The hard part is deciding what corresponds to what

Multimodal learning usually begins with separate encoders or modality-specific input paths. Each converts its own data into an internal representation. Training then encourages representations from the same place, time or semantic object to become mutually informative. The difficult word in that sentence is same.

Spatial alignment is rarely perfect. A radar image and an optical image may have different geometry, viewing angle and resolution. A weather variable may describe a grid cell covering thousands of optical pixels. A field observation may be a point inside a parcel. Putting all of them on one grid creates a common coordinate system, not a common physical footprint.

Time adds another mismatch. A storm can alter radar backscatter between two optical acquisitions. A field report may describe conditions observed in the morning while the satellite passed in the afternoon. If the model treats every nominally co-located observation as simultaneous, it can learn disagreement that was created by the clock rather than by the Earth.

Semantic alignment is harder still. A text note saying flooding blocked the eastern road contains a relation between an event, an object and a direction. A satellite image contains no explicit word road. The model has to learn that a linear feature in one representation corresponds to a named concept in another. This is where multimodality begins to look less like data fusion and more like translation.

FUSION INSIDE THE MODEL

There is more than one way to build a shared space

One design combines modalities early. Measurements are aligned as closely as possible, encoded and allowed to interact from the first layers. This can capture fine cross-modal relationships, but it assumes the inputs are available together and makes the model sensitive to missing sources.

Another design lets each modality build a stronger representation first, then combines the representations later. This preserves modality-specific structure and makes missing inputs easier to handle. It can also postpone useful interaction until after some information has already been compressed away.

Cross-attention offers a more selective arrangement. One modality can query another for the pieces relevant to the current task. An optical representation can attend to radar where cloud creates ambiguity. Text can attend to image regions associated with a named object. The model does not need every source to contribute equally to every decision, which is fortunate because the sensors have never agreed to equal importance.

The architectural choice is not cosmetic. It determines whether the model treats disagreement as noise, preserves it as evidence, or learns to route around a missing input. A planetary system will routinely encounter all three situations. Multimodality only helps if the design can tell them apart.

A REAL SYSTEM

TerraMind made generation part of the reasoning process

ESA and IBM released TerraMind in April 2025 as a multimodal Earth-observation foundation model. The research paper describes pretraining across nine geospatial modalities using a large globally distributed dataset, while ESA's public summary describes more than nine million samples spanning radar, optical imagery, topography, land cover, elevation, geolocation and related data types.

TerraMind uses both token-level and pixel-level representations. The token representation carries higher-level context efficiently. The pixel-level path preserves finer spatial information that can matter for segmentation and localized tasks. The model therefore does not have to choose entirely between global context and spatial detail before learning begins.

Its more unusual idea is Thinking-in-Modalities. The model can generate an intermediate modality and use that generated representation while solving a downstream task. A missing or unprovided representation can become a temporary hypothesis that helps the model reason across the sources it does have.

That is technically interesting and epistemically dangerous in exactly the right way. A generated modality is not an observation. It is a model-produced estimate that may be useful because the learned relationships are strong. If the rest of the system forgets that distinction, synthetic evidence can quietly acquire the authority of measured evidence.

PROVENANCE

A shared representation still needs to remember who actually saw what

Multimodal models are attractive because they can turn several partial views into one useful representation. That compression can also erase the reason the views differed. Radar may be noisy because of speckle. Optical imagery may be obscured by cloud. A text report may be delayed or subjective. A generated modality may never have been measured at all.

For low-stakes classification, the model may only need the combined answer. For a system that will trigger another observation, change a forecast or influence a decision, provenance has to survive. Which source contributed? When was it observed? Was it measured, inferred or generated? How much did the output depend on it?

This is the difference between a shared representation and a shared truth. The representation can be excellent while one contributor is stale or wrong. If the system has no way to recover that lineage, later reasoning becomes confident about evidence whose origin it cannot inspect.

The next chapters push this architecture further. A foundation model asks whether one representation can serve many tasks. Sensor fusion asks how several sources update one state. Both are more useful when the model can combine evidence without forgetting the route by which the evidence arrived.