One concept, reached by different senses
A picture of a cat and the sound of its meow reach us through different senses. Yet both mean the same animal. Some neurons in the brain respond to a concept like a cat, rather than to the exact image shown, and some respond to both sights and sounds. We take this as an inspiration rather than a claim about the brain. It suggests a space where different inputs lead to one concept, whichever sense carried them.
Such a space must also let things go. An apple is a shape, a colour and a form. It is not the exact pixels of one photograph, the lighting of one room or the wording of one description. We call that detail noise. It belongs to one view of a thing, rather than to the thing itself.
An AI model turns every input into a long list of numbers. This list acts as coordinates, placing the input as a point in a space of many dimensions (an embedding). We want this space to have two properties. It must be modality-free: a scene lands at the same point whether it arrives as a picture or as text, the two modalities we use here. It must also be noise-free, keeping only what different views of a scene share.
Two regions and a gap
The best-known models connecting pictures and words are CLIP, released by OpenAI in 2021, and Google’s SigLIP and its 2025 successor SigLIP‑2. They act as the eyes for many AI systems that look at pictures and talk about them. We compare Phi‑0 with these two models throughout this work. Both learn from vast collections of web images paired with captions. Given a batch of pairs, they learn to score true matches higher than mismatched ones (contrastive learning).
The space they produce is split. Images gather in one region and captions in another. A distance remains between the two (the modality gap). Even an image and its own caption stay far apart. Nothing in their training prevents this. A true pair can rank first while its two points sit in different regions. Such a model scores pairs. It does not predict one side from the other.
Generative models handle both pictures and words in one network by rebuilding every pixel and every word. Yet this training rewards keeping exactly the detail that a noise-free space should drop.
SigLIP‑2 trains its image and text encoders to score every picture–caption pair in a batch: true pairs high, mismatched pairs low. Nothing asks an image and its caption to meet, so images and captions settle in separate regions.
Phi‑0 uses no mismatched pairs. From the image it predicts where the caption lies, and from the caption where the image lies. Each guess is scored by its distance from the truth. The two points are drawn together while SIGReg keeps the cloud spread out.
Predicting one side from the other
Phi‑0 learns by guessing the caption from the picture, and the picture from the caption. It predicts the meaning of one view from another, rather than recreating the exact input (a joint-embedding predictive architecture, or Φ-JEPA).
The model starts with two existing components, or encoders, which we leave unchanged (frozen). For pictures, it uses V-JEPA 2.1, a vision model from Meta’s AI research lab that learned to see mostly by watching video. For text, it uses ModernBERT, an open language model released in 2024 that reads text and turns it into numbers, a modern update of the widely used BERT. Above them sits a small shared trunk. The picture and the caption pass through it separately, but through mostly the same machinery, which turns each into a single point in the space.
Two small networks, the predictors, then guess across. One predicts the caption’s point from the picture’s point. The other predicts the reverse. Each guess is scored by how far it lands from the true point (an L2 loss). Because both points are learned, each is shaped to be predictable from the other. Detail carried by only one side, like lighting or exact phrasing, cannot be guessed across. The model is pushed to drop it. What survives is only what the picture and the caption share.
Prediction alone has a shortcut. If the model sent every input to the exact same point, every guess would be perfect and the space would be useless (collapse). A rule called SIGReg stops this cheating. It keeps the cloud of points evenly spread in every direction and penalises it for bunching up. Technically, it pulls each batch towards an isotropic Gaussian.
Each picture is also seen as eight crops, including close-ups. Each caption is seen as eight passages taken from up to three descriptions of the picture. Crops are never flipped or recoloured, because captions name sides and colours. The picture’s final point is the average of its eight views, and so is the caption’s.
The model is small. It was trained on 160M pictures from the web, each with three detailed captions. Training took 192 GPU-hours on one machine with eight NVIDIA H100 GPUs. The parts that learn hold about 73M parameters, the numbers adjusted in training.
One space, not two
The clearest result is the shape of the space. We take 5,000 everyday photographs from a standard test collection, each with a one-line caption written by a person. Phi‑0 never saw them in training, so the test is out of distribution: on data unlike its training data. Drawn as points, SigLIP‑2’s images and captions form two separate caps. Phi‑0’s mix over one sphere.


The drawing keeps only three of the space’s 768 directions. The table below measures the same pairs in the full space, four different ways. It scales from single pairs up to the layout of the whole cloud.
One space, or two?
We take 5,000 everyday photographs and their one-line captions. Phi‑0 never saw these in training. We measure where each model places a picture relative to its own caption.
marks the best of the three in each row
Closeness to own caption
Each picture and caption becomes a direction in the space. This compares a picture’s direction with its own caption’s. A score of 1 means the same direction. It shows if seeing a scene and reading about it end in the same place.
Mean matched image–caption cosine similarityhigher is better
A note for the careful reader
Averaged over the 5,000 pairs. A low value does not stop a model from matching, as SigLIP‑2 and CLIP rank by relative similarity. Phi‑0’s two points are never trained to be equal, only to be predictable from each other.
Gap between pictures and captions
We find the centre point for all pictures and the centre for all captions, then measure the distance between them. It runs from 0 (one shared centre) to 2. A large value means pictures and captions occupy separate regions.
Modality gap ‖μv − μt‖ on L2-normalised embeddings (Liang et al.)lower is better
A note for the careful reader
Points are scaled to length 1, so the maximum distance is 2. This is the plain distance between centres. The paper’s sphere figures print a different number, gap divided by spread, which should not be mixed with this measure.
Shared nearest neighbours
For each pair, we find the 10 most similar pictures and the 10 most similar captions. We count how many name the same pairs. 100% means all ten match. It asks whether looking and reading agree on what resembles what.
Mutual k-nearest-neighbour alignment, k = 10 (Huh et al.)higher is better
A note for the careful reader
Averaged over the 5,000 pairs. A score of 21.3% means about two of the ten neighbours are shared. Two random lists of ten would overlap by about 0.2%. The two sides are never compared directly.
Same overall layout
This compares the pattern of similarities among all pictures with the pattern among all captions. 0% means unrelated patterns, while 100% means they are identical apart from rotation and scale. A high value means both sides are arranged in the same way.
Linear centred kernel alignment (CKA), image against caption embeddingshigher is better
A note for the careful reader
Computed on the image and caption points of the same 5,000 pairs.
In shortPhi‑0 places a picture and its caption in nearly the same spot. SigLIP‑2 and CLIP keep pictures and captions in separate regions, even though they still rank the right pairs together.
Each measure tells the same story. Where the contrastive models keep two regions, Phi‑0 builds one. An image and its own caption point in nearly the same direction. The picture side and the caption side also agree on what resembles what. On held-out pairs similar to its training data (in distribution), they are closer still. On the table’s first measure, Phi‑0 reaches 0.885 against 0.133 for SigLIP‑2. The gap is small rather than gone.
What the encoders knew is kept
A shared space would be of little use if pictures lost their detail on the way in. To check, we freeze each model and train small probes on its picture features. These are simple add-ons that learn to outline objects, judge depth, name the main object, or box the thing a phrase describes. Every model gets the same probes, code and data.
Does it still see?
Each model is frozen. A small add-on is trained on its picture features, using the same code and data for all three models.
marks the best of the three in each row
Outlining objects
Every pixel is labelled with what it shows, like a person or background. The predicted region is scored by its overlap with the true one. 100% is a perfect outline. It tests whether the features remember where things are.
mIoU, semantic segmentation, linear probe per patchhigher is better
A note for the careful reader
The score is averaged over the object classes. The frozen V-JEPA 2.1 image encoder that Phi‑0 is built on scores 73.9% on its own.
Judging distance and depth
From one indoor photo, the features estimate each point’s distance from the camera. The score is the typical error in metres, so 0 is perfect. Much of Phi‑0’s lead comes from its frozen image encoder, which errs slightly less on its own.
RMSE (m), indoor depth estimation, linear probe per patchlower is better
A note for the careful reader
Large mistakes count for more. Phi‑0 is typically off by about 54 cm, SigLIP‑2 by 66 cm and CLIP by 72 cm. The shared trunk costs a little depth accuracy compared to the raw V-JEPA 2.1 encoder.
Naming the main object
An add-on reads the frozen features and picks one of 1,000 categories, like a dog breed. The score is the share of correct first guesses. It tests what visual details the features hold, not how well they match a phrase.
Top-1 accuracy over 1,000 categories, attention-pooling (MAP) probehigher is better
A note for the careful reader
Compare this to the last row of Table C: 81.1% here against 29.2% there for Phi‑0. The visual information is present in the features; the weakness lies in matching a short label.
Finding described objects
Given a photo and a phrase, the model must draw a box around that specific thing. A box is correct if it overlaps the true box by at least half. The score is the share of correct boxes.
[email protected] (box IoU ≥ 0.5), referring expressions, frozen-feature probehigher is better
A note for the careful reader
The phrase is read by the same frozen text encoder for every model. With an optional grounding decoder trained on extra box data, Phi‑0 reaches 76.4%. The frozen V-JEPA 2.1 encoder alone scores 72.9%.
In shortPhi‑0 keeps most of what its image encoder could see. It leads on outlining and depth, sits in the middle on naming objects, and trails SigLIP‑2 at finding what a phrase describes.
Much of this skill is inherited. On its own, the frozen V-JEPA 2.1 encoder judges depth slightly better than Phi‑0, at 0.495 against 0.538 metres. The trunk keeps most of that ability. Because both encoders stay frozen, Φ-JEPA is also a way to distil two encoders into one space. It largely keeps what each could do alone, and adds what neither had: a place where an image and its caption meet.
The text side shows the same trade. ModernBERT’s ability to find relevant passages is nearly unchanged after the trunk. What it loses is mostly sensitivity to the exact form of a sentence, such as whether it is grammatical, scoring 5.3 points lower on a standard grammar test. A space built around what a caption describes has little reason to keep that detail.
It finds the scene, not the snapshot
In retrieval, a model is given a caption and must find its picture among many, or the reverse. The usual score is how often the true match comes first. On the standard benchmarks, which use short captions, Phi‑0 is well behind.
The reason lies in how it learns. From a caption, Phi‑0 predicts where the picture should lie. No mismatched pairs push that guess away from look-alikes. When a short caption fits many photographs, the best guess is their centre: the concept, not one photograph. The right picture is usually among the first few rather than first. Two examples from the paper show this from both ends.
“A man cooking food on the stove.”
Phi‑0 misses first place, yet all twenty of its first pictures show the scene the caption describes.
“In the center of the image, a young girl is the main subject. She’s dressed in a vibrant blue jacket … holding up a large slice of pizza …”
SigLIP‑2 reads at most 64 text tokens, so it sees only the opening of this description.
Caption length matters. The usual test captions are short, about ten to twelve words on average. Phi‑0 was trained on captions of about 49 words. A short caption fits many photographs; a long one fits few. From the shortest captions to those of 31–60 words, Phi‑0’s score nearly triples. On the longest captions, it overtakes SigLIP‑2, which reads only the first 64 tokens, or word pieces, of a caption. CLIP does better still there.
Short prompts show the same pattern in zero-shot classification. Here, a phrase such as “a photo of a dog” stands in for each of 1,000 everyday categories, from breeds of dog to kinds of tool. Phi‑0 does poorly. Yet a probe trained on the same picture features names objects well. The visual information is there. The weakness lies in matching a short phrase to one class.
Matching words to pictures
Given a caption, each model must find the right picture among a thousand or more. Given a picture, it must pick its name from 1,000 labels. Nothing is retrained.
marks the best of the three in each row
Short caption, exact match first
The model ranks 1,000 pictures for 1,000 short captions. The score is how often the exact picture comes first. Phi‑0 predicts the typical picture for a description, so a similar photo often comes first instead of the exact one.
Text→image Recall@1, 1,000-image test set, zero-shothigher is better
A note for the careful reader
Captions average about 12 words. Random ranking would score 0.1%. For Phi‑0, the caption is turned into a predicted picture point, which is then ranked against the pictures.
Short caption, in top ten
The same test, but a hit counts if the exact picture is anywhere in the first ten. From first place to top ten, Phi‑0’s score doubles. This shows that Phi‑0’s misses are often near misses.
Text→image Recall@10, 1,000-image test set, zero-shothigher is better
A note for the careful reader
Random ranking would score 1%. Across three standard test collections, Phi‑0’s score rises 2.0 to 2.6 times from first place to top ten, compared to 1.2 to 1.6 times for SigLIP‑2.
Long description, exact match first
The same first-place test, but with very detailed descriptions of 151 to 200 words. A long description fits far fewer photos than a short one, which suits a space built around meaning. Phi‑0 beats SigLIP‑2 here but stays behind CLIP.
Text→image Recall@1, 151–200-word captionshigher is better
A note for the careful reader
Each length group comes from a different caption collection. SigLIP‑2 reads at most 64 text tokens, so it only sees the opening of these long captions; the paper does not measure how much of its lower score this explains. Random ranking would score about 0.02%.
Naming from a short label
Category names become phrases like ‘a photo of a dog’. The model picks the best fit for each picture, with no labelled training. The score is the share named correctly. A trained add-on reaches 81.1% on the same features.
Zero-shot top-1 accuracy, 1,000 categories, class promptshigher is better
A note for the careful reader
Random choice would score 0.1%.
In shortPhi‑0 finds the right kind of picture more easily than the exact one. It trails on short captions, narrows the gap within the top ten, and beats SigLIP‑2, though not CLIP, on very detailed descriptions.
Agreeing with other models
The Platonic Representation Hypothesis suggests that AI models trained on different data and different senses are all converging on one shared representation of the world. As models improve, they increasingly agree on which things are similar. We test this on 1,024 image–text pairs from Wikipedia. The two frozen encoders that Phi‑0 starts from agree on only 6.3% of neighbours. Trained only to predict one from the other, Phi‑0’s image and text sides agree on 23.5%. This is more than SigLIP‑2 at 19.0% and CLIP at 20.5%. Agreement stays modest for every model, as in the original study. The hypothesis describes a direction, not an endpoint.
Where it falls short
- It predicts the centre. Phi‑0 is trained to land on the centre of everything a caption could describe, its average or centroid. That keeps the concept, but it can lose fine-grained detail. It may fail to respond to a jersey number or the exact breed of a dog. The fix is to steer the model’s attention with the question being asked, so that the parts of a picture the question is about are kept in full detail. That work is in progress.
- Pointing at things needs an add-on. Without a grounding decoder, used only in training, Phi‑0 trails SigLIP‑2 at drawing a box around the thing a phrase describes. The decoder also needs data labelled with boxes.
- It cannot read text in pictures. Its image encoder learned from video, and its captions describe scenes rather than transcribe writing. Words inside a picture are lost.
- Small, and only two senses. Phi‑0 uses a base-size image encoder, about 73M trainable parameters and 160M pairs. It sees images and text only. The picture that motivates it includes sound.
What comes next
This work builds the space. Reasoning inside it comes next, and a reasoner that works directly in this space is under way. A larger model will follow. The idea behind this work also reaches beyond sight and language, to senses such as sound. A picture and its description now arrive close together; the next task is to think there.