Research·Phi0

One space for what is seen and what is said

Today’s leading image–text models keep pictures and words in separate places, learning only to judge whether they match. Phi‑0 learns by predicting one from the other, drawing a scene and its description together into one shared space.

Rudraksh Arora, Raunak ThakurNovember 20268 minute read
Two routes to one pointA picture and a description run as two bundles of fine lines into one copper point, the concept. seendescribedthe concept

From the paperLearning a Modality-Free World Space with Bidirectional Prediction

In four numbers
0.808
How close matched pairs sit

On unseen pairs, an image and its caption score 0.808 for closeness (1 means the same direction), against 0.165 for Google’s SigLIP‑2 and 0.323 for OpenAI’s CLIP.

6–8×
Smaller image–text gap

On the same unseen pairs, the distance between the centre of the images and the centre of the captions is 6–8 times smaller than in SigLIP‑2 and CLIP.

73M
Trainable parameters

About 73M parameters, trained on 160M image–caption pairs in 192 H100 GPU-hours. The two encoders are reused unchanged; their own pretraining is not counted.

23.5%
Image–text agreement

On 1,024 Wikipedia pairs, Phi‑0’s image and text sides share 23.5% of nearest neighbours, up from 6.3% for its starting encoders; SigLIP‑2 reaches 19.0% and CLIP 20.5%.

IThe idea

One concept, reached by different senses

A picture of a cat and the sound of its meow reach us through different senses. Yet both mean the same animal. Some neurons in the brain respond to a concept like a cat, rather than to the exact image shown, and some respond to both sights and sounds. We take this as an inspiration rather than a claim about the brain. It suggests a space where different inputs lead to one concept, whichever sense carried them.

Such a space must also let things go. An apple is a shape, a colour and a form. It is not the exact pixels of one photograph, the lighting of one room or the wording of one description. We call that detail noise. It belongs to one view of a thing, rather than to the thing itself.

An AI model turns every input into a long list of numbers. This list acts as coordinates, placing the input as a point in a space of many dimensions (an embedding). We want this space to have two properties. It must be modality-free: a scene lands at the same point whether it arrives as a picture or as text, the two modalities we use here. It must also be noise-free, keeping only what different views of a scene share.

IIToday

Two regions and a gap

The best-known models connecting pictures and words are CLIP, released by OpenAI in 2021, and Google’s SigLIP and its 2025 successor SigLIP‑2. They act as the eyes for many AI systems that look at pictures and talk about them. We compare Phi‑0 with these two models throughout this work. Both learn from vast collections of web images paired with captions. Given a batch of pairs, they learn to score true matches higher than mismatched ones (contrastive learning).

The space they produce is split. Images gather in one region and captions in another. A distance remains between the two (the modality gap). Even an image and its own caption stay far apart. Nothing in their training prevents this. A true pair can rank first while its two points sit in different regions. Such a model scores pairs. It does not predict one side from the other.

Generative models handle both pictures and words in one network by rebuilding every pixel and every word. Yet this training rewards keeping exactly the detail that a noise-free space should drop.

SigLIP-2 learns by scoring pairs A picture and its caption pass through two encoders, trained together, and land in two separate regions of the space, pictures above and captions below, with a gap between them. Training only scores which caption goes with which picture: a grid of pair scores, high on the matched diagonal and low elsewhere. Even a matched pair stays far apart. SigLIP-2learns by scoring pairs a picture its caption “two boats on a still lake” WHAT TRAINING ASKS image encoder trained text encoder trained a matched pair, still far apart the gap pictures captions Does this picture go with this caption, more than with the others? Matched pairs high, the rest low.

SigLIP‑2 trains its image and text encoders to score every picture–caption pair in a batch: true pairs high, mismatched pairs low. Nothing asks an image and its caption to meet, so images and captions settle in separate regions.

Phi-0 learns by predicting A picture and its caption pass through two frozen encoders and then one shared trunk, each becoming a point in a single region of the space, where pictures and captions mix. Training asks the model to predict the caption's point from the picture's point and the picture's point from the caption's, so a matched pair ends up close together. Phi-0learns by predicting a picture its caption “two boats on a still lake” WHAT TRAINING ASKS image encoder frozen text encoder frozen one shared trunk a matched pair, close together pictures and captions one region, a small gap From the picture’s point, predict the caption’s point; from the caption’s, the picture’s. No mismatched pairs are used.

Phi‑0 uses no mismatched pairs. From the image it predicts where the caption lies, and from the caption where the image lies. Each guess is scored by its distance from the truth. The two points are drawn together while SIGReg keeps the cloud spread out.

Fig. 1Two ways to learn from pictures and their captions. Copper is a picture, ink is a caption.
IIIThe method

Predicting one side from the other

Phi‑0 learns by guessing the caption from the picture, and the picture from the caption. It predicts the meaning of one view from another, rather than recreating the exact input (a joint-embedding predictive architecture, or Φ-JEPA).

The model starts with two existing components, or encoders, which we leave unchanged (frozen). For pictures, it uses V-JEPA 2.1, a vision model from Meta’s AI research lab that learned to see mostly by watching video. For text, it uses ModernBERT, an open language model released in 2024 that reads text and turns it into numbers, a modern update of the widely used BERT. Above them sits a small shared trunk. The picture and the caption pass through it separately, but through mostly the same machinery, which turns each into a single point in the space.

Two small networks, the predictors, then guess across. One predicts the caption’s point from the picture’s point. The other predicts the reverse. Each guess is scored by how far it lands from the true point (an L2 loss). Because both points are learned, each is shaped to be predictable from the other. Detail carried by only one side, like lighting or exact phrasing, cannot be guessed across. The model is pushed to drop it. What survives is only what the picture and the caption share.

Prediction alone has a shortcut. If the model sent every input to the exact same point, every guess would be perfect and the space would be useless (collapse). A rule called SIGReg stops this cheating. It keeps the cloud of points evenly spread in every direction and penalises it for bunching up. Technically, it pulls each batch towards an isotropic Gaussian.

Each picture is also seen as eight crops, including close-ups. Each caption is seen as eight passages taken from up to three descriptions of the picture. Crops are never flipped or recoloured, because captions name sides and colours. The picture’s final point is the average of its eight views, and so is the caption’s.

The model is small. It was trained on 160M pictures from the web, each with three detailed captions. Training took 192 GPU-hours on one machine with eight NVIDIA H100 GPUs. The parts that learn hold about 73M parameters, the numbers adjusted in training.

How Phi-0 is trained A picture (384 pixels square) and its caption are read by two frozen, pretrained encoders, V-JEPA 2.1 for the picture and ModernBERT for the caption. One small shared trunk, mostly the same weights for both streams, which never attend to each other, turns each into a point in one space: the picture's point Zv and the caption's point Zt. Two predictors cross over: one predicts the caption's point from the picture's, the other the picture's point from the caption's, and training pulls each prediction onto its target (an L2 loss). SIGReg keeps the points spread out so the model cannot cheat by putting everything in one place. Only the trunk and the predictors are trained, about 73 million parameters. Phi-0how it is trained a picture 384 × 384 px its caption “two boats on a still lake” image encoder V-JEPA 2.1 text encoder ModernBERT both frozen, not trained shared trunk mostly shared weights for both streams, which never look at each other one space the picture’s pointZv the caption’s pointZt predictor caption to picture a small network predictor picture to caption a small network each point is the other’s target frozen, pretrained: not trained trained: shared trunk and both predictors, about 73M parameters a prediction and its target: training pulls one onto the other SIGReg keeps the points spread out, so the model cannot cheat by putting everything in one place
How Phi-0 is trained A picture (384 pixels square) and its caption are read by two frozen, pretrained encoders, V-JEPA 2.1 for the picture and ModernBERT for the caption. One small shared trunk, mostly the same weights for both streams, which never attend to each other, turns each into a point in one space: the picture's point Zv and the caption's point Zt. Two predictors cross over: one predicts the caption's point from the picture's, the other the picture's point from the caption's, and training pulls each prediction onto its target (an L2 loss). SIGReg keeps the points spread out so the model cannot cheat by putting everything in one place. Only the trunk and the predictors are trained, about 73 million parameters. Phi-0how it is trained a picture 384 × 384 px its caption “two boats on a still lake” V-JEPA 2.1 ModernBERT two encoders, frozen shared trunk mostly shared weights for both streams; they never mix the picture’s pointZv the caption’s pointZt one space each prediction is pulled onto its target predictor caption to picture a small network predictor picture to caption a small network SIGReg keeps the points spread out, so the model cannot cheat by putting everything in one place trained: trunk + predictors, about 73M parameters
Fig. 2Two frozen encoders read the picture and the caption. A small shared trunk turns each into one point. Two predictors each guess the other side’s point, scored by distance. SIGReg keeps the points spread out, so the model cannot cheat by mapping everything to one place.
IVOne space

One space, not two

The clearest result is the shape of the space. We take 5,000 everyday photographs from a standard test collection, each with a one-line caption written by a person. Phi‑0 never saw them in training, so the test is out of distribution: on data unlike its training data. Drawn as points, SigLIP‑2’s images and captions form two separate caps. Phi‑0’s mix over one sphere.

SigLIP‑2 two regions
Points for SigLIP-2: the pictures form one thin arc and the captions a separate cap, far apart.
Phi‑0 one space
Points for Phi-0: pictures and captions are mixed over one sphere.
a pictureits caption
Fig. 3Five thousand everyday photographs and their captions, which Phi‑0 never saw in training, reduced to their three main directions and drawn on a sphere. SigLIP‑2 places images and captions on two separate caps. Phi‑0 mixes them over one sphere.

The drawing keeps only three of the space’s 768 directions. The table below measures the same pairs in the full space, four different ways. It scales from single pairs up to the layout of the whole cloud.

Table A

One space, or two?

We take 5,000 everyday photographs and their one-line captions. Phi‑0 never saw these in training. We measure where each model places a picture relative to its own caption.

marks the best of the three in each row

Closeness to own caption

Each picture and caption becomes a direction in the space. This compares a picture’s direction with its own caption’s. A score of 1 means the same direction. It shows if seeing a scene and reading about it end in the same place.

Mean matched image–caption cosine similarityhigher is better

A note for the careful reader

Averaged over the 5,000 pairs. A low value does not stop a model from matching, as SigLIP‑2 and CLIP rank by relative similarity. Phi‑0’s two points are never trained to be equal, only to be predictable from each other.

Phi‑00.808
SigLIP‑20.165
CLIP0.323

Gap between pictures and captions

We find the centre point for all pictures and the centre for all captions, then measure the distance between them. It runs from 0 (one shared centre) to 2. A large value means pictures and captions occupy separate regions.

Modality gap ‖μv − μt‖ on L2-normalised embeddings (Liang et al.)lower is better

A note for the careful reader

Points are scaled to length 1, so the maximum distance is 2. This is the plain distance between centres. The paper’s sphere figures print a different number, gap divided by spread, which should not be mixed with this measure.

picture centrecaption centre
Phi‑00.120
SigLIP‑21.004
CLIP0.779

Shared nearest neighbours

For each pair, we find the 10 most similar pictures and the 10 most similar captions. We count how many name the same pairs. 100% means all ten match. It asks whether looking and reading agree on what resembles what.

Mutual k-nearest-neighbour alignment, k = 10 (Huh et al.)higher is better

A note for the careful reader

Averaged over the 5,000 pairs. A score of 21.3% means about two of the ten neighbours are shared. Two random lists of ten would overlap by about 0.2%. The two sides are never compared directly.

Phi‑021.3%
SigLIP‑210.7%
CLIP10.0%

Same overall layout

This compares the pattern of similarities among all pictures with the pattern among all captions. 0% means unrelated patterns, while 100% means they are identical apart from rotation and scale. A high value means both sides are arranged in the same way.

Linear centred kernel alignment (CKA), image against caption embeddingshigher is better

A note for the careful reader

Computed on the image and caption points of the same 5,000 pairs.

Phi‑072.5%
SigLIP‑232.6%
CLIP28.7%

In shortPhi‑0 places a picture and its caption in nearly the same spot. SigLIP‑2 and CLIP keep pictures and captions in separate regions, even though they still rank the right pairs together.

Each measure tells the same story. Where the contrastive models keep two regions, Phi‑0 builds one. An image and its own caption point in nearly the same direction. The picture side and the caption side also agree on what resembles what. On held-out pairs similar to its training data (in distribution), they are closer still. On the table’s first measure, Phi‑0 reaches 0.885 against 0.133 for SigLIP‑2. The gap is small rather than gone.

VSight kept

What the encoders knew is kept

A shared space would be of little use if pictures lost their detail on the way in. To check, we freeze each model and train small probes on its picture features. These are simple add-ons that learn to outline objects, judge depth, name the main object, or box the thing a phrase describes. Every model gets the same probes, code and data.

Table B

Does it still see?

Each model is frozen. A small add-on is trained on its picture features, using the same code and data for all three models.

marks the best of the three in each row

Outlining objects

Every pixel is labelled with what it shows, like a person or background. The predicted region is scored by its overlap with the true one. 100% is a perfect outline. It tests whether the features remember where things are.

mIoU, semantic segmentation, linear probe per patchhigher is better

A note for the careful reader

The score is averaged over the object classes. The frozen V-JEPA 2.1 image encoder that Phi‑0 is built on scores 73.9% on its own.

Phi‑076.3%
SigLIP‑269.7%
CLIP70.8%

Judging distance and depth

From one indoor photo, the features estimate each point’s distance from the camera. The score is the typical error in metres, so 0 is perfect. Much of Phi‑0’s lead comes from its frozen image encoder, which errs slightly less on its own.

RMSE (m), indoor depth estimation, linear probe per patchlower is better

A note for the careful reader

Large mistakes count for more. Phi‑0 is typically off by about 54 cm, SigLIP‑2 by 66 cm and CLIP by 72 cm. The shared trunk costs a little depth accuracy compared to the raw V-JEPA 2.1 encoder.

here a shorter bar is better
Phi‑00.538m
SigLIP‑20.657m
CLIP0.719m

Naming the main object

An add-on reads the frozen features and picks one of 1,000 categories, like a dog breed. The score is the share of correct first guesses. It tests what visual details the features hold, not how well they match a phrase.

Top-1 accuracy over 1,000 categories, attention-pooling (MAP) probehigher is better

A note for the careful reader

Compare this to the last row of Table C: 81.1% here against 29.2% there for Phi‑0. The visual information is present in the features; the weakness lies in matching a short label.

Phi‑081.1%
SigLIP‑284.6%
CLIP79.6%

Finding described objects

Given a photo and a phrase, the model must draw a box around that specific thing. A box is correct if it overlaps the true box by at least half. The score is the share of correct boxes.

[email protected] (box IoU ≥ 0.5), referring expressions, frozen-feature probehigher is better

A note for the careful reader

The phrase is read by the same frozen text encoder for every model. With an optional grounding decoder trained on extra box data, Phi‑0 reaches 76.4%. The frozen V-JEPA 2.1 encoder alone scores 72.9%.

Phi‑067.0%
SigLIP‑275.2%
CLIP56.7%

In shortPhi‑0 keeps most of what its image encoder could see. It leads on outlining and depth, sits in the middle on naming objects, and trails SigLIP‑2 at finding what a phrase describes.

Much of this skill is inherited. On its own, the frozen V-JEPA 2.1 encoder judges depth slightly better than Phi‑0, at 0.495 against 0.538 metres. The trunk keeps most of that ability. Because both encoders stay frozen, Φ-JEPA is also a way to distil two encoders into one space. It largely keeps what each could do alone, and adds what neither had: a place where an image and its caption meet.

The text side shows the same trade. ModernBERT’s ability to find relevant passages is nearly unchanged after the trunk. What it loses is mostly sensitivity to the exact form of a sentence, such as whether it is grammatical, scoring 5.3 points lower on a standard grammar test. A space built around what a caption describes has little reason to keep that detail.

VIRetrieval

It finds the scene, not the snapshot

In retrieval, a model is given a caption and must find its picture among many, or the reverse. The usual score is how often the true match comes first. On the standard benchmarks, which use short captions, Phi‑0 is well behind.

The reason lies in how it learns. From a caption, Phi‑0 predicts where the picture should lie. No mismatched pairs push that guess away from look-alikes. When a short caption fits many photographs, the best guess is their centre: the concept, not one photograph. The right picture is usually among the first few rather than first. Two examples from the paper show this from both ends.

A short caption · 7 words
“A man cooking food on the stove.”
Where the caption’s own photograph is ranked, among 1,000
Phi‑06th
SigLIP‑23rd

Phi‑0 misses first place, yet all twenty of its first pictures show the scene the caption describes.

A detailed caption · 151 words
“In the center of the image, a young girl is the main subject. She’s dressed in a vibrant blue jacket … holding up a large slice of pizza …”
Where the caption’s own photograph is ranked, among 5,000
Phi‑01st
SigLIP‑2245thbeyond the first twenty

SigLIP‑2 reads at most 64 text tokens, so it sees only the opening of this description.

the caption’s own photographanother photograph matching the caption
Fig. 4Two single examples from the paper, chosen to show the pattern; the table gives averages. For a seven-word caption, Phi‑0 ranks the true photograph 6th, though its first twenty pictures all show the scene. For a 151-word description, Phi‑0 ranks it 1st; SigLIP‑2, 245th.

Caption length matters. The usual test captions are short, about ten to twelve words on average. Phi‑0 was trained on captions of about 49 words. A short caption fits many photographs; a long one fits few. From the shortest captions to those of 31–60 words, Phi‑0’s score nearly triples. On the longest captions, it overtakes SigLIP‑2, which reads only the first 64 tokens, or word pieces, of a caption. CLIP does better still there.

How often the right picture comes first, by caption lengthPhi-0 rises from 18.9% on the shortest captions to 51.7% at 31 to 60 words and reaches 39.2% on the longest, above SigLIP-2 at 26.4% and below CLIP at 46.6%.020406080≤1516–3031–6061–100101–150151–200words in the captionright picture ranked first, %CLIPPhi-0SigLIP-2
Fig. 5How often the caption’s own picture is ranked first, by caption length. Phi‑0 nearly triples from the shortest captions to 31–60 words and passes SigLIP‑2 on the longest; CLIP, the dashed line, stays ahead there. Each length group comes from a different caption collection.

Short prompts show the same pattern in zero-shot classification. Here, a phrase such as “a photo of a dog” stands in for each of 1,000 everyday categories, from breeds of dog to kinds of tool. Phi‑0 does poorly. Yet a probe trained on the same picture features names objects well. The visual information is there. The weakness lies in matching a short phrase to one class.

Table C

Matching words to pictures

Given a caption, each model must find the right picture among a thousand or more. Given a picture, it must pick its name from 1,000 labels. Nothing is retrained.

marks the best of the three in each row

Short caption, exact match first

The model ranks 1,000 pictures for 1,000 short captions. The score is how often the exact picture comes first. Phi‑0 predicts the typical picture for a description, so a similar photo often comes first instead of the exact one.

Text→image Recall@1, 1,000-image test set, zero-shothigher is better

A note for the careful reader

Captions average about 12 words. Random ranking would score 0.1%. For Phi‑0, the caption is turned into a predicted picture point, which is then ranked against the pictures.

Phi‑037.1%
SigLIP‑281.6%
CLIP62.1%

Short caption, in top ten

The same test, but a hit counts if the exact picture is anywhere in the first ten. From first place to top ten, Phi‑0’s score doubles. This shows that Phi‑0’s misses are often near misses.

Text→image Recall@10, 1,000-image test set, zero-shothigher is better

A note for the careful reader

Random ranking would score 1%. Across three standard test collections, Phi‑0’s score rises 2.0 to 2.6 times from first place to top ten, compared to 1.2 to 1.6 times for SigLIP‑2.

Phi‑075.3%
SigLIP‑297.8%
CLIP91.9%

Long description, exact match first

The same first-place test, but with very detailed descriptions of 151 to 200 words. A long description fits far fewer photos than a short one, which suits a space built around meaning. Phi‑0 beats SigLIP‑2 here but stays behind CLIP.

Text→image Recall@1, 151–200-word captionshigher is better

A note for the careful reader

Each length group comes from a different caption collection. SigLIP‑2 reads at most 64 text tokens, so it only sees the opening of these long captions; the paper does not measure how much of its lower score this explains. Random ranking would score about 0.02%.

Phi‑039.2%
SigLIP‑226.4%
CLIP46.6%

Naming from a short label

Category names become phrases like ‘a photo of a dog’. The model picks the best fit for each picture, with no labelled training. The score is the share named correctly. A trained add-on reaches 81.1% on the same features.

Zero-shot top-1 accuracy, 1,000 categories, class promptshigher is better

A note for the careful reader

Random choice would score 0.1%.

Phi‑029.2%
SigLIP‑279.9%
CLIP68.3%

In shortPhi‑0 finds the right kind of picture more easily than the exact one. It trails on short captions, narrows the gap within the top ten, and beats SigLIP‑2, though not CLIP, on very detailed descriptions.

VIIAgreement

Agreeing with other models

The Platonic Representation Hypothesis suggests that AI models trained on different data and different senses are all converging on one shared representation of the world. As models improve, they increasingly agree on which things are similar. We test this on 1,024 image–text pairs from Wikipedia. The two frozen encoders that Phi‑0 starts from agree on only 6.3% of neighbours. Trained only to predict one from the other, Phi‑0’s image and text sides agree on 23.5%. This is more than SigLIP‑2 at 19.0% and CLIP at 20.5%. Agreement stays modest for every model, as in the original study. The hypothesis describes a direction, not an endpoint.

VIIILimits

Where it falls short

  1. It predicts the centre. Phi‑0 is trained to land on the centre of everything a caption could describe, its average or centroid. That keeps the concept, but it can lose fine-grained detail. It may fail to respond to a jersey number or the exact breed of a dog. The fix is to steer the model’s attention with the question being asked, so that the parts of a picture the question is about are kept in full detail. That work is in progress.
  2. Pointing at things needs an add-on. Without a grounding decoder, used only in training, Phi‑0 trails SigLIP‑2 at drawing a box around the thing a phrase describes. The decoder also needs data labelled with boxes.
  3. It cannot read text in pictures. Its image encoder learned from video, and its captions describe scenes rather than transcribe writing. Words inside a picture are lost.
  4. Small, and only two senses. Phi‑0 uses a base-size image encoder, about 73M trainable parameters and 160M pairs. It sees images and text only. The picture that motivates it includes sound.
IXNext

What comes next

This work builds the space. Reasoning inside it comes next, and a reasoner that works directly in this space is under way. A larger model will follow. The idea behind this work also reaches beyond sight and language, to senses such as sound. A picture and its description now arrive close together; the next task is to think there.

sightsoundtouchlanguageone spacereasoning, next
The paper

Learning a Modality-Free World Space with Bidirectional Prediction
Rudraksh Arora and Raunak Thakur, Phi9, 2026.

Step inside the Phi0 space