Yann LeCun, AI pioneer, is investing heavily in an alternative AI approach called Joint Embedding Predictive Architectures (JEPA), which contrasts with large language models (LLMs) by focusing on predicting embeddings rather than generating text or images. This method aims to overcome limitations of generative models, especially in video prediction, and build world models that can predict consequences of actions, potentially leading to more reliable and intelligent AI systems.
Yann LeCun, a legendary figure in artificial intelligence, has raised a billion dollars to pursue an alternative approach to AI that diverges significantly from the popular large language models (LLMs). Unlike LLMs, which are generative and rooted in language, LeCun's approach, known as Joint Embedding Predictive Architectures (JEPA), does not generate text, images, or videos. Instead, it focuses on learning powerful internal representations through a novel architecture.
JEPA is not a single model but a framework for training AI models differently from traditional methods. Typically, AI models are trained to predict an output Y given an input X. For example, LLMs predict the next word in a text sequence, and image classifiers predict labels for images.
JEPA, however, uses encoders to transform both inputs and outputs into embeddings—vectors or matrices of numbers representing the data. A predictor model is then trained to predict the embedding of Y given the embedding of X. This approach sidesteps the need to generate raw data directly.
LeCun argues that while LLMs excel at language manipulation, they are limited outside this domain. Language itself serves as a substrate for reasoning in LLMs, but this does not translate well to other modalities like vision or robotics.
Generative models, especially for video, face significant challenges. Predicting the next video frame at the pixel level leads to blurry and unrealistic outputs due to the immense complexity and uncertainty in video data. For instance, the number of possible next frames in a full HD video is astronomically large, making discrete output prediction infeasible.
JEPA avoids this by predicting embeddings rather than raw pixels, focusing on salient features rather than every detail. This allows the model to learn meaningful representations without the pitfalls of blurry generative outputs.
LeCun has been a pioneer in AI since the 1980s, notably developing convolutional neural networks (CNNs) that laid the groundwork for modern deep learning. Despite the rise of deep learning and LLMs, LeCun and others have been concerned about the heavy reliance on labeled data, which is scarce and expensive.
This concern led to exploring self-supervised learning, where models learn from data without explicit labels. LLMs like GPT series have demonstrated the power of self-supervised learning in language, but similar success in vision and video has been elusive.
Attempts to apply generative self-supervised learning to video involved predicting pixel values of future frames. However, these models produce blurry frames due to averaging over multiple possible outcomes, a problem known as uncertainty handling.
Unlike language, where the vocabulary is fixed and discrete, video pixels are continuous and high-dimensional, making generative prediction much harder.
Joint embedding architectures, such as Siamese networks developed by LeCun in the 1990s, learn to produce similar embeddings for different views or corruptions of the same input. This approach has been effective in learning useful internal representations without generating data.
However, these architectures suffer from representation collapse, where the model outputs the same embedding for any input, failing to learn meaningful distinctions.
LeCun's team addressed this with contrastive learning, training the model to produce similar embeddings for positive pairs and dissimilar embeddings for negative pairs. While effective, contrastive methods require large amounts of computation and many negative samples.
Inspired by the work of neuroscientist Horace Barlow, who proposed that neurons reduce redundant information, LeCun and collaborators developed a method called Barlo Twins. This technique reduces redundancy between neuron outputs in the embedding layer by encouraging decorrelation.
The method involves computing a cross-correlation matrix between neuron outputs from two encoders processing different views of the same input. The goal is to make this matrix close to the identity matrix, meaning corresponding neurons are highly correlated while different neurons are uncorrelated.
Barlo Twins effectively avoids representation collapse and learns powerful image representations.
Using Barlo Twins, LeCun's team achieved impressive results on the ImageNet classification benchmark. A frozen encoder trained with Barlo Twins, combined with a simple linear classifier, achieved 73.2% accuracy, surpassing the original AlexNet's 59.3% accuracy.
Further developments, such as VicReg and DINO, have pushed self-supervised learning closer to fully supervised performance. The DINO V3 model, released in 2023, achieved near state-of-the-art accuracy of 88.4% on ImageNet without any labeled data.
These advances demonstrate that joint embedding architectures can learn high-quality representations for vision tasks, rivaling supervised methods.
In 2022, LeCun published a comprehensive position paper outlining a path towards autonomous machine intelligence. He emphasized that current AI systems are far from human-level learning efficiency, citing how teenagers learn to drive in hours, while AI requires millions of hours of data.
LeCun believes the missing piece is world models—models that predict the consequences of actions and enable reasoning, planning, and imagination.
JEPA provides a foundation for building such world models by learning embeddings of observations and predicting future embeddings conditioned on actions.
JEPA models have been applied to robotics, where the model predicts the embedding of the next video frame of a robot arm given the current frame and control signals. This learned world model enables planning by searching for action sequences that lead to desired goal states.
This approach revives classical optimal control ideas but with learned models and abstract state representations, offering a new twist on old concepts.
LeCun predicts that JEPA and world model-based approaches will eventually replace LLMs, as LLMs are limited to language manipulation and lack the ability to predict consequences of actions.
He argues that reliable agentic systems must have world models to plan and ensure safety, capabilities that current LLMs do not possess.
Yann LeCun's billion-dollar investment in JEPA represents a bold bet on an alternative AI paradigm focused on learning internal representations and world models rather than generative outputs. This approach addresses fundamental challenges in video prediction and robotics and aims to build more intelligent, reliable, and autonomous AI systems.
The ongoing research and development in joint embedding architectures, self-supervised learning, and world models promise exciting advancements beyond the current dominance of large language models.
If you want to explore these topics further, consider reading Welch Labs' illustrated guide to AI, which covers neural networks, deep learning, neural scaling laws, diffusion models, and more, providing a comprehensive foundation for understanding modern AI developments.
