Semester 4 · Computer vision
Vision-Language, 3D Vision and World Models
https://epochschool.com/subjects/vision-language-3d
Computer vision now reads and writes language, reconstructs the 3D world and predicts what happens next.
You derive and build the ideas behind these systems: contrastive image-text models and zero-shot transfer, captioning, visual question answering and vision-language models with connectors and staged training; 3D representations, structure from motion and bundle adjustment, point-cloud networks, neural radiance fields and Gaussian splatting; and action-conditioned video prediction and world models you can plan with. Every lab trains a small but real system from scratch on one CPU core with data generated in the notebook, and every result is checked against a baseline that would expose a shortcut.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Analyse a multimodal problem in terms of representation, alignment and fusion, implement and compare fusion operators (concatenation, FiLM, low-rank bilinear, cross-attention), and use unimodal, zeroed and shuffled-input baselines to show which modality a model really uses.
- LO2Derive, implement and train contrastive image-text models (symmetric InfoNCE with a learned temperature, and the sigmoid loss), and explain their behaviour in terms of temperature, batch size and false negatives.
- LO3Build and evaluate zero-shot classifiers and cross-modal retrieval systems (prompt ensembling, linear probes, Recall@K with partial relevance), and diagnose compositional failures such as bag-of-words behaviour with swap tests.
- LO4Build and evaluate image-conditioned language models for captioning, visual question answering and instruction following: connectors, visual prefix tokens, staged training with frozen components, answer-only losses, caption metrics, VQA accuracy and hallucination probes.
- LO5Represent, transform, convert and compare 3D data (point clouds, meshes, voxels, occupancy and signed-distance fields) using rotation parametrisations, similarity alignment and metrics such as Chamfer distance, F-score and IoU.
- LO6Recover camera poses and sparse structure from multiple views by triangulation, camera registration and sparse bundle adjustment, and explain gauge freedom, scale ambiguity and the expected residual.
- LO7Implement and analyse permutation-invariant point-cloud networks and neural implicit fields, including rotation handling and eikonal regularisation.
- LO8Derive and implement differentiable volume rendering, train radiance fields from posed images, and compare MLP fields, voxel grids and 3D Gaussian splatting on quality, speed and memory using the EWA projection of Gaussians.
- LO9Build and evaluate action-conditioned video prediction and latent world models against copy-last-frame and shuffled-action baselines, and plan with them by model-predictive control with the cross-entropy method.
Chapters
The chapter and lab list for this subject is being finalised and will be published here.
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Sequence Models & Transformers
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
Image Formation and Classical Computer Vision
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.