research hub

The algorithmic process of mapping 3D human motion (captured via mocap or video) onto a robot's kinematic structure. This involves solving inverse kinematics (IK) optimization problems to match end-effector poses and joint angles while respecting the robot's physical limits.

Essential for learning from human demonstrations ("Human2Robot"). In HumanPlus and DexCap, retargeting allows humanoids to learn complex skills directly from human data, bypassing the need for slow teleoperation.

A controlled experiment where a single component is removed or changed (e.g., remove touch, remove wrist camera, remove proprioception) while keeping everything else fixed.

Ablations are used to attribute performance changes to specific inputs or architectural choices. VISK reports modality ablations (vision-only vs visuotactile, wrist camera on/off, proprioception on/off) to show which signals matter for contact-rich manipulation.

A sequence modeling technique where the policy predicts a fixed-length series of future actions (a "chunk") in a single inference step (a_t, a_{t+1}, ..., a_{t+k}), rather than just the immediate next action.

Used in Transformer-based policies (like ACT and ARP) to combat latency and ensure temporal smoothness in robot motion. It prevents the "jitter" often seen in single-step predictions.

The mathematical set of all valid control commands a robot can execute. Can be Continuous (real-valued numbers) or Discrete (finite set of choices).

Humanoids typically use high-dimensional continuous action spaces (e.g., 33 joint angles). EgoVLA and ARP explore discretizing this space to apply language-model architectures to control.

Using the presence of a human hand interacting with an object in a video as a "free" supervision label. The area where the hand touches the object is labeled as "active" or "affordance-rich."

HRP uses this to turn passive Youtube videos into a supervised learning dataset for detecting interaction hotspots without human annotation.

A concept coined by James J. Gibson (1966) referring to the actionable properties of an environment relative to an agent. An affordance is not a property of an object (e.g., "a handle"), but a relationship (e.g., "this handle fits my gripper"). In robotics, this defines what an agent can do with an object (e.g., graspability, approach vectors).

The core thesis of HRP is that robots cannot learn manipulation solely from observing objects; they must learn these affordances. This bridges the gap between seeing a "cup" and knowing how to pick it up. In HRP, the model is pre-trained to predict these "human affordances" to guide the robot's attention.

A replaceable magnetic tactile skin designed for robot fingertips. It measures 3-axis magnetic field signals from multiple sensors under a deformable skin to infer contact forces and shear, while keeping the sensing electronics reusable across skin replacements.

VISK uses AnySkin as a low-dimensional tactile input, which makes end-to-end policy learning easier than high-dimensional optical tactile images. AnySkin is designed to be swapped when it tears, without requiring recalibration or retraining.

An object composed of multiple rigid parts connected by joints, such as a refrigerator (door + body) or a laptop (screen + base).

Kinematic-aware prompting specifically targets these objects, as standard planners often fail to account for the specific motion constraints (arcs, slides) required to manipulate them.

A control policy that generates action sequences token-by-token, where each predicted action is conditioned on the previous actions in the sequence.

Proposed as a computationally efficient alternative to Diffusion Policies. It allows robots to model complex, multi-modal behavior without the heavy inference cost of iterative denoising.

The main representation-learning network that produces features for downstream prediction. In robotics papers, backbone can refer to the visual encoder backbone (e.g., ResNet) or the policy backbone (e.g., a transformer trunk that fuses modalities).

In VISK, camera images are encoded by ResNet-18, while the policy uses a transformer decoder trunk as the backbone that fuses camera, tactile, and proprioception tokens before action prediction.

A transformer-based, multi-task behavior cloning policy architecture designed to efficiently learn from multiple camera views and tasks.

BAKU is an explicit evaluation of design choices for multi-task, multi-modal behavior cloning. It tests which components and configurations actually impact performance, then proposes a strong baseline architecture that was state of the art in mid-2024 (submitted Jun 11, 2024; revised Jul 16, 2024). VISK builds on BAKU by adding a tactile token stream (AnySkin) alongside visual tokens.

The simplest form of Imitation Learning used in robotics. Instead of using complex reward functions (like Reinforcement Learning), BC treats robot control as a standard supervised learning problem. It uses a dataset of expert demonstrations (human pilots) and trains a neural network to simply mimic the expert's actions for each video frame. It learns to directly map what it sees to what it does, assuming the expert is always right. Its weakness is an inability to adapt: if the environment changes or the robot's grip slips, it will simply follow its memorized path, even if the outcome is unacceptable.

Historically rooted in ALVINN (1989), where a neural network learned to steer a vehicle. In modern manipulation (MCR, EgoVLA), it serves as the standard baseline for imitation learning. However, it suffers from "covariate shift"—if the robot drifts from the training distribution, it cannot recover—necessitating more robust methods like Diffusion Policies or Implicit BC.

A transformer architecture that ensures predictions for a given time step depend only on past observations, strictly preventing information leakage from future tokens.

Used in ARP and DexCap to process sequences of observations (images, proprioception) and predict stable action trajectories.

A prompting technique where a Large Language Model (LLM) is encouraged to generate intermediate reasoning steps before outputting a final answer.

Kinematic-Aware Prompting uses CoT to force the LLM to "think" about the robot's physical constraints (e.g., "The door hinge is on the left, so I must pull from the right") before generating the plan.

A three-stage framework for learning from "in-the-wild" human videos: (1) Contrast to learn task distinctions, (2) Imitate to learn motion priors via retargeting, and (3) Adapt to fine-tune with RL to correct physical mismatches.

Addresses the "correspondence problem" where human video lacks robot action labels. It extracts useful signals from noisy internet video to pre-train robot policies.

Training a single model simultaneously on multiple distinct datasets or tasks (e.g., human video data AND robot teleoperation data).

HAT (Human Action Transformer) relies on co-training to align human and robot representations in a shared latent space, allowing the massive scale of human data to improve the robot's performance.

The specific points on an object's surface where interaction occurs.

HRP and ViSk focus on predicting these locations. In HRP, it's visually predicted; in ViSk, it's sensed via tactile skin to confirm successful grasping or manipulation.

Tasks where the primary difficulty involves complex physical interaction, friction, and force modulation (e.g., inserting a tight peg, sliding a card) rather than just free-space motion.

These tasks are notoriously hard for vision-only policies due to occlusion. ViSk and Sparsh-skin demonstrate that tactile sensors are essential here.

A self-supervised learning technique that learns representations by pulling "positive" pairs (e.g., two views of the same scene) close together in embedding space and pushing "negative" pairs apart.

The core engine behind R3M (Time-Contrastive) and MCR. It allows robots to learn structure from video without human labels.

A classic deep learning architecture (like the ResNet-50 used in MCR) that processes images using a sliding window filter. It exhibits a strong inductive bias toward locality, meaning it naturally assumes that pixels adjacent to one another are related. This makes it highly data-efficient at learning edges, textures, and boundaries (crucial for grasping), but it struggles to capture long-range relationships (e.g., knowing that the switch on the wall controls the light on the ceiling) without being very deep.

Popularized by LeCun (LeNet-5, 1998) and later AlexNet (2012), CNNs became the backbone of computer vision for a decade. In the MCR paper, a ResNet-50 (a deep CNN) is chosen over a Transformer to avoid overfitting on the smaller DROID dataset, leveraging the CNN's strong inductive bias for locality.

A control strategy designed to be transferred from one domain to another target domain with different characteristics.

While typically used for Sim2Real transfer, it also covers embodiment transfer (Human2Humanoid). HumanPlus is a prime example of a cross-domain policy architecture.

A neural mechanism that dynamically weights the importance of different sensory inputs (e.g., Vision vs. Touch) based on the context.

Used in ViSk to ignore vision when the camera is blocked and rely heavily on tactile data, or vice versa when the hand is moving in free space.

Techniques used to artificially increase the diversity of training data by applying random transformations (cropping, rotating, color jitter, noise injection) to the inputs.

Critical in R3M and HumanPlus to prevent overfitting and ensure the visual encoder is robust to lighting changes or slight camera shifts in the real world.

A portable, wearable hand motion capture system that combines SLAM-based tracking and electromagnetic field sensors to collect precise 3D hand motion data.

DexCap solves the portability and occlusion issues of fixed-camera mocap systems, enabling "wild" data collection for dexterous manipulation policies.

A class of robot control policies that generate action sequences by iteratively denoising random noise, conditioned on observations. They model the distribution of expert trajectories rather than just the mean action.

Becoming a standard for dexterous manipulation because they can represent multi-modal distributions (e.g., different ways to grasp the same object) and are highly stable.

A compact optical tactile sensor that uses an internal camera to observe deformation of an elastomer gel, producing high-dimensional tactile images.

DIGIT sensors are widely used for tactile perception and manipulation, but their image-based output often requires a heavier encoder or intermediate state estimation. VISK compares AnySkin against DIGIT and reports that low-dimensional magnetic sensing can be easier to learn from for contact-rich manipulation with limited demonstrations.

The process of converting continuous values (like joint angles 0.0 to 1.0) into discrete bins or "tokens" (e.g., integer 0 to 255).

ARP and EgoVLA discretize robot actions so they can be processed by Transformers, which are natively designed for discrete text tokens.

The process of training a smaller "student" model to reproduce the behavior or outputs of a larger/more complex "teacher" model.

Used in HumanPlus to compress a heavy history-based policy into a reactive policy, and in Sparsh (Self-distillation) to learn representations without labels.

The number of independent parameters that define the configuration of a mechanical system.

General-purpose humanoids discussed (e.g., Unitree H1) typically have 19-33 DoF. Managing high-DoF coordination is the primary challenge for whole-body control.

A strategy used to bridge the "Sim2Real gap" by intentionally injecting variability into the training environment (e.g., altering mass, friction, textures, or lighting).

Its primary assumption is that if a policy works across a wide distribution of simulated worlds, it will likely succeed in the real world. Critical for the HumanPlus walking policy.

A cross-modal contrastive learning objective used in MCR. It effectively forces the visual model to feel what the robot is doing, not just see it. It is a relevance filter. It does so by comparing the vision and robot state between two chunks of time, then forcing the model to disregard all of the irrelevant data between those two chunks (such as lighting changes, camera flickers, and background motion) and retain the important information such as the movement of the arm and the object being manipulated.

Derived from cross-modal contrastive learning principles (like CrossCLR). In MCR, it functions as a "relevance filter," effectively grounding the visual representation in physical reality. By enforcing high mutual information between the visual embedding and the proprioceptive state, it ensures the model attends only to objects that are physically interacting with the robot.

A discrepancy between the transition dynamics (physics) of the source domain (simulation) and the target domain (real world), often caused by unmodeled disturbances like wind, uneven floors, or sensor noise.

HumanPlus uses robust low-level policies to handle these disturbances. CIA uses an adaptation phase to correct for this mismatch.

A massive-scale dataset of egocentric (first-person) video captured by humans performing daily tasks (cooking, cleaning, working).

The foundational dataset for R3M, EgoVLA, and HRP. It provides the "human experience" data that robots attempt to learn from.

Visual data captured from the first-person perspective (camera mounted on the robot's head or human's smart glasses).

EgoVLA, EgoZero, and DexCap focus on this perspective to minimize the visual gap between human data (via glasses) and robot data (via head cams).

A Vision-Language-Action (VLA) model specifically trained on egocentric human videos to predict robot actions from video and language prompts.

Represents the convergence of foundation models and robotics, leveraging the massive scale of human egocentric video to bootstrap robotic manipulation.

A motion capture technology that uses magnetic fields to track the position and orientation of sensors, distinct from optical (camera-based) tracking.

DexCap uses EM sensors because they are not affected by "occlusion" (being blocked from view), allowing for precise hand tracking even when fingers are curled or inside objects.

A high-dimensional list of numbers (e.g., a vector of size 768) that acts as the mathematical distillation of an input. Whether the input is an image (Vision Encoder), text (LLM), or audio, the model converts it into this vector. If training is successful, inputs that are semantically or functionally similar result in vectors that are numerically close.

This is the "language" the robot's policy brain speaks. In HRP, the goal is to produce an embedding vector that contains dense information about where the hand should go. In LLMs, it captures the meaning of words. This vector is then fed into downstream components (like a control policy) to generate output.

The physical form of the robot, including the number of limbs, joint types, mass, and sensor suite.

HAT and HumanPlus explicitly design architectures to bridge the "Embodiment Gap," learning representations that can be retargeted across different physical forms.

Visual data captured from an external, fixed perspective relative to the agent (e.g., a tripod-mounted camera, surveillance feed, or an observer).

In contrast to egocentric data, exocentric views provide global context but suffer from the "correspondence problem" during deployment if the robot only has onboard cameras. Used heavily in HumanPlus shadowing and CIA.

High-quality examples of a task being performed correctly, either by a human or a programmed expert policy.

The fuel for Behavior Cloning. DexCap and HumanPlus exist primarily to make collecting these demonstrations easier and cheaper.

The process of taking a pre-trained model (which has learned general features) and training it further on a smaller, specific dataset to specialize in a particular task.

R3M and HRP encoders are often "frozen" or fine-tuned with RL. VLA models are fine-tuned on robot action data after being pre-trained on internet text/video.

A model that predicts the next state (s_{t+1}) given the current state (s_t) and action (a_t). Effectively, a "simulator" inside the neural network.

MCR learns this to ensure the visual representation understands the consequences of actions, not just the appearance of objects.

Large-scale models pre-trained on diverse internet data that can be fine-tuned for downstream robotic tasks.

R3M (vision foundation), EgoVLA (VLA foundation), and Sparsh-skin (tactile foundation) represent the shift from training scratch policies to adapting pre-trained "brains."

A pre-trained neural network layer (usually vision) whose weights are not updated during the training of the downstream policy.

R3M is designed to be used as a frozen encoder to prevent "catastrophic forgetting" of general visual knowledge while training on specific tasks.

A probabilistic model that assumes all data points are generated from a mixture of a finite number of Gaussian distributions (bell curves). It is used to cluster messy, irregular data into clean, mathematical probabilities.

Widely used in robotics for handling uncertainty, modeling sensor noise, and encoding motion primitives (Task-Parameterized GMMs). In HRP specifically, it is used to take the irregular cluster of pixels where a hand touches an object and standardize it into a fixed "contact descriptor" that the neural network can process.

A family of high-resolution optical tactile sensors that estimate contact geometry and force by imaging deformation of a gel surface with a camera.

GelSight-style sensors provide rich spatial detail, which is valuable for shape reconstruction and perception, but their high-dimensional tactile images often increase the complexity of policy learning relative to low-dimensional magnetic skins like AnySkin.

The ability of a robot to perform a task in new environments, with new objects, or under new lighting conditions that were not seen during training.

The ultimate goal of HRP, R3M, and EgoZero. Training on massive human datasets (Ego4D) is the primary strategy to achieve this.

A debugging and visualization technique. It effectively asks: Which pixels in this image contributed the most math to the final score? For CNNs such as MCR and R3M, this works by backpropagating the gradients of the final decision to the final image-processing layer. For ViTs such as MVP, VC-1, and HRP, the final image-processing layer contains no spatial information; the authors use the gradients of the normalization layer.

Introduced by Selvaraju et al. (2017) to demystify "black box" deep learning models. In robotics, it is strictly an evaluation tool, not a control mechanism. Researchers use it to audit models (like in the MCR paper) to verify if the policy is focusing on task-relevant features (gripper/object) or cheating by tracking background noise.

The fundamental optimization algorithm used to train neural networks. It iteratively adjusts the network's weights to minimize the error (loss) between predictions and targets.

Underlying all learning methods in these papers. RL methods (like PPO in HumanPlus) use policy gradient versions of this.

The problem of linking abstract concepts (words like "cup" or "fast") to concrete physical data (pixels of a cup, motor velocity).

EgoVLA and Instruction Tuning focus on grounding language instructions into specific motor actions.

A magnetic field sensor that outputs a signal proportional to magnetic flux density, based on the Hall effect. In tactile skins, these sensors can be used to measure changes in a magnetic field caused by skin deformation.

Magnetic tactile sensors often rely on multiple magnetic field sensors to capture both normal and shear components. AnySkin-style magnetic skins use multi-axis magnetic sensing so contact signals can be fused by the policy backbone without image processing.

The configuration of the hand, typically described by the position and rotation of the wrist (6-DoF) and the angles of all finger joints.

DexCap captures this at high fidelity. HRP infers this from video to understand how humans grasp objects.

A policy architecture co-trained on both human and humanoid data. It utilizes a unified state-action space where human actions are treated as a different "embodiment."

Mitigates data scarcity by treating human data not just as a visual prior, but as a direct training signal for the policy.

A framework for pre-training visual representations using internet-scale human videos to extract "affordances."

Focuses on learning where to interact from human data, providing a spatial prior that boosts downstream robotic task performance.

Training processes where human feedback (demonstrations, corrections, or preferences) guides the robot's learning online.

HumanPlus enables a "shadowing" mode where a human operator teleoperates the robot by moving their own body, generating high-quality training data.

A self-supervised training objective where the model is shown two sequential frames and must predict the specific motor command (action vector) that bridged the gap. This is the primary driver of manipulation centricity because the only way to mathematically solve this problem is to focus on the moving parts (gripper/object) and ignore the static background.

Popularized in reinforcement learning by Pathak's "Curiosity-Driven Exploration" (2017). In representation learning (MCR), it is used as a self-supervised objective to enforce causality. Since static backgrounds do not explain robot actions, solving this objective mathematically forces the encoder to attend to the agent and the manipulated object.

The mathematical calculation of the joint angles required to place a robot's end-effector at a specific position and orientation in space.

Used heavily in Action Retargeting to convert 3D coordinates of human hands (from mocap) into joint commands for the robot.

A method for using Large Language Models (LLMs) to plan manipulation tasks by explicitly including kinematic constraints and object articulation info in the prompt.

Bridges the gap between high-level reasoning and low-level physics, allowing LLMs to generate actionable plans for articulated objects (e.g., opening doors).

A training technique used in Vision-Language-Action (VLA) models to teach specific motor skills without overwriting the model's general semantic knowledge. It typically involves freezing the majority of a pre-trained backbone while only fine-tuning specific components (like Layer Normalization layers) or stopping gradient flow from the action head back to the vision backbone.

Coined by Physical Intelligence for their pi0 model to prevent "catastrophic forgetting" during co-training. HRP utilizes a similar principle by finding that fine-tuning only the LayerNorm parameters yielded better results than full model fine-tuning, effectively insulating the pre-trained visual capabilities from the new affordance tasks.

The strategic reuse of previously acquired information to solve new, related problems more efficiently.

Includes transferring modules/policies (pre-trained brains), representations (how a robot interprets data), or experiences. R3M transfers visual knowledge; HAT transfers motor knowledge.

Modeling the physics and transitions of the environment within a compressed "latent" representation rather than in raw pixel space.

MCR uses this to ensure that visual representations capture temporal and dynamic properties (how things move), not just static appearance.

A visual representation learning framework that captures both visual features and dynamics information (proprioception and actions).

Argues that for manipulation, the representation must encode "how things move" (dynamics), leading to better performance than purely semantic representations like CLIP.

The core metric of the MCR paper. It quantifies the quality of a vision encoder’s attention. It is computed as the intersection between the model's attention heatmap (from Grad-CAM) and the ground-truth object location (from SAM 2).

A novel metric introduced in the "Robots Pre-train Robots" (MCR) paper. It serves as a quantitative Key Performance Indicator (KPI) for attention quality. By measuring the Intersection over Union (IoU) between the model's Grad-CAM heatmap and Ground Truth masks (from SAM 2), it predicts downstream task success more reliably than loss curves.

A self-supervised learning method for pre-training Vision Transformers. The algorithm randomly masks a high percentage (e.g., 75%) of the image patches and forces the model to reconstruct the missing pixels from the visible ones. This forces the encoder to learn a deep, semantic understanding of shapes and structures rather than just surface-level textures.

MAE is considered a gold standard for initializing visual encoders in robotics because it is highly data-efficient. HRP utilizes an MAE-pre-trained ViT-B backbone, which is then further fine-tuned on the extracted affordance data.

A modular approach where sub-skills are handled by distinct policy architectures optimized for the dominant sensory modality (e.g., Vision-Language for planning, Tactile-RL for grasp).

Argues against monolithic "end-to-end" models, proposing "specialist" models for different stages of a task to achieve higher dexterity.

Decoupling a learned policy into robot-specific modules (e.g., joint control) and task-specific modules (e.g., high-level goals).

Allows a new robot to reuse task-specific modules by simply "stitching" a pre-trained task module to its hardware control module.

A control method that optimizes a finite time-horizon of future actions using a dynamic model, executing the first action and repeating.

Often used as a low-level stabilizer for humanoids or for trajectory optimization in Sim2Real pipelines before distillation into a neural network.

A simple, shallow feed-forward neural network consisting of dense layers of neurons. In this context, it is used in two distinct instances. Once, as the policy head, used in pre-training to learn action alignment; and again, as a fresh model, used in behavior cloning of tasks to control the robot's actions. It is deliberately kept simple (typically 3 layers) so that it cannot perform complex image processing on its own. It relies entirely on the input features being perfect. It maps the abstract concepts from the vision encoder to concrete coordinate values (x, y, z).

The foundational building block of deep learning (Rosenblatt, 1958). In modern architectures, MLPs are rarely used for perception (images) but are standard for "Policy Heads"—the final layers that map extracted features to motor commands. In MCR, a lightweight 3-layer MLP is used for behavior cloning to ensure performance comes from the visual encoder, not the policy's capacity.

The pattern of apparent motion of image objects between two consecutive video frames.

Used in EgoVLA and HRP to detect motion and interaction. It helps the robot understand what is moving in the scene, often indicating active use.

The resolution unit of a Vision Transformer. The input image is sliced into fixed square grids, typically 16x16 or 14x14 pixels. Each square is flattened and treated as a single "token" (similar to a word in a sentence) for the Transformer to process.

Patch size determines the trade-off between speed and detail. Smaller patches (14x14) allow the robot to see finer details (better for grasping small objects) but quadruple the computational cost compared to larger patches. HRP uses ViT-B/16 (16x16 patches).

A set of data points in space, usually produced by 3D scanners or LiDAR, representing the external surface of an object.

DexCap converts depth maps into point clouds to reason about the 3D geometry of the hand and object during manipulation.

The function that maps observations to actions. In robotics, policies may output a single action, an action distribution, or an action sequence (action chunk).

Behavior cloning trains a policy by copying expert actions. Reinforcement learning trains a policy by maximizing reward.

Transferring knowledge from a large "teacher" model (or ensemble) into a smaller, efficient "student" model.

Used to compress heavy offline RL policies or complex planner outputs into real-time controllers suitable for onboard robot compute.

The robot's internal sense of its own body configuration (joint angles, velocities, torques).

A critical input for all control policies. In ViSk and MCR, proprioception is fused with vision/tactile to form a complete state estimate.

A universal visual representation pre-trained on diverse human video data (Ego4D) using time-contrastive learning.

Acts as a "frozen" perception module, allowing robots to learn new tasks with significantly fewer demonstrations.

The comprehensive set of data collected from the robot during interactions with the target environment.

Includes proprioceptive data, exteroceptive data (camera, lidar, tactile), and action space data. This is the "Ground Truth" for System Identification.

A foundational machine learning framework where an agent learns effective behavior via trial-and-error interactions to maximize a reward signal.

Used in HumanPlus (PPO) for walking, and in CIA for the adaptation phase.

A mathematical formula that calculates a score (reward) based on the robot's state and action. The robot's goal in RL is to maximize the sum of these rewards.

Defining the correct reward function is the hardest part of EgoZero and HumanPlus; if poorly defined, the robot learns incorrect behaviors (reward hacking).

A foundation model from Meta designed to extract any object from a video. It uses a memory bank to track an object's visual features frame by frame, handling occlusions (when the object is hidden) and rotation. In the MCR paper, it provides the ground truth, as the researchers use it to track all of the relevant objects in an episode.

Released by Meta (2024) as a successor to the original SAM. Unlike standard vision models that categorize ("this is a cup"), SAM 2 tracks specific pixel clusters across time. In robotics research (MCR), it is used as an "Oracle" to generate ground-truth labels for evaluation, replacing the need for expensive manual human annotation of video data.

A learning method where the data provides the supervision itself (e.g., predicting the next frame in a video), eliminating the need for human labels.

HRP, R3M, and Sparsh are all SSL methods. They allow models to learn from millions of hours of video where no human has labeled "this is a cup."

A learning approach that combines a small amount of labeled data with a large amount of unlabeled data.

HRP uses this to leverage the few videos where we know what's happening to guide learning on the massive pile of videos where we don't.

A real-time teleoperation mode where the humanoid robot mimics the human operator's full-body motion instantly.

The core data collection mechanic in HumanPlus. It allows for intuitively collecting "expert" robot data by simply having a human perform the task.

The process of training a robot policy in a physics simulator and deploying it on physical hardware.

Nearly all listed papers utilize this. HumanPlus trains walking/balance in sim and transfers to real; DexCap trains hand policies in sim using retargeted human data.

A technique for online dynamics adaptation that adjusts the physical parameters of a simulator (like gravity or friction) to better match real-world observations.

A key limitation is that it often requires a manipulable simulator that can be paused and updated in real-time.

A pre-trained encoder specifically designed for magnetic tactile skins (e.g., uSkin). Uses self-supervised learning on hand-object interactions.

Compresses high-dimensional tactile data into a useful latent embedding for tasks like slip detection without retraining from scratch.

The fundamental unit of data in robot learning, consisting of an observation (State, s) and the command executed (Action, a).

EgoZero and Behavior Cloning rely on collecting millions of these pairs to approximate the function a = π(s).

A machine learning embedding that captures the specific features necessary for a downstream task. In robotics, a "suitable" representation must encode not just semantic class (what is it?) but also geometry, physics, and interaction potential (how do I move it?), which are often lost in standard Internet-data pre-training.

HRP argues that standard human videos do not yield "suitable representations" for robots by default. The HRP pipeline is designed specifically to force the vision encoder to produce these representations by training it on contact and hand-pose losses.

A method for handling mismatch by explicitly estimating the physical parameters of the environment or robot during deployment.

Typically relies on the assumption that the target domain’s parameters fall within a distribution the agent has seen or can mathematically model.

Sensors covering the robot's body (or hands) that measure contact forces, vibration, or deformation.

ViSk and Sparsh emphasize that vision is insufficient for fine manipulation. Tactile skins provide the necessary feedback for "contact-rich" tasks.

Remote control of a robot by a human operator.

The traditional bottleneck in robotics data. DexCap and HumanPlus innovate here by making teleop more intuitive (shadowing) or portable (wearable mocap).

A specific type of self-supervised learning that encourages the model to recognize that frames close together in time (in a video) are semantically similar, while frames far apart are different.

The foundational algorithm for R3M and HRP. It enables the robot to learn the "plot" or sequence of a task (approach -> grasp -> lift).

A neural network architecture that uses 1D convolutions to process sequential data over time. Unlike Recurrent Neural Networks (RNNs) which process steps sequentially, TCNs process history in parallel using dilated convolutions, allowing them to capture long-range temporal dependencies (cause and effect) efficiently.

Critical for Behavior Cloning and Imitation Learning, where the robot must understand that an action (like pouring) is a sequence, not a single snapshot. While HRP uses Transformers for temporal prediction, TCNs are the backbone of many "Action Chunking" policies like Diffusion Policy.

The process of converting continuous data (images, robot actions) into discrete units (tokens) for processing by a Transformer.

ARP and VLA models tokenize robot actions (e.g., discretizing joint angles) to treat control as a sequence modeling problem, similar to text generation.

The decoder component of a transformer architecture. In policy learning, it can fuse a set of observation tokens and produce an action feature (often via a learnable action token) that is fed to an action head.

VISK and BAKU use transformer decoder trunks as the main policy backbone for fusing multiple camera views and tactile or proprioception inputs before predicting action chunks.

Transformation to the action spaces that ensures both the source and target robots use the same command structure, even if their physical bodies differ.

HumanPlus maps human motion into a 33-DoF humanoid space. HAT uses this to co-train on human and robot data effectively.

The modern SOTA (state of the art) architecture for image processing that treats images as a sequence of "tokens" (patches), similar to how LLMs treat words. It has a global receptive field from the very first layer, meaning every pixel can talk to every other pixel instantly via Self-Attention. It can group pixels on the screen by semantics, making it highly effective for understanding what it is looking at. However, it lacks the inductive bias of CNNs, meaning it requires millions of examples to learn that adjacent pixels form an edge.

**Analogy:** The vision transformer is like the master architect for a building. A bricklayer (CNN) inspects a wall one brick at a time with a magnifying glass. They know if the mortar is cracked (local detail), but they might not realize the wall is in the wrong room. Meanwhile, the Architect (ViT) considers the entire blueprint simultaneously. They immediately understand the house's flow, the relationship between the kitchen and the living room (Global Context), and the structural integrity. However, because they are looking at the big picture, they might miss a tiny scratch on a single brick.

A policy learning framework that integrates visual input with data from uncalibrated magnetic tactile skins.

Demonstrates that integrating tactile sensing significantly improves precision in tasks like peg insertion or card swiping compared to vision-only policies.

Learning policies or representations that jointly use vision and touch signals to improve contact-rich manipulation.

Visual signals help with global localization and approach, while tactile signals provide high-bandwidth feedback at the contact interface for alignment, slip detection, and force modulation. VISK shows that adding tactile tokens improves precision and spatial generalization in real-world manipulation.

A class of foundation models that take vision (images/video) and language (instructions) as input and directly output robot actions (control commands).

The "Generative AI" of robotics. Papers explore training VLAs on human data (EgoVLA) or using them for high-level planning.

Control strategies that coordinate all degrees of freedom (legs, torso, arms) simultaneously to achieve a task while maintaining balance.

HumanPlus emphasizes this, noting that simple arm manipulation is insufficient for humanoids; the robot must actively balance and use its torso to extend reach.

The ability of a model to perform a task in a new domain or with new objects without any specific training examples for that scenario.

EgoZero claims this capability, training purely on human data and deploying on robots without fine-tuning.