Physical AI

The language layer
your training data is missing.

MetaMule Enterprise turns raw video into time-coded, natural-language action labels for physical AI. It's an automated first pass that feeds your annotators and fits your pipeline, not a platform to replace them.

Talk to us See the output
The bottleneck

Vision-Language-Action models need a per-step instruction.
The field is short on them.

Robot-learning and world-model training run on datasets where every step carries a natural-language instruction, the RLDS language annotation. It's demonstrably under-supplied. Only about 70% of Open X-Embodiment episodes carry language annotations, and quality varies sharply across sub-datasets. The standard tooling even inserts a placeholder string when a dataset has none. Writing them by hand is slow and expensive, so they get skipped. That's exactly the work to automate.

The output

Time-coded action segments, at the granularity the field already uses.

MetaMule Enterprise watches video and emits segments at the Ego4D activity → action → sub-action hierarchy. Every segment carries:

instruction

The language annotation

An imperative, one-line VLA / RLDS instruction. The highest-value, scarcest field.

description

Dense natural language

Exactly what happens in the segment, grounded in the frames.

contact_events

Approximate contact

Hand/actor and object contact, flagged for a human to confirm. Never asserted as ground truth.

confidence · needs_review

On every label

So low-confidence segments route straight to your annotators.

"segments": [{
  "level": "action", "start_s": 83.0, "end_s": 96.8,
  "instruction": "Taxi the aircraft to the runway for takeoff",
  "description": "The aircraft rolls along the tarmac as the passenger raises a hand…",
  "contact_events": [{ "type": "grasp", "actor": "a1", "object": "o5" }],
  "confidence": 0.90, "needs_review": false
}]  // maps onto RLDS episode / step and LeRobot

Not geometry. No 3D boxes, pose, grasp, segmentation, or LiDAR. A model watching video can't produce those, and they're your strength. MetaMule Enterprise fills only the language layer.

First pass, not a platform

It makes your annotators faster. It doesn't replace them.

Feeds, not competes

Every label carries a confidence and a review flag, so the output drops into your existing annotation stack as a pre-pass. Humans confirm or correct instead of labeling the language layer from scratch.

Honest by design

Uncertainty is surfaced, not hidden. Contact events are approximate. Ambiguous segments are flagged. That triage is what makes an automated pass trustworthy at volume.

Fits your pipeline

Labeled to your action taxonomy, your granularity, your ingest format: RLDS, LeRobot, or your own. A specialized supplier for one layer, not a platform to adopt.

Proof

Real footage in, structured labels out.

A single 4K clip, labeled automatically into time-coded segments with instructions, descriptions, contact events and review flags. Hand-verified against the footage, and it runs across a whole library.

21
segments from one 4-minute clip
100%
labels carry confidence + review
3
temporal levels: activity → action → sub-action

Bring us a clip and your output spec.

We'll label a representative clip to your schema so you see concrete output, not a promise. Then we adjust to your exact taxonomy and target format.

enterprise@metamule.io

Grounded in the field's standards. Ego4D: 3,670 hours of egocentric video with timestamped narrations (CVPR 2022 / IEEE TPAMI 2025). Open X-Embodiment / RLDS: 1M+ real robot trajectories across 22 embodiments, episode→step structure with a language annotation per step (Google DeepMind + 21 institutions, 2023). MetaMule Enterprise segments to the Ego4D granularity and structures output to map onto RLDS, conventions the field already uses.