Reference implementation · FH-Ego v1
One clip. Every layer. Including what it gets wrong.
This is the complete annotation output of a single real egocentric recording, produced by our pipeline — not a mock-up and not a best case. Every figure below is computed from the JSON at build time, and the file itself is downloadable so you can verify each one yourself.
We deliver LeRobot v2.1, HDF5 (EgoDex-style paired files) and the open FH-Ego v1 JSON schema; Zarr v2 and TFRecord (RLDS-style) are produced on request. Every format is verified by an automated export bench before we quote it. We do not offer MCAP or WebDataset — MCAP is a robotics logging container rather than a training format, and labs shard WebDataset internally; we would rather tell you which is which.
Every figure above is computed from reference-run.json at build time. The mean is an aggregate of the hand-tracking model's own confidence output, so it is derived, not measured.
Download the full annotation (JSON)
Verify it yourself, in about thirty seconds
The page above says the file is downloadable so you can check our figures. That is only worth saying if we tell you how, so: here is the digest of the exact file that button serves, and the three commands that pull our own headline numbers back out of it.
shasum -a 256 reference-run.json
ce6591895a1d3b2071bcf4b8c11b3575b3385d03aefc99c531d7be72a1edee7a reference-run.jsonjq '.actions | length' reference-run.json # 16
jq '.tracks.kinematics_3d | length' reference-run.json # 294
jq -r '.provenance.caption_source' reference-run.json # heuristic_stub
The third one is the one to run. caption_source is where the file admits
the action track was templated rather than detected — the same admission this page
makes in words below. We would rather you read it out of the artifact than take it
from our copy, because a claim you can execute is worth more than a claim you have to
believe.
If a number on this page disagrees with the file, the file is right and we have a bug. Tell us and we will publish the correction here.
What this does not prove
Stated first, because it is the part most vendors omit.
This is one clip. A single validated end-to-end run demonstrates the pipeline executes correctly. It says nothing about consistency across Capture Partners, scenes, or lighting, and we do not claim otherwise.
No inter-annotator agreement study has been run. We have not yet published a Krippendorff's α for this schema. Until we do, treat label quality as unmeasured rather than assured.
Confidence is the model's, not ground truth. The values below come from the hand-tracking model. They are not validated against motion capture.
The action track in this run is templated, not detected. The
hand tracking below is real measurement. The action layer is not: on this clip
the segment boundaries are fixed four-second windows, the verbs cycle a closed
five-item list, and scene_semantic and camera_motion are
single constants. Nothing in that layer was inferred from the video. The artifact says
so about itself — provenance.caption_source is
heuristic_stub, and provenance.derived_fields names all nine
fields that are not measurements. A production captioner exists and is gated on
per-clip consent; this clip is a DEMO capture with no consent receipt, so
it has not run here.
Dense captions are derived, not measured. Where language layers are model-generated they are labelled as such in the schema and must not be read as human annotation.
We hold no third-party attestation. No SOC 2, no ISO 27001. We publish the consent mechanism instead, because that is verifiable today.
Known failure modes — measured on this run
Computed from the same file you can download. We publish the floor, not just the mean, because worst case is what determines whether a dataset is usable.
Confidence distribution, computed from reference-run.json
| Measure | Value |
|---|---|
| Frames below 0.70 confidence | 45 of 294 (15.3%) |
| Lowest joint confidence observed | 0.508 |
| Median joint confidence | 0.9415 |
| Highest joint confidence observed | 0.995 |
| Kinematic coverage of total frames | 10.0% — the pipeline emits keyframes, not every frame |
| Hand distribution | left: 183 · right: 111 |
How to read this. Roughly one kinematic frame in seven on this run sits below 0.70 confidence, concentrated where the hand is partially occluded. For a buyer that is the number that matters: it sets the filtering threshold you would apply before training, and it is why we ship confidence per frame rather than a single quality score.
Four frames from this run
The renders below are pipeline output, not illustrations. Each overlay is drawn by the engine from the same JSON you can download: frame index, action code, hand, detection confidence and the camera-relative position vector. The kinematics track is sampled every ten frames, so each caption names both the frame that was rendered and the nearest sampled entry behind it — two different numbers, with two different confidences, which is why we never print one under the other's label. The callouts on each still, and the readout a mouse pointer brings up, say exactly what the caption says; nothing is behind the hover.
Frame 1197 is the honest one. The right hand is tracked at 0.60 and the sampled entry behind it at 0.548 — under the 0.70 floor in the table above, because the hand is partially behind the machine head. The overlay still draws it, and the confidence value is what tells you not to trust it. That is the whole argument for shipping confidence per frame instead of one quality score.
The three layers, from this run
Tier 1 — session
{
"task_id": "20260722_Egocentric_Sample_2_task_1",
"task_label": "T0_Other",
"scene": "Industrial_Floor",
"usable": true,
"start_frame": 0,
"end_frame": 1843,
"fps": 30
}Tier 2 — actions
16 action segments in this run; the first 8 shown
Read this table as schema shape, not as detection. On this clip the boundaries are fixed four-second windows and the verbs cycle a closed five-item list — see what this does not prove. The columns show what the format carries; they do not show a model reading the video.
| Frames | Hand | Action | Description |
|---|---|---|---|
| 0–119 | left | grasp_object | The left hand operates sewing machinery and reaches forward to grasp fabric. |
| 120–239 | left | organize_items | The left hand aligns and organizes fabric items on the workstation surface. |
| 240–359 | both | wipe_surface | Both hands hold material cloth and wipe the surface in smooth motions. |
| 360–479 | both | inspect_component | Both hands pick up the component to inspect alignment and seam orientation. |
| 480–599 | both | place_item | Both hands place the processed item back into the designated workstation tray. |
| 600–719 | left | grasp_object | The left hand operates sewing machinery and reaches forward to grasp fabric. |
| 720–839 | left | organize_items | The left hand aligns and organizes fabric items on the workstation surface. |
| 840–959 | both | wipe_surface | Both hands hold material cloth and wipe the surface in smooth motions. |
Verbs observed across all 16 segments: grasp_object,
inspect_component, organize_items, place_item,
wipe_surface.
Tier 3 — timeline tracks
Entry counts per track in this run
| Track | Entries |
|---|---|
| hand_control | 5 |
| object_interactions | 5 |
| action_verbs | 5 |
| dense_captions | 16 |
| kinematics_3d | 294 |
One kinematic frame, verbatim
This is the shape the coordinate data on this page is read from — 294 of these,
sampled every ten frames, in tracks.kinematics_3d.
{
"frame_idx": 0,
"timestamp_sec": 0.0,
"hand": "left",
"position_3d_m": {
"x": -0.384,
"y": 0.282,
"z": 0.356
},
"velocity_3d_mps": {
"vx": 0.0,
"vy": 0.0,
"vz": 0.0
},
"confidence": 0.889,
"keypoints_21": [
{
"x": 0.1801,
"y": 0.8129,
"z": 0.0
},
{
"x": 0.2565,
"y": 0.8872,
"z": -0.0038
},
{
"x": 0.3251,
"y": 0.8668,
"z": 0.0054
},
{
"x": 0.3645,
"y": 0.7889,
"z": 0.0156
},
{
"x": 0.3798,
"y": 0.7276,
"z": 0.0259
},
{
"x": 0.35,
"y": 0.7193,
"z": 0.0082
},
{
"x": 0.3903,
"y": 0.6318,
"z": 0.0165
},
{
"x": 0.411,
"y": 0.5881,
"z": 0.0228
},
{
"x": 0.4252,
"y": 0.5607,
"z": 0.0275
},
{
"x": 0.3191,
"y": 0.6446,
"z": 0.0119
},
{
"x": 0.3648,
"y": 0.5688,
"z": 0.0269
},
{
"x": 0.3939,
"y": 0.5386,
"z": 0.0363
},
{
"x": 0.409,
"y": 0.516,
"z": 0.0426
},
{
"x": 0.2807,
"y": 0.6066,
"z": 0.0145
},
{
"x": 0.3157,
"y": 0.5379,
"z": 0.0167
},
{
"x": 0.3174,
"y": 0.5457,
"z": 0.0163
},
{
"x": 0.3084,
"y": 0.5569,
"z": 0.0191
},
{
"x": 0.2441,
"y": 0.5904,
"z": 0.0171
},
{
"x": 0.2671,
"y": 0.542,
"z": 0.0113
},
{
"x": 0.2677,
"y": 0.5459,
"z": 0.0084
},
{
"x": 0.2591,
"y": 0.5593,
"z": 0.0104
}
]
}Provenance — the part you can audit
Every clip we deliver carries a consent receipt in a SHA-256 ledger with revocation semantics. A receipt records who consented, when, on which device, under which DPDP policy, and whether consent was later withdrawn — and the hash lets you verify the record has not been altered. A certification attests to a process a third party reviewed once; this emits evidence about each individual clip.
Stated plainly: the ledger is implemented and running on SQLite. It is not yet backed by production-grade persistence, and this reference clip is engineering test material, not Capture Partner cohort data. We would rather tell you that than have you discover it.