twanghcmut/gr00t-n1d5-so101-left-arm-multitask

GR00T-N1.5 SO101 Left Arm, 11-task

NVIDIA Isaac GR00T N1.5 3B finetuned with the N1.5 trainer of Isaac-GR00T (scripts/gr00t_finetune.py, commit 4af2b62) on the 11 datasets of the SO101_LeftArmOnly_Task collection, merged into one multitask set (hungho77/so101-left-arm-multitask). One model covers all 11 tasks; the language instruction selects the behaviour.

Same data and sample budget (6,000 x 128 = 768k) as the GR00T N1.6, GR00T N1.7 and pi0.5 left-arm fine-tunes.

Tasks

Use the exact wording, including trailing punctuation; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	Pick up the red cube and place it into the steel pot.	50
1	Pick up the blue cube and place it into the steel pot	50
2	Take the banana and place it into the steel pot then close the lid.	50
3	Pick up the blue cube and place it next to the blue cube.	50
4	Take the blue cube out of the steel pot and place it on the blue cube.	51
5	Take the red cube out of the steel pot and place it on the red cube	50
6	Stack the yellow cube on top of the orange cube.	50
7	Pick up one orange block and stack it neatly on top of the other orange block	50
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.	50
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board	50
10	Take out the blue cubes from shortest to tallest and place it on the table	50

551 episodes, 253,745 frames in total.

Embodiment and how the data enters N1.5

SO101 follower arm (left), 5 joints plus gripper; embodiment tag new_embodiment, data config so101_multitask.

State / action dim	6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper (continuous aperture)
Cameras	top (overhead, 360x640), wrist (480x640) RGB
Action horizon	16
Data fps	30
Representation	absolute joint targets for the arm and the gripper (N1.5 has no relative-action mode); state and action normalized min_max per dimension

N1.5 does not run out of the box on this data; so101/groot_n15_so101.patch (against Isaac-GR00T 4af2b62) contains what it needed:

• So101MultitaskDataConfig (so101_multitask): top and wrist have different resolutions, and every N1.5 video transform concatenates its views along one axis, so to-tensor, crop and resize run per camera (both are 224x224 afterwards).
• dataset.py: LeRobot v2.1 stats.json carries a scalar count per feature, which broke slicing per-modality statistics.
• gr00t_finetune.py: imports torchcodec before anything that pulls in decord (both bundle FFmpeg; the wrong order makes every AV1 video fail to open with "No valid stream found"), and tolerates one MIG slice per rank.
• eval.py prints the MAE next to the MSE. scripts/eval_policy.py does not get the import-order fix: run it with --video-backend torchvision_av, or add the torchcodec import to it.

Serving

git clone https://github.com/NVIDIA/Isaac-GR00T && cd Isaac-GR00T && git checkout 4af2b62 && git apply groot_n15_so101.patch
python scripts/inference_service.py --server --model_path twanghcmut/gr00t-n1d5-so101-left-arm-multitask \
  --data-config so101_multitask --embodiment-tag new_embodiment

Training

Base	nvidia/GR00T-N1.5-3B; trained: action-head projector and diffusion (DiT); Eagle backbone (LLM and vision) frozen
Steps	6,000 (3.03 epochs)
Batch	128 per optimizer step = 16 per micro-batch x 8 gradient accumulation
Learning rate	1e-4, cosine, 5% warmup; weight decay 1e-5; AdamW
Hardware	one H100 MIG 3g.40gb slice (27 GiB peak), 13 h 20 min (16.0 samples/s)
Final train loss	0.0099 (mean over steps 5,760-6,000)

Loss at steps 500 / 1,000 / 2,000 / 3,000 / 4,000 / 5,000 / 6,000: 0.033 -> 0.026 -> 0.023 -> 0.016 -> 0.012 -> 0.010 -> 0.010. Launcher: so101/run_so101_leftarm_n15.sh.

Open-loop evaluation (in-sample)

One trajectory per task (episodes 0, 50, ..., 501), up to 400 steps, action horizon 16, 4 denoising steps, against the training data, unnormalized, in the dataset's action units. Inference is stochastic (flow matching from random noise); a first full run of the same checkpoint gave MSEs within about 20% of these per trajectory.

traj	task	MAE
0	0 red cube → pot	1.825
50	1 blue cube → pot	1.951
100	2 banana → pot, close lid	2.580
150	3 blue cube next to blue cube	2.378
200	4 blue cube out of pot → blue cube	2.239
251	5 red cube out of pot → red cube	2.338
301	6 stack yellow on orange	2.639
351	7 stack orange blocks	3.260
401	8 balance red/blue on board	2.395
451	9 red cube via hollow cube	2.059
501	10 blue cubes shortest → tallest	2.450
	average	2.374 (MSE 15.84)

Divided by the absolute action std (28.67) that is 8.28% of action scale at horizon 16. On the same trajectories holding the action at the chunk start for 16 steps scores 11.45% and copying the current state 19.23%. Other models in this series on the same protocol (all in-sample): MolmoAct2 4.37%, TurboVLA 4.62%, SmolVLA 6.72%, GR00T N1.6 7.67%, X-VLA 8.18%, pi0.5 9.82%, GR00T N1.7 10.27%, Evo-1 11.46%, RLDX-1 15.85%. Each number comes from its framework's own open-loop script.

Limitations

No held-out split. All 551 episodes were used for training, so these numbers measure fit, not generalization (held-out evaluation of SmolVLA and TurboVLA on this dataset showed errors about 3.5x their in-sample values), and they are not a success rate. No real-robot result exists for this model. One trajectory per task is a small sample.

Deployment timing

The data is 30 fps; a client that interpolates between model steps must derive its sub-step count from that ratio: smooth_step = control_hz / 30.

License

Derivative of nvidia/GR00T-N1.5-3B, distributed under the NVIDIA License (LICENSE, copied from the base model): use is limited to non-commercial purposes and the license text must accompany redistribution.