RoboScript

01Video

Learn in a world, not in the weights

02Idea

The world is the robot's terminal

Coding agents learn a repository by running code and reading what comes back. RoboScript lets a robot learn a task the same way, in a simulator.

Skill in context, not weights A frozen model writes what it learns into a readable Task Memory.

Rehearse in a world One demonstration, then practice in a simulator: the feedback plain ICL lacks.

Transfer across worlds The memory is semantic, so it holds in randomized scenes and on a real robot.

Learning in weights, in-context learning and World In-Context Learning; the world as the robot's terminal; success on RoboTwin 2.0

(a) Skill in weights is bound to the training domain, ICL to the shown scene; WICL rehearses in a world and stores semantics that transfer. (b) The coding-agent stack, with the world in place of the terminal. (c) RoboTwin 2.0 success from one demo.

Abstract

Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights.

03Method

World In-Context Learning

Rehearse from one demonstration, freeze the context, deploy it on scenes it has never seen. Weights never change.

The WICL pipeline, recursive self-improvement, and transfer across worlds

(a) Rehearse in a world, freeze the context, deploy on unseen instances. (b) Self-improvement over disjoint blocks. (c) Context refers to shared semantics σ and transfers across worlds; weights do not.

Learn

\[ C_{k+1} = \mathcal{U}_\theta(C_k, \tau_k, r_k) \]

rehearse, then revise the context

Deploy

\[ a_t \sim \pi_\theta(\cdot \mid o_t, C^\star) \]

frozen context, unseen instances

Improve

\[ G_{j+1} = \mathcal{V}_\theta(G_j, D_j) \]

reflect on records, write guidance

RoboScript: four stages, frozen weights

The four stages of RoboScript: skills, task learning, deployment and reflection

D deployable data, the same inputs a real robot has · P privileged simulator data, used only as supervision while learning.

Hover over a stage.

04Rollouts

A wall of rollouts

05Infrastructure

Infrastructure of RoboScript

RoboScript infrastructure: foundation models, model gateway, agent plane, world, control plane, services, operations and plugins

Every component is a replaceable service. The agent reaches the world only through the deployable interface (D); only the control plane touches privileged data (P).

06Task Memory

What it learns is readable

The frozen model writes each task down as plain text: objects, workflow, parameters and lessons, with arms named by role, not left or right. This is the one for place_object_basket.

TASKby role, for every episode

Drop toycar into basket, let toycar settle, then move basket and stop while holding basket with toycar inside, without further adjustment.

OBJECTSrole, description, x axis

toycar: the object to drop into the container; a small rigid toy vehicle, grasped across its body [...]. x toward the front nose.
basket: receives the object and is then moved while containing it [...]. x toward the raised handle crossbar end.

WORKFLOWstages with goal, arm, calls, done

1 prepare→2 grasp_toycar→3 handover_toycar→4 lift_toycar→5 drop_toycar→6 retreat→7 grasp_basket→8 move_basket

arm: the arm plan_arms gives for drop_toycar
call: place_object(arm=<arm>, obj=toycar, place={toycar_drop}, approach_m={drop_pre_dis}, stop_back_m={drop_dis}, is_open=true, constrain='align'); [...]

PARAMETERSfitted from the demonstration

toycar_drop = place_spec(target=basket top_center, frame=target, offset_m=[0.0781, 0.0067, 0.0234], orient=relative, placement_point=bottom, [...])
lift_height = 0.1 m [...] 19 more specifications

LESSONSwritten from failures

L2 When the object has been released with the fitted tilted placement and the empty gripper must clear the basket, do keep the arm selected for the object through retreat and use return_home(arm=<arm>) for the empty gripper. [...]

Blue: added by rehearsal. Quoted from the stored file; [...] marks cuts.

07Real Robot

Rehearsed in simulation, run on a real robot

Memories learned on a simulated ALOHA run on a dual-arm PiPER after a lightweight adapter rewrites them. No new demonstration, no re-rehearsal.

Real-robot setup: a dual-arm PiPER, RGB-D cameras and the task objects

Dual-arm PiPER with RGB-D cameras.

Learn in simulationone demo + rehearsal on a simulated ALOHA
Adapt the memoryrewrite frames, kinematics and gripper; keep the workflow
Run on the PiPERsame tools, deployable inputs only

08Results

One demonstration, a new state of the art

Main results on RoboTwin 2.0

Method	Demos /task	Pol.	Agt.	Success rate (%)			Retention (%)
Method	Demos /task	Pol.	Agt.	Clean	Rand.	Avg.	Retention (%)

Mean over all 50 tasks; retention is randomized divided by clean. Pol./Agt.: uses a trained policy / an agent. Bold best, underlined second. ^†Randomized only, five seeds per task; N demonstrations fine-tune the LingBot-VLA base.

Analysis

Per-task success clean versus randomized, and failure causes

Per-task success stays within ±10 points under randomization, and knowledge failures do not grow (37 clean, 38 randomized).

09BibTeX

Cite RoboScript

@article{fang2026roboscript,
  title={World In-Context Learning for Agentic Robot Manipulation},
  author={Fang, Heng and Guo, Zehui and Zhang, Yuehan and Du, Mingyang and Liang, Dingkang and Bai, Xiang},
  journal={arXiv preprint arXiv:2611.12885},
  year={2026}
}