huggingface · adithya-s-k · Aug 27, 2026 · chatgpt-codex-connector · Aug 27, 2026 · cursor
diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
@@ -51,8 +51,6 @@
 - sections:
   - local: deepspeed_integration
     title: DeepSpeed
-  - local: harbor
-    title: Harbor
   - local: kernels_hub
     title: Kernels Hub
   - local: liger_kernel_integration

diff --git a/docs/source/example_overview.md b/docs/source/example_overview.md
@@ -35,7 +35,6 @@ Check for additional optional dependencies [here](https://github.com/huggingface
 | [`grpo_catch`](https://github.com/huggingface/trl/tree/main/examples/grpo_catch) | GRPO with the Catch (OpenSpiel) [OpenEnv](openenv) environment. | |
 | [`grpo_continuous_batching`](https://github.com/huggingface/trl/tree/main/examples/grpo_continuous_batching) | GRPO with transformers' continuous batching engine for faster generation on large batches with variable completion lengths. | |
 | [`grpo_echo`](https://github.com/huggingface/trl/tree/main/examples/grpo_echo) | Minimal GRPO training with the Echo [OpenEnv](openenv) environment. | |
-| [`grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/grpo_harbor) | GRPO training against a Harbor task suite with a pluggable base agent (`bash` / `jupyter` / `terminal_notes` harnesses). See the [Harbor Integration](harbor) guide. | |
 | [`grpo_ministral3_vl`](https://github.com/huggingface/trl/tree/main/examples/grpo_ministral3_vl) | GRPO Ministral 3 with QLoRA on free Colab. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/grpo_ministral3_vl/grpo_ministral3_vl.ipynb) |
 | [`grpo_multi_env`](https://github.com/huggingface/trl/tree/main/examples/grpo_multi_env) | Multi-environment GRPO training: Wordle + Catch [OpenEnv](openenv) environments in the same training run. | |
 | [`grpo_qlora`](https://github.com/huggingface/trl/tree/main/examples/grpo_qlora) | GRPO using QLoRA on free Colab. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/grpo_qlora/grpo_qlora.ipynb) |

diff --git a/docs/source/grpo_trainer.md b/docs/source/grpo_trainer.md
@@ -984,7 +984,6 @@ All environments plug into the same `environment_factory` slot, so they are inte
 |---|---|---|
 | [OpenEnv](openenv) | The open environment standard (Gymnasium-style API, served over WebSocket or containerised execution), backed by Hugging Face and the community. | You're using a ready-made OpenEnv environment from the Hub, or defining your own against the open standard (e.g. Wordle, Sudoku, Catch). |
 | [OpenReward](openreward) | An integration with ORS-speaking environments (the [openreward.ai](https://openreward.ai) catalog or your own ORS server); tasks **and** rewards are served over HTTP. | You want to train against an ORS environment: the catalog (e.g. `Eigent/SETA`), one you self-host on your own infra, or a local server you're developing. |
-| [Harbor](harbor) | An integration with Harbor task suites: each task is an instruction, a real sandbox image (`docker`, `e2b`, ...), and an in-sandbox verifier. | You want to train against a Harbor task suite: a tree of tasks, each a self-contained sandbox plus verifier (e.g. a data-analysis agent that explores files in a sandbox and writes an answer a grader checks). |
 
 ## Vision-Language Model (VLM) Training
 

diff --git a/docs/source/harbor.md b/docs/source/harbor.md
diff --git a/docs/source/openenv.md b/docs/source/openenv.md
@@ -13,7 +13,7 @@ This guide covers **how to integrate OpenEnv with TRL**. For more on OpenEnv its
 
 ## Choosing an environment integration
 
-OpenEnv is the native path documented here. Two further integrations — [OpenReward](openreward) and [Harbor](harbor) — conform to the same `environment_factory` contract and are interchangeable at the TRL level. See the [comparison of environment integrations](grpo_trainer#agent-training) in the GRPO guide to pick the one whose ecosystem fits your task.
+OpenEnv is the native path documented here. [OpenReward](openreward) conforms to the same `environment_factory` contract and is interchangeable at the TRL level. See the [comparison of environment integrations](grpo_trainer#agent-training) in the GRPO guide to pick the one whose ecosystem fits your task.
 
 ## Installation
 

diff --git a/examples/async_grpo_harbor/PR_DESCRIPTION.md b/examples/async_grpo_harbor/PR_DESCRIPTION.md
@@ -0,0 +1,64 @@
+<!-- PR description for examples/async_grpo_harbor. Scaffolding — delete before pushing, or keep it
+     untracked. -->
+
+Adds an AsyncGRPO example that trains against any [Harbor](https://www.harborframework.com) task suite served through **[OpenEnv](https://github.com/huggingface/OpenEnv)**.
+
+The shape is: **pick a Harbor dataset, pick a sandbox, pick a harness, and train.** All three are per-rollout choices against one long-lived OpenEnv server — so switching harness or sandbox is an argument, not a rebuild, and the same server serves training and evaluation at the same time.
+
+```python
+HarborSessionFactory(
+    server,                    # one OpenEnv server owns the dataset + sandbox templates
+    split="<any Harbor suite>",
+    sandbox="<any backend>",   # e2b, docker, daytona, modal, gke, ...
+    harness="<any harness>",   # any agent the server reports as validated
+    llm_url=vllm_url,          # the engine is chosen PER ROLLOUT
+    model=model,
+)
+```
+
+This is the case [#6018](https://github.com/huggingface/trl/pull/6018) explicitly left out. That PR supported Harbor's *external* agents only, because "RL needs the trainer to drive generation turn-by-turn and capture the policy's tokens/log-probs + env mask — which an opaque in-container agent can't expose." OpenEnv's capture proxy exposes exactly that, so **installed agents that own their own loop become trainable without reimplementing them**.
+
+```mermaid
+flowchart LR
+    A["harness<br/>(any sandbox)"] -->|OpenAI-compatible calls| P["OpenEnv<br/>capture proxy"]
+    P -->|forwards| V["vLLM"]
+    P -.->|"token_ids + processed logprobs"| T["AsyncGRPOTrainer"]
+    T -->|NCCL weight sync| V
+    A -->|writes workspace| G["Harbor verifier"]
+    G -.->|reward| T
+```
+
+The harness owns its loop; TRL never calls `step()`. It stands up an endpoint, lets the agent drive, and reads back what happened. Because the agent's calls and the trainer's weight updates go to the **same** vLLM, rollouts stay on-policy — and OpenEnv decides the tier by probing that engine: token ids plus processed logprobs mean `train`; anything less means `eval`, and the session yields no trainable turns rather than rows of zeros.
+
+Nothing is added to TRL. Everything Harbor-specific lives in OpenEnv (`harbor_env.harness`), so the example file is the whole integration.
+
+## Usage
+
+```sh
+# 1. One OpenEnv server owns the dataset and the sandbox templates. Long-lived: the engine is named
+#    per rollout, so changing engines needs no restart.
+openenv harbor serve --dataset <hf-dataset> --port 8200 --capture-port 8300 --expose gradio
+
+# 2. Serve the policy. The token-id and logprob flags are load-bearing, not optional.
+CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
+    --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
+    --default-chat-template-kwargs '{"enable_thinking": false}' \
+    --logprobs-mode processed_logprobs --return-tokens-as-token-ids \
+    --weight-transfer-config '{"backend":"nccl"}'
+
+# 3. Train.
+CUDA_VISIBLE_DEVICES=1 python examples/async_grpo_harbor/async_grpo_harbor.py \
+    --server http://localhost:8200 --vllm-url http://localhost:8000 \
+    --model Qwen/Qwen3.5-2B --split <hf-dataset> --max-steps 20
+```
+
+## Defaults, and why they are the defaults
+
+`--harness mini-swe-agent --sandbox e2b`. Any harness the server reports works, but two properties decide which one to *train* on, and they were measured across a 15-harness sweep on the same 50 tasks:
+
+- **Prompt re-render must be byte-exact** against the engine's own `prompt_token_ids`. TRL rebuilds each prompt locally because `TraceEntry` carries no prompt ids, and for three of twelve harnesses measured that drifts — `claude-code` +2 tokens, `gemini-cli` +2, `kimi-cli` −10 per tool call. Invisible for eval; forks the trajectory *every turn* when training.
+- **A step limit must be expressible.** Every turn re-sends the whole conversation, so a rollout's packed length grows with the **square** of its turn count; unbounded 58-turn rollouts were enough to OOM the loss step on an 80 GiB card. `mini-swe-agent` is the one harness that honours a limit.
+
+## Depends on
+
+**[huggingface/OpenEnv#1036](https://github.com/huggingface/OpenEnv/pull/1036)**, which adds `envs/harbor_env` and the capture layer this example is built on. The PEP 723 header references it by git subdirectory, so the example is not installable until that lands.