diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 12ba34b3fb3..727fc07c32a 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -51,8 +51,6 @@ - sections: - local: deepspeed_integration title: DeepSpeed - - local: harbor - title: Harbor - local: kernels_hub title: Kernels Hub - local: liger_kernel_integration diff --git a/docs/source/example_overview.md b/docs/source/example_overview.md index d614d8a9745..ccc52b602af 100644 --- a/docs/source/example_overview.md +++ b/docs/source/example_overview.md @@ -35,7 +35,6 @@ Check for additional optional dependencies [here](https://github.com/huggingface | [`grpo_catch`](https://github.com/huggingface/trl/tree/main/examples/grpo_catch) | GRPO with the Catch (OpenSpiel) [OpenEnv](openenv) environment. | | | [`grpo_continuous_batching`](https://github.com/huggingface/trl/tree/main/examples/grpo_continuous_batching) | GRPO with transformers' continuous batching engine for faster generation on large batches with variable completion lengths. | | | [`grpo_echo`](https://github.com/huggingface/trl/tree/main/examples/grpo_echo) | Minimal GRPO training with the Echo [OpenEnv](openenv) environment. | | -| [`grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/grpo_harbor) | GRPO training against a Harbor task suite with a pluggable base agent (`bash` / `jupyter` / `terminal_notes` harnesses). See the [Harbor Integration](harbor) guide. | | | [`grpo_ministral3_vl`](https://github.com/huggingface/trl/tree/main/examples/grpo_ministral3_vl) | GRPO Ministral 3 with QLoRA on free Colab. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/grpo_ministral3_vl/grpo_ministral3_vl.ipynb) | | [`grpo_multi_env`](https://github.com/huggingface/trl/tree/main/examples/grpo_multi_env) | Multi-environment GRPO training: Wordle + Catch [OpenEnv](openenv) environments in the same training run. | | | [`grpo_qlora`](https://github.com/huggingface/trl/tree/main/examples/grpo_qlora) | GRPO using QLoRA on free Colab. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/grpo_qlora/grpo_qlora.ipynb) | diff --git a/docs/source/grpo_trainer.md b/docs/source/grpo_trainer.md index 25546593490..7ce930b68c8 100644 --- a/docs/source/grpo_trainer.md +++ b/docs/source/grpo_trainer.md @@ -984,7 +984,6 @@ All environments plug into the same `environment_factory` slot, so they are inte |---|---|---| | [OpenEnv](openenv) | The open environment standard (Gymnasium-style API, served over WebSocket or containerised execution), backed by Hugging Face and the community. | You're using a ready-made OpenEnv environment from the Hub, or defining your own against the open standard (e.g. Wordle, Sudoku, Catch). | | [OpenReward](openreward) | An integration with ORS-speaking environments (the [openreward.ai](https://openreward.ai) catalog or your own ORS server); tasks **and** rewards are served over HTTP. | You want to train against an ORS environment: the catalog (e.g. `Eigent/SETA`), one you self-host on your own infra, or a local server you're developing. | -| [Harbor](harbor) | An integration with Harbor task suites: each task is an instruction, a real sandbox image (`docker`, `e2b`, ...), and an in-sandbox verifier. | You want to train against a Harbor task suite: a tree of tasks, each a self-contained sandbox plus verifier (e.g. a data-analysis agent that explores files in a sandbox and writes an answer a grader checks). | ## Vision-Language Model (VLM) Training diff --git a/docs/source/harbor.md b/docs/source/harbor.md deleted file mode 100644 index bcd3ced7e7c..00000000000 --- a/docs/source/harbor.md +++ /dev/null @@ -1,171 +0,0 @@ -# Harbor Integration for Training LLMs with Environments - -[Harbor](https://www.harborframework.com) is a framework for running agentic tasks in sandboxes. It decouples a **task** (instruction + sandbox image + verifier), a **harness/agent** (the tool surface + loop), and a **sandbox** (`docker`, `e2b`, `daytona`, `gke`, …) so they can be mixed freely. This makes it a natural fit for RL: the same task suite can be trained with different tool surfaces, on whichever sandbox backend you prefer. - -This guide covers **how to integrate Harbor with TRL**. For Harbor itself, see the [Harbor docs](https://www.harborframework.com/docs). - -> [!NOTE] -> The integration lives at `trl.experimental.harbor` and is gated behind the `trl[harbor]` extra (lazy-imported — non-users pay nothing). - -## When to use Harbor environments - -[`GRPOTrainer`] supports environment-based training via the `environment_factory` slot — see [OpenEnv](openenv) for the general contract. Use Harbor when you want to train against a **Harbor task suite**: a directory tree of tasks, each a self-contained sandbox + verifier (for example, a data-analysis agent suite where the model explores files in a sandbox and writes an answer that a grader checks). - -## Installation - -```bash -pip install trl[harbor] -``` - -> [!IMPORTANT] -> Harbor drives generation through vLLM and uses `environment_factory`, which requires `vllm>=0.22.0` and `transformers>=5.2.0`. -> -> ```bash -> pip install 'vllm>=0.22.0' -> ``` - -This installs the `harbor` framework (Python >= 3.12). The integration imports `harbor` lazily and runs it **in-process**, so users who don't touch `trl.experimental.harbor` aren't affected. - -A sandbox backend must also be installed and reachable at train time. Harbor keeps cloud backends behind its own extras, so install the one you intend to use and provide its credentials: - -```bash -pip install "harbor[e2b]" # E2B cloud sandbox -> environment_type="e2b", needs E2B_API_KEY -# docker backend (environment_type="docker", Harbor's default) just needs a reachable Docker daemon -``` - -## Quick start - -`HarborSpec` wires a single Harbor task suite into the three TRL trainer slots — `train_dataset`, `environment_factory`, `reward_funcs` — by exposing properties that map 1:1 to those kwarg names: - -```python -from trl import GRPOConfig, GRPOTrainer -from trl.experimental.harbor import HarborSpec - -spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64) - -trainer = GRPOTrainer( - model="Qwen/Qwen3-4B", - args=GRPOConfig( - num_generations=8, - max_steps=50, - max_tool_calling_iterations=25, - log_completions=True, - ), - train_dataset=spec.train_dataset, - environment_factory=spec.environment_factory, - reward_funcs=spec.reward_funcs, -) -trainer.train() -``` - -Under the hood `HarborSpec` does three things, lazily on first access: - -1. **`spec.train_dataset`**: resolves the task suite to local task directories (downloading the HF dataset if needed) and builds a `datasets.Dataset` with `prompt` (empty — the env's instruction is appended at `reset`), `task_dir`, `task_index`, plus per-task `task.toml` metadata columns. -2. **`spec.environment_factory`**: returns a zero-arg callable producing a fresh per-rollout [`~trl.experimental.harbor.HarborEnv`]. On `reset(task_dir)` it starts the task's Harbor sandbox and returns its instruction; tool methods exec into the sandbox; `env.reward` runs the verifier once after the rollout. -3. **`spec.reward_funcs`**: an outcome reward that reads the Harbor verifier's scalar per rollout. - -## The dataset - -`dataset` is either a Hugging Face dataset repo id holding a Harbor task tree, or a local path containing a `tasks/` subtree. Each task is a directory: - -``` -tasks// -├── instruction.md # the task prompt (returned by reset) -├── task.toml # config + metadata (gold answer, difficulty, ...) -├── environment/ # Dockerfile (+ any pre-agent data hooks) -└── tests/ # test.sh / grader → writes the reward -``` - -Select a subset with `num_tasks` or `indices` (mutually exclusive): - -```python -spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", num_tasks=10) # first 10 -spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", indices=[0, 5, 13]) # specific -``` - -## Agents: external vs installed - -Harbor supports two ways an agent drives a task, and the distinction determines what can be trained with RL: - -- [**External agents**](https://www.harborframework.com/docs/agents#external-agents) run *outside* the sandbox and drive the loop themselves, issuing commands into the container through Harbor's environment interface ("typically by executing bash commands via the `exec` method"). The agent decides each action and interprets each result; the sandbox only executes. -- [**Installed agents**](https://www.harborframework.com/docs/agents#installed-agents) are installed *into the container image* and run there as a headless subprocess (extending `BaseInstalledAgent`). Harbor launches the agent inside the sandbox and parses its trajectory file afterward (`populate_context_post_run`); the agent runs autonomously with its own inference. - -**TRL's integration is the external-agent pattern, and only that pattern is supported for now.** RL training requires the trainer to drive the rollout turn by turn: the *policy model being trained* generates each turn, and TRL captures its tokens and log-probs and applies the environment mask — exactly what `environment_factory` provides over a black-box `rollout_func`. An installed agent is opaque to this: it runs inside the container with its *own* model and only emits a trajectory after the fact, so there are no policy tokens or log-probs for the trainer to optimize, and the model under training is never invoked. A [`~trl.experimental.harbor.HarborEnv`] is therefore an external agent — its tool methods `exec` into the sandbox, but the loop, and the model under training, stay in TRL. - -## Selecting the base agent (harness) - -The **base agent** is the harness — which tool methods the env exposes and how it submits. Select it with `agent=`: - -```python -HarborSpec(dataset, agent="bash") # built-in single-bash-tool harness -HarborSpec(dataset, agent="my_pkg.harnesses:JupyterEnv") # import path to your HarborEnv subclass -HarborSpec(dataset, agent="path/to/harness.py:JupyterEnv") # file path to your HarborEnv subclass -HarborSpec(dataset, agent=MyHarborEnv) # a HarborEnv subclass directly -``` - -The built-in `"bash"` harness ([`~trl.experimental.harbor.HarborBashEnv`]) exposes one `bash` tool and submits by writing `/workdir/answer.txt`. Two richer harnesses ship as examples — each in its own folder with a README listing its tools — under [`examples/grpo_harbor/harnesses/`](https://github.com/huggingface/trl/tree/main/examples/grpo_harbor/harnesses): - -- [`jupyter/`](https://github.com/huggingface/trl/tree/main/examples/grpo_harbor/harnesses/jupyter) (`JupyterEnv`) — a stateful Python kernel (variables persist across cells) + a shell tool. -- [`terminal_notes/`](https://github.com/huggingface/trl/tree/main/examples/grpo_harbor/harnesses/terminal_notes) (`TerminalNotesEnv`) — 6 shell tools (incl. background processes) + a 4-tool persistent note toolkit. - -```python -HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/jupyter/env.py:JupyterEnv") -HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/terminal_notes/env.py:TerminalNotesEnv") -``` - -To write your own harness, subclass [`~trl.experimental.harbor.HarborEnv`] and add tool methods — every public method becomes a tool (TRL discovers them with `inspect.getmembers`), so give each a typed signature and a docstring (used to build the tool schema). Keep helpers underscore-prefixed. Use `self._exec(cmd)` to run shell commands in the sandbox, and set `PROMPT_SUFFIX` to append harness guidance to the task instruction: - -```python -from trl.experimental.harbor import HarborEnv - -class GrepEnv(HarborEnv): - PROMPT_SUFFIX = "\n\nUse `grep` and `read_file`. Submit by writing /workdir/answer.txt." - - def grep(self, pattern: str, path: str) -> str: - """Search for `pattern` under `path`. - - Args: - pattern: The regex to search for. - path: The file or directory to search. - """ - return self._exec(f"grep -rn {pattern!r} {path!r}") -``` - -## The sandbox backend - -`environment_type` is passed straight through to Harbor (not validated by TRL): - -```python -HarborSpec(dataset, environment_type="e2b") # cloud sandbox (offloads provisioning), needs E2B_API_KEY -HarborSpec(dataset, environment_type="docker") # default; needs a local Docker daemon -``` - -`e2b` is recommended for cluster training: only `environment.exec` crosses into the cloud sandbox, so the GPUs stay dedicated to the policy and you can run many rollouts concurrently. - -## Reward functions - -`spec.reward_funcs` defaults to an outcome reward — per rollout it reads the Harbor verifier's scalar (`env.reward`), computed once after the rollout by running the task's `tests/` verifier in the sandbox. For a custom reward, write a regular TRL reward function: - -```python -def my_reward(environments, **kwargs) -> list[float]: - return [env.reward for env in environments] -``` - -## API - -[[autodoc]] trl.experimental.harbor.HarborSpec - -[[autodoc]] trl.experimental.harbor.HarborEnv - -[[autodoc]] trl.experimental.harbor.HarborBashEnv - -## Limitations - -- The integration is in `trl.experimental` — APIs may change. Set `TRL_EXPERIMENTAL_SILENCE=1` to silence the warning in CI logs. -- Harbor's async sandbox client is bound to one event loop, so each env drives start/exec/verify synchronously on its own loop; sandbox provisioning is therefore sequential across the generation batch (cloud backends like `e2b` mitigate the per-sandbox cost). -- A single `HarborSpec` covers one task suite + one harness; multi-suite training is not supported yet. - -## Reference - -- [Harbor framework](https://www.harborframework.com) -- [Harbor RL training docs](https://www.harborframework.com/docs/training-workflows/rl) diff --git a/docs/source/openenv.md b/docs/source/openenv.md index 81fc383775e..06e5f24530d 100644 --- a/docs/source/openenv.md +++ b/docs/source/openenv.md @@ -13,7 +13,7 @@ This guide covers **how to integrate OpenEnv with TRL**. For more on OpenEnv its ## Choosing an environment integration -OpenEnv is the native path documented here. Two further integrations — [OpenReward](openreward) and [Harbor](harbor) — conform to the same `environment_factory` contract and are interchangeable at the TRL level. See the [comparison of environment integrations](grpo_trainer#agent-training) in the GRPO guide to pick the one whose ecosystem fits your task. +OpenEnv is the native path documented here. [OpenReward](openreward) conforms to the same `environment_factory` contract and is interchangeable at the TRL level. See the [comparison of environment integrations](grpo_trainer#agent-training) in the GRPO guide to pick the one whose ecosystem fits your task. ## Installation diff --git a/examples/async_grpo_harbor/PR_DESCRIPTION.md b/examples/async_grpo_harbor/PR_DESCRIPTION.md new file mode 100644 index 00000000000..faf281e137d --- /dev/null +++ b/examples/async_grpo_harbor/PR_DESCRIPTION.md @@ -0,0 +1,64 @@ + + +Adds an AsyncGRPO example that trains against any [Harbor](https://www.harborframework.com) task suite served through **[OpenEnv](https://github.com/huggingface/OpenEnv)**. + +The shape is: **pick a Harbor dataset, pick a sandbox, pick a harness, and train.** All three are per-rollout choices against one long-lived OpenEnv server — so switching harness or sandbox is an argument, not a rebuild, and the same server serves training and evaluation at the same time. + +```python +HarborSessionFactory( + server, # one OpenEnv server owns the dataset + sandbox templates + split="", + sandbox="", # e2b, docker, daytona, modal, gke, ... + harness="", # any agent the server reports as validated + llm_url=vllm_url, # the engine is chosen PER ROLLOUT + model=model, +) +``` + +This is the case [#6018](https://github.com/huggingface/trl/pull/6018) explicitly left out. That PR supported Harbor's *external* agents only, because "RL needs the trainer to drive generation turn-by-turn and capture the policy's tokens/log-probs + env mask — which an opaque in-container agent can't expose." OpenEnv's capture proxy exposes exactly that, so **installed agents that own their own loop become trainable without reimplementing them**. + +```mermaid +flowchart LR + A["harness
(any sandbox)"] -->|OpenAI-compatible calls| P["OpenEnv
capture proxy"] + P -->|forwards| V["vLLM"] + P -.->|"token_ids + processed logprobs"| T["AsyncGRPOTrainer"] + T -->|NCCL weight sync| V + A -->|writes workspace| G["Harbor verifier"] + G -.->|reward| T +``` + +The harness owns its loop; TRL never calls `step()`. It stands up an endpoint, lets the agent drive, and reads back what happened. Because the agent's calls and the trainer's weight updates go to the **same** vLLM, rollouts stay on-policy — and OpenEnv decides the tier by probing that engine: token ids plus processed logprobs mean `train`; anything less means `eval`, and the session yields no trainable turns rather than rows of zeros. + +Nothing is added to TRL. Everything Harbor-specific lives in OpenEnv (`harbor_env.harness`), so the example file is the whole integration. + +## Usage + +```sh +# 1. One OpenEnv server owns the dataset and the sandbox templates. Long-lived: the engine is named +# per rollout, so changing engines needs no restart. +openenv harbor serve --dataset --port 8200 --capture-port 8300 --expose gradio + +# 2. Serve the policy. The token-id and logprob flags are load-bearing, not optional. +CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \ + --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ + --default-chat-template-kwargs '{"enable_thinking": false}' \ + --logprobs-mode processed_logprobs --return-tokens-as-token-ids \ + --weight-transfer-config '{"backend":"nccl"}' + +# 3. Train. +CUDA_VISIBLE_DEVICES=1 python examples/async_grpo_harbor/async_grpo_harbor.py \ + --server http://localhost:8200 --vllm-url http://localhost:8000 \ + --model Qwen/Qwen3.5-2B --split --max-steps 20 +``` + +## Defaults, and why they are the defaults + +`--harness mini-swe-agent --sandbox e2b`. Any harness the server reports works, but two properties decide which one to *train* on, and they were measured across a 15-harness sweep on the same 50 tasks: + +- **Prompt re-render must be byte-exact** against the engine's own `prompt_token_ids`. TRL rebuilds each prompt locally because `TraceEntry` carries no prompt ids, and for three of twelve harnesses measured that drifts — `claude-code` +2 tokens, `gemini-cli` +2, `kimi-cli` −10 per tool call. Invisible for eval; forks the trajectory *every turn* when training. +- **A step limit must be expressible.** Every turn re-sends the whole conversation, so a rollout's packed length grows with the **square** of its turn count; unbounded 58-turn rollouts were enough to OOM the loss step on an 80 GiB card. `mini-swe-agent` is the one harness that honours a limit. + +## Depends on + +**[huggingface/OpenEnv#1036](https://github.com/huggingface/OpenEnv/pull/1036)**, which adds `envs/harbor_env` and the capture layer this example is built on. The PEP 723 header references it by git subdirectory, so the example is not installable until that lands. diff --git a/examples/grpo_harbor/grpo_harbor.py b/examples/grpo_harbor/grpo_harbor.py deleted file mode 100644 index 37369c609d0..00000000000 --- a/examples/grpo_harbor/grpo_harbor.py +++ /dev/null @@ -1,134 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# /// script -# dependencies = ["trl[vllm,harbor]", "harbor[e2b]"] # harbor[e2b]: the --env e2b backend (swap per --env) -# /// - -"""GRPO training against a Harbor task suite. - -One ``HarborSpec`` fans out into TRL's three slots — ``.train_dataset`` / ``.environment_factory`` / -``.reward_funcs``. The base agent (harness) is selected with ``--agent``: the built-in ``"bash"``, or -an import/file path to a custom ``HarborEnv`` subclass. Harbor runs in-process, so install with -``trl[harbor]`` (Python >= 3.12) plus the chosen sandbox backend's extra (``pip install "harbor[e2b]"`` -for the recommended ``--env e2b``; the ``docker`` backend just needs a reachable Docker daemon). - -Usage (server vLLM, single-node 2+2 GPU split): - -```sh -# Terminal 1 — vLLM -CUDA_VISIBLE_DEVICES=2,3 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3-4B --tensor-parallel-size 2 --port 8000 \ - --weight-transfer-config '{"backend": "nccl"}' \ - --logprobs-mode processed_logprobs \ - --max-logprobs -1 - -# Terminal 2 — training -CUDA_VISIBLE_DEVICES=0,1 accelerate launch \ - --config_file examples/accelerate_configs/deepspeed_zero2.yaml --num_processes 2 \ - examples/grpo_harbor/grpo_harbor.py \ - --vllm-mode server --vllm-server-base-url http://localhost:8000 --env e2b -``` -""" - -import argparse - -from trl import GRPOConfig, GRPOTrainer -from trl.experimental.harbor import HarborSpec - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser(description="GRPO training against a Harbor task suite.") - - parser.add_argument("--model", type=str, default="Qwen/Qwen3-4B") - parser.add_argument( - "--dataset", - type=str, - default="AdithyaSK/data_agent_rl_environment_train", - help="HF dataset repo id with a Harbor task tree, or a local path containing a tasks/ subtree.", - ) - parser.add_argument( - "--agent", - type=str, - default="bash", - help="Base agent/harness: built-in name ('bash'), import path 'pkg.mod:Class', or file 'path.py:Class'.", - ) - parser.add_argument( - "--env", - dest="environment_type", - type=str, - default="docker", - help="Harbor sandbox backend (docker, e2b, daytona, gke, modal, runloop, ...).", - ) - parser.add_argument("--num-tasks", type=int, default=64) - - parser.add_argument("--learning-rate", type=float, default=1e-6) - parser.add_argument("--per-device-train-batch-size", type=int, default=8) - parser.add_argument("--num-generations", type=int, default=8) - parser.add_argument("--max-completion-length", type=int, default=2048) - parser.add_argument("--max-steps", type=int, default=50) - parser.add_argument("--max-tool-calling-iterations", type=int, default=25) - - parser.add_argument("--vllm-mode", choices=("colocate", "server"), default="colocate") - parser.add_argument("--vllm-server-base-url", type=str, default="http://localhost:8000") - parser.add_argument("--vllm-gpu-memory-utilization", type=float, default=0.3) - - parser.add_argument("--output-dir", type=str, default=None) - parser.add_argument("--report-to", type=str, default="none") - - return parser.parse_args() - - -def main() -> None: - args = parse_args() - - # One spec object — fans out into TRL's three slots. - spec = HarborSpec( - args.dataset, - agent=args.agent, - environment_type=args.environment_type, - num_tasks=args.num_tasks, - ) - - config_kwargs: dict = dict( - learning_rate=args.learning_rate, - per_device_train_batch_size=args.per_device_train_batch_size, - num_generations=args.num_generations, - max_completion_length=args.max_completion_length, - max_steps=args.max_steps, - max_tool_calling_iterations=args.max_tool_calling_iterations, - chat_template_kwargs={"enable_thinking": False}, - log_completions=True, - use_vllm=True, - vllm_mode=args.vllm_mode, - report_to=[s.strip() for s in args.report_to.split(",") if s.strip() and s.strip() != "none"] or "none", - ) - if args.output_dir: - config_kwargs["output_dir"] = args.output_dir - if args.vllm_mode == "colocate": - config_kwargs["vllm_gpu_memory_utilization"] = args.vllm_gpu_memory_utilization - else: - config_kwargs["vllm_server_base_url"] = args.vllm_server_base_url - - trainer = GRPOTrainer( - model=args.model, - args=GRPOConfig(**config_kwargs), - train_dataset=spec.train_dataset, - environment_factory=spec.environment_factory, - reward_funcs=spec.reward_funcs, - ) - trainer.train() - - -if __name__ == "__main__": - main() diff --git a/examples/grpo_harbor/harnesses/__init__.py b/examples/grpo_harbor/harnesses/__init__.py deleted file mode 100644 index 8ea16e54408..00000000000 --- a/examples/grpo_harbor/harnesses/__init__.py +++ /dev/null @@ -1,20 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .bash import BashEnv -from .jupyter import JupyterEnv -from .terminal_notes import TerminalNotesEnv - - -__all__ = ["BashEnv", "JupyterEnv", "TerminalNotesEnv"] diff --git a/examples/grpo_harbor/harnesses/bash/README.md b/examples/grpo_harbor/harnesses/bash/README.md deleted file mode 100644 index 7746a405e44..00000000000 --- a/examples/grpo_harbor/harnesses/bash/README.md +++ /dev/null @@ -1,20 +0,0 @@ -# `bash` harness - -The minimal harness: a single shell tool. This is the **built-in** [`HarborBashEnv`](../../../../../trl/experimental/harbor/_env.py) (`trl.experimental.harbor.HarborBashEnv`); this folder just documents it and re-exports it as `BashEnv`. - -## Tools - -| Tool | Signature | What it does | -|---|---|---| -| `bash` | `bash(command: str) -> str` | Run a shell command in the sandbox; returns combined stdout+stderr (truncated to 8k). Non-stateful between calls. | - -## Submission - -No submit tool — write the answer to `/workdir/answer.txt`, e.g. `echo -n "" > /workdir/answer.txt`. The task's verifier reads that file. - -## Use it - -```python -from trl.experimental.harbor import HarborSpec -spec = HarborSpec(dataset, agent="bash") # built-in name -``` diff --git a/examples/grpo_harbor/harnesses/bash/__init__.py b/examples/grpo_harbor/harnesses/bash/__init__.py deleted file mode 100644 index d6b6e337db1..00000000000 --- a/examples/grpo_harbor/harnesses/bash/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from trl.experimental.harbor import HarborBashEnv as BashEnv # the built-in bash harness - - -__all__ = ["BashEnv"] diff --git a/examples/grpo_harbor/harnesses/jupyter/README.md b/examples/grpo_harbor/harnesses/jupyter/README.md deleted file mode 100644 index cb3f8d93d8a..00000000000 --- a/examples/grpo_harbor/harnesses/jupyter/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# `jupyter` harness - -A **stateful Python kernel** harness. Variables, imports, and side-effects persist across cells, so the -model builds up state like a notebook. Implemented by `JupyterEnv` in [`env.py`](env.py). - -## How it works - -On `reset`, the harness (`_setup`) uploads two helpers into the sandbox and starts a tiny kernel server: - -- [`kernel_server.py`](kernel_server.py) — a local HTTP server (`127.0.0.1:8765`) holding one persistent Python kernel. -- [`run_cell.py`](run_cell.py) — sends a base64-encoded cell to the kernel server and prints its JSON result. - -Each `add_and_execute_code_cell` call runs `python3 /opt/run_cell.py` against that kernel. - -## Tools - -| Tool | Signature | What it does | -|---|---|---| -| `add_and_execute_code_cell` | `add_and_execute_code_cell(code: str) -> str` | Execute Python in the **stateful** kernel; state persists across calls. Use for all computation. | -| `execute_shell_command` | `execute_shell_command(command: str) -> str` | Run a shell command (pip install, ls, …). **Not** stateful with the Python kernel. | - -## Submission - -Write the answer to `/workdir/answer.txt` — e.g. `add_and_execute_code_cell(code="open('/workdir/answer.txt','w').write(str(ans))")`. - -## Use it - -```python -from trl.experimental.harbor import HarborSpec -spec = HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/jupyter/env.py:JupyterEnv") -``` diff --git a/examples/grpo_harbor/harnesses/jupyter/__init__.py b/examples/grpo_harbor/harnesses/jupyter/__init__.py deleted file mode 100644 index 882eb6482bc..00000000000 --- a/examples/grpo_harbor/harnesses/jupyter/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .env import JupyterEnv - - -__all__ = ["JupyterEnv"] diff --git a/examples/grpo_harbor/harnesses/jupyter/env.py b/examples/grpo_harbor/harnesses/jupyter/env.py deleted file mode 100644 index 913378f2502..00000000000 --- a/examples/grpo_harbor/harnesses/jupyter/env.py +++ /dev/null @@ -1,104 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Jupyter-style base agent — a stateful-kernel Harbor harness. - -A custom [`~trl.experimental.harbor.HarborEnv`] subclass exposing a stateful Python kernel -(variables/imports persist across cells) plus a shell tool. Point a trainer at it with -``HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/jupyter/env.py:JupyterEnv")``. - -The kernel is a tiny HTTP server (`kernel_server.py`, uploaded to /opt/ and started in `_setup`); each -cell runs via `python3 /opt/run_cell.py` (see `run_cell.py`). Submission is by writing -``/workdir/answer.txt`` (same verifier as the bash harness). -""" - -import asyncio -import base64 -import json -import shlex -from pathlib import Path - -from trl.experimental.harbor import HarborEnv - - -_HERE = Path(__file__).parent - -_JUPYTER_PROMPT_SUFFIX = ( - "\n\nYou are a data-analysis agent with a **stateful Python kernel**: variables, imports, and " - "side-effects persist across `add_and_execute_code_cell` calls. Dataset files are in " - "/home/user/input/. Use `execute_shell_command` for shell (pip install, ls). **Submit your final " - "answer by writing it to /workdir/answer.txt** (e.g. " - "`add_and_execute_code_cell(code=\"open('/workdir/answer.txt','w').write(str(ans))\")`). Keep it " - "short; do not end your turn without submitting." -) - - -class JupyterEnv(HarborEnv): - """Stateful-Jupyter-kernel harness over a Harbor sandbox.""" - - PROMPT_SUFFIX = _JUPYTER_PROMPT_SUFFIX - - async def _setup(self) -> None: - # Upload the kernel server + cell runner, ensure curl, start the kernel, wait for it to bind. - await self._env.upload_file(_HERE / "kernel_server.py", "/opt/kernel_server.py") - await self._env.upload_file(_HERE / "run_cell.py", "/opt/run_cell.py") - await self._env.exec("which curl >/dev/null 2>&1 || apt-get install -y curl", timeout_sec=120) - await self._env.exec( - "nohup setsid python3 /opt/kernel_server.py >/tmp/kernel.log 2>&1 < /dev/null &", timeout_sec=30 - ) - for _ in range(30): - r = await self._env.exec("curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8765/", timeout_sec=5) - if (r.stdout or "").strip() == "200": - return - await asyncio.sleep(0.5) - log = await self._env.exec("cat /tmp/kernel.log", timeout_sec=5) - raise RuntimeError(f"kernel_server failed to bind 127.0.0.1:8765\n--- kernel.log ---\n{log.stdout}") - - def _run_cell(self, code: str) -> str: - b64 = base64.b64encode(code.encode()).decode() - result = self._loop.run_until_complete( - self._env.exec(f"python3 /opt/run_cell.py --code-b64 {shlex.quote(b64)}", timeout_sec=180) - ) - raw = (result.stdout or "").strip() - if not raw: - return f"[run_cell empty stdout, rc={result.return_code}, stderr={result.stderr or ''}]" - try: - return str(json.loads(raw).get("output", "")) - except json.JSONDecodeError: - return f"[run_cell unparseable: {raw[:500]}]" - - def add_and_execute_code_cell(self, code: str) -> str: - """ - Execute Python code in the stateful kernel. Variables, imports, and side-effects persist across - calls. Use this for all computation. - - Args: - code: The Python code to execute. - - Returns: - The textual output of the executed cell. - """ - return self._run_cell(code) - - def execute_shell_command(self, command: str) -> str: - """ - Run a shell command in the sandbox (pip install, ls, etc.). Not stateful with the Python kernel. - - Args: - command: The shell command to run. - - Returns: - The command's combined stdout and stderr. - """ - return self._exec(command) diff --git a/examples/grpo_harbor/harnesses/jupyter/kernel_server.py b/examples/grpo_harbor/harnesses/jupyter/kernel_server.py deleted file mode 100644 index 41985f4d41b..00000000000 --- a/examples/grpo_harbor/harnesses/jupyter/kernel_server.py +++ /dev/null @@ -1,98 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Tiny stateful Python execution server. - -Uploaded by JupyterToolAgent into the Harbor container at /opt/kernel_server.py -and started in the background. Listens on 127.0.0.1:8765 and accepts: - - POST / Content-Type: application/json - Body: {"code": "..."} - Reply: {"output": "", "ok": true|false} - -A single persistent globals dict survives across requests — that's the -"stateful kernel" the agent's `add_and_execute_code_cell` tool relies on. -No IPython, no jupyter_client. Just compile(...) + exec(...). -""" - -from __future__ import annotations - -import contextlib -import io -import json -import traceback -from http.server import BaseHTTPRequestHandler, HTTPServer - - -PORT = 8765 -G: dict = {"__name__": "__main__"} - - -def _exec(code: str) -> tuple[str, bool]: - out = io.StringIO() - err = io.StringIO() - ok = True - try: - # Try "single" mode first so a bare expression auto-prints (mimics Jupyter). - try: - compiled = compile(code, "", "single") - except SyntaxError: - compiled = compile(code, "", "exec") - with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): - exec(compiled, G) - except SystemExit: - pass - except BaseException: - ok = False - err.write(traceback.format_exc()) - return out.getvalue() + err.getvalue(), ok - - -class Handler(BaseHTTPRequestHandler): - def do_POST(self): - try: - n = int(self.headers.get("Content-Length", "0")) - body = self.rfile.read(n).decode("utf-8", errors="replace") - payload = json.loads(body) - code = payload.get("code", "") - output, ok = _exec(code) - except Exception: - output, ok = traceback.format_exc(), False - # Cap to keep the per-cell response small. - if len(output) > 8000: - output = output[:8000] + f"\n... [truncated {len(output) - 8000} chars]" - body_out = json.dumps({"output": output, "ok": ok}).encode() - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(body_out))) - self.end_headers() - self.wfile.write(body_out) - - def do_GET(self): - # Health check. - msg = b'{"ready": true}' - self.send_response(200) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(msg))) - self.end_headers() - self.wfile.write(msg) - - def log_message(self, *a, **k): - # Quiet — Harbor agent's exec captures stdout/stderr. - pass - - -if __name__ == "__main__": - print(f"[kernel_server] starting on 127.0.0.1:{PORT}", flush=True) - HTTPServer(("127.0.0.1", PORT), Handler).serve_forever() diff --git a/examples/grpo_harbor/harnesses/jupyter/run_cell.py b/examples/grpo_harbor/harnesses/jupyter/run_cell.py deleted file mode 100644 index 3bde9becafa..00000000000 --- a/examples/grpo_harbor/harnesses/jupyter/run_cell.py +++ /dev/null @@ -1,61 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""CLI client for the kernel_server, invoked by the Harbor agent via env.exec. - -Usage (inside the container): - python3 /opt/run_cell.py --code-b64 - -Writes the kernel server's response payload (already JSON-encoded) to stdout. -The agent base64-decodes it on the host side. Base64 sidesteps shell escaping -for code containing quotes, newlines, etc. -""" - -from __future__ import annotations - -import argparse -import base64 -import json -import sys -import urllib.request - - -PORT = 8765 - - -def main() -> int: - p = argparse.ArgumentParser() - p.add_argument("--code-b64", required=True) - p.add_argument("--timeout", type=int, default=120) - args = p.parse_args() - - code = base64.b64decode(args.code_b64).decode("utf-8") - payload = json.dumps({"code": code}).encode() - req = urllib.request.Request( - f"http://127.0.0.1:{PORT}", - data=payload, - method="POST", - headers={"Content-Type": "application/json"}, - ) - try: - with urllib.request.urlopen(req, timeout=args.timeout) as r: - sys.stdout.write(r.read().decode("utf-8", errors="replace")) - return 0 - except Exception as exc: - sys.stdout.write(json.dumps({"output": f"[run_cell err] {exc}", "ok": False})) - return 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/examples/grpo_harbor/harnesses/terminal_notes/README.md b/examples/grpo_harbor/harnesses/terminal_notes/README.md deleted file mode 100644 index a538462c96f..00000000000 --- a/examples/grpo_harbor/harnesses/terminal_notes/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# `terminal_notes` harness - -A richer shell harness: **6 shell tools** (including background processes) plus a **4-tool persistent -note toolkit**. Implemented by `TerminalNotesEnv` in [`env.py`](env.py). - -## Tools - -### Shell (backed by the sandbox) - -| Tool | Signature | What it does | -|---|---|---| -| `shell_exec` | `shell_exec(command: str, blocking: bool = True) -> str` | Run a command. Blocking → combined stdout+stderr; non-blocking → detach and return a PID. | -| `shell_write_content_to_file` | `shell_write_content_to_file(path: str, content: str) -> str` | Write `content` to `path` (used to commit `/workdir/answer.txt`). | -| `shell_write_to_process` | `shell_write_to_process(pid: str, content: str) -> str` | Write to a background process's stdin. | -| `shell_view` | `shell_view(pid: str) -> str` | Read the captured stdout of a background process. | -| `shell_wait` | `shell_wait(pid: str) -> str` | Wait (≤5 min) for a background process to exit, then return its output. | -| `shell_kill_process` | `shell_kill_process(pid: str) -> str` | SIGKILL a background process. | - -### Notes (in-env state, persist across turns of a rollout) - -| Tool | Signature | What it does | -|---|---|---| -| `create_note` | `create_note(title: str, content: str) -> str` | Create a note. | -| `append_note` | `append_note(title: str, content: str) -> str` | Append to an existing note. | -| `read_note` | `read_note(title: str) -> str` | Read a note's content. | -| `list_note` | `list_note() -> str` | List note titles + sizes. | - -> Unlike the original SETA agent, notes are **not** auto-injected into the prompt each turn (TRL owns the -> prompt under `environment_factory`); recall them on demand with `read_note` / `list_note`. - -## Submission - -Write the answer to `/workdir/answer.txt`, e.g. `shell_write_content_to_file(path="/workdir/answer.txt", content="")`. - -## Use it - -```python -from trl.experimental.harbor import HarborSpec -spec = HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/terminal_notes/env.py:TerminalNotesEnv") -``` diff --git a/examples/grpo_harbor/harnesses/terminal_notes/__init__.py b/examples/grpo_harbor/harnesses/terminal_notes/__init__.py deleted file mode 100644 index ac589231397..00000000000 --- a/examples/grpo_harbor/harnesses/terminal_notes/__init__.py +++ /dev/null @@ -1,18 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .env import TerminalNotesEnv - - -__all__ = ["TerminalNotesEnv"] diff --git a/examples/grpo_harbor/harnesses/terminal_notes/env.py b/examples/grpo_harbor/harnesses/terminal_notes/env.py deleted file mode 100644 index 51bb5a9f612..00000000000 --- a/examples/grpo_harbor/harnesses/terminal_notes/env.py +++ /dev/null @@ -1,221 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Shell + notes base agent — a 10-tool Harbor harness (6 shell + 4 notes). - -A custom [`~trl.experimental.harbor.HarborEnv`] subclass with a richer toolset than the bash harness: -six shell tools (including background processes) and a persistent note-taking toolkit. Point a trainer -at it with ``HarborSpec(dataset, agent="examples/grpo_harbor/harnesses/terminal_notes/env.py:TerminalNotesEnv")``. -Submission is by writing ``/workdir/answer.txt`` (same verifier as the bash harness). - -Notes live in-env (a dict) and survive across turns of the same rollout. They're recalled on demand via -``read_note`` / ``list_note`` (TRL owns the prompt under ``environment_factory``, so the env can't -inject them automatically each turn). -""" - -import base64 -import shlex -import uuid - -from trl.experimental.harbor import HarborEnv - - -_PROMPT_SUFFIX = ( - "\n\nYou are an autonomous data-analysis agent in a sandboxed Linux container (Python preinstalled). " - "Dataset files are in /home/user/input/. You have shell tools (shell_exec, " - "shell_write_content_to_file, shell_view/wait/kill for background procs) and a persistent note system " - "(create_note, append_note, read_note, list_note) — use notes as a scratchpad and read them back. " - "**Submit your final answer by writing it to /workdir/answer.txt via a shell tool** (e.g. " - "shell_write_content_to_file(path='/workdir/answer.txt', content=)). Keep it short; do not " - "end your turn without submitting." -) - - -class TerminalNotesEnv(HarborEnv): - """10-tool harness (6 shell + 4 notes) over a Harbor sandbox.""" - - PROMPT_SUFFIX = _PROMPT_SUFFIX - - def __init__(self, *args, **kwargs): - super().__init__(*args, **kwargs) - self._notes: dict[str, str] = {} - self._bg: dict[str, dict] = {} - - def reset(self, task_dir=None, **kwargs) -> str: - self._notes.clear() - self._bg.clear() - return super().reset(task_dir=task_dir, **kwargs) - - # ── shell toolkit ─────────────────────────────────────────────────────── - - def shell_exec(self, command: str, blocking: bool = True) -> str: - """ - Execute a shell command in the sandbox. If `blocking` (default), run synchronously and return - combined stdout+stderr; otherwise detach into the background and return the new PID. - - Args: - command: The shell command to run. - blocking: Run synchronously (True) or in the background (False). - - Returns: - Combined stdout+stderr (blocking) or the background PID. - """ - if blocking: - return self._exec(command) - token = uuid.uuid4().hex[:8] - log, pipe = f"/tmp/sh_{token}.log", f"/tmp/sh_{token}.in" - out = self._exec( - f"mkfifo {pipe} 2>/dev/null; ( nohup setsid bash -c {shlex.quote(command)} <{pipe} >{log} 2>&1 ) & echo $!", - timeout=15, - ) - pid = out.strip().split()[-1] if out.strip() else "" - if not pid.isdigit(): - return f"[shell_exec bg] failed to spawn: {out}" - self._bg[pid] = {"log": log, "pipe": pipe} - return f"Started background process PID={pid} log={log}" - - def shell_write_content_to_file(self, path: str, content: str) -> str: - """ - Write `content` to `path` in the sandbox (overwrites). Use to commit the final answer to - /workdir/answer.txt. - - Args: - path: Destination path in the sandbox. - content: File contents. - - Returns: - A confirmation string. - """ - b64 = base64.b64encode(content.encode()).decode() - out = self._exec( - f"mkdir -p $(dirname {shlex.quote(path)}) && echo {b64} | base64 -d > {shlex.quote(path)}", timeout=30 - ) - return f"Wrote {len(content)} bytes to {path}" if "rc=" not in out else f"[write_file] {out}" - - def shell_write_to_process(self, pid: str, content: str) -> str: - """ - Send `content` (with a trailing newline) to the stdin of a background process. - - Args: - pid: PID returned by shell_exec(blocking=False). - content: Text to write to stdin. - - Returns: - A confirmation string. - """ - proc = self._bg.get(pid) - if proc is None: - return f"Unknown PID={pid}. Started: {list(self._bg)}" - b64 = base64.b64encode((content + "\n").encode()).decode() - self._exec(f"echo {b64} | base64 -d > {shlex.quote(proc['pipe'])}", timeout=30) - return f"Wrote {len(content)} bytes to PID={pid} stdin" - - def shell_view(self, pid: str) -> str: - """ - Return the current captured stdout of a background process. - - Args: - pid: PID returned by shell_exec(blocking=False). - - Returns: - The captured stdout so far. - """ - proc = self._bg.get(pid) - return self._exec(f"tail -c 4000 {shlex.quote(proc['log'])} 2>/dev/null") if proc else f"Unknown PID={pid}" - - def shell_wait(self, pid: str) -> str: - """ - Wait (up to ~5 min) for a background process to terminate, then return its captured stdout. - - Args: - pid: PID returned by shell_exec(blocking=False). - - Returns: - The process output after it exits. - """ - proc = self._bg.get(pid) - if proc is None: - return f"Unknown PID={pid}" - return self._exec( - f"for i in $(seq 1 300); do [ ! -d /proc/{pid} ] && break; sleep 1; done; " - f"echo '--- exited ---'; tail -c 4000 {shlex.quote(proc['log'])} 2>/dev/null", - timeout=320, - ) - - def shell_kill_process(self, pid: str) -> str: - """ - Send SIGKILL to a background process. - - Args: - pid: PID returned by shell_exec(blocking=False). - - Returns: - A confirmation string. - """ - if pid not in self._bg: - return f"Unknown PID={pid}" - return f"Sent SIGKILL to PID={pid}: {self._exec(f'kill -9 {pid} 2>&1', timeout=10)}" - - # ── note toolkit (in-env state) ────────────────────────────────────────── - - def create_note(self, title: str, content: str) -> str: - """ - Create a persistent note (recall it later with read_note/list_note). - - Args: - title: Note title. - content: Note body. - - Returns: - A confirmation string. - """ - self._notes[title] = content - return f"Note '{title}' created ({len(content)} chars). Total: {len(self._notes)}." - - def append_note(self, title: str, content: str) -> str: - """ - Append `content` (on a new line) to an existing note. - - Args: - title: Note title. - content: Text to append. - - Returns: - A confirmation string. - """ - if title not in self._notes: - return f"Note '{title}' not found. Use create_note first." - self._notes[title] += "\n" + content - return f"Note '{title}' updated -> {len(self._notes[title])} chars." - - def read_note(self, title: str) -> str: - """ - Return the full content of a note. - - Args: - title: Note title. - - Returns: - The note content, or a not-found message. - """ - return self._notes.get(title, f"Note '{title}' not found.") - - def list_note(self) -> str: - """ - List all note titles with their character counts. - - Returns: - One line per note, or a message if there are none. - """ - return "\n".join(f"- {t} ({len(c)} chars)" for t, c in self._notes.items()) or "(no notes yet)" diff --git a/pyproject.toml b/pyproject.toml index e14cfabafd4..33017be265e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -97,9 +97,6 @@ math_verify = [ openreward = [ "openreward>=0.1.109; python_version >= '3.11'", # openreward requires Python 3.11+ ] -harbor = [ - "harbor>=0.13.0; python_version >= '3.12'", # harbor requires Python 3.12+ (pulls its sandbox backends) -] dev = [ # bco "scikit-learn", diff --git a/tests/experimental/test_harbor.py b/tests/experimental/test_harbor.py deleted file mode 100644 index f59497c8062..00000000000 --- a/tests/experimental/test_harbor.py +++ /dev/null @@ -1,138 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Tests for the Harbor x TRL integration that don't need a running Harbor sandbox. - -`harbor` is imported lazily (only when an env is *started*), so spec construction, agent resolution, dataset building, -and the reward function are all testable without `harbor` / a sandbox backend. -""" - -from pathlib import Path - -import pytest - -from trl.experimental.harbor import AGENTS, HarborBashEnv, HarborEnv, HarborSpec -from trl.experimental.harbor._spec import _outcome_reward_func, _resolve_agent - -from ..testing_utils import TrlTestCase - - -def _write_task(tasks_dir: Path, task_id: str, gold: str, difficulty: int) -> None: - d = tasks_dir / task_id - (d / "environment").mkdir(parents=True) - (d / "tests").mkdir() - (d / "instruction.md").write_text(f"Solve task {task_id}.") - # Built from a joined list (not a triple-quoted block) so doc-builder doesn't reflow the TOML. - lines = [ - "[task]", - f'name = "{task_id}"', - "[metadata]", - f'gold_answer = "{gold}"', - 'reward_mode_initial = "exact_short"', - f"difficulty_level = {difficulty}", - f'kaggle_dataset_name = "owner/{task_id}"', - ] - (d / "task.toml").write_text("\n".join(lines)) - - -class TestResolveAgent(TrlTestCase): - def test_builtin_name(self): - assert _resolve_agent("bash") is HarborBashEnv - assert AGENTS["bash"] is HarborBashEnv - - def test_class_passthrough(self): - assert _resolve_agent(HarborBashEnv) is HarborBashEnv - - def test_import_path(self): - assert _resolve_agent("trl.experimental.harbor:HarborBashEnv") is HarborBashEnv - - def test_file_path(self): - path = Path(self.tmp_dir) / "my_harness.py" - path.write_text( - "from trl.experimental.harbor import HarborEnv\n" - "class MyEnv(HarborEnv):\n" - " def run_cmd(self, command: str) -> str:\n" - " 'Run a command.\\n\\nArgs:\\n command: cmd.'\n" - " return self._exec(command)\n" - ) - cls = _resolve_agent(f"{path}:MyEnv") - assert issubclass(cls, HarborEnv) and cls.__name__ == "MyEnv" - - def test_unknown_name_raises(self): - with pytest.raises(ValueError): - _resolve_agent("not-a-harness") - - def test_non_harborenv_raises(self): - with pytest.raises(TypeError): - _resolve_agent("trl.experimental.harbor:HarborSpec") # not a HarborEnv subclass - - -class TestHarborSpecDataset(TrlTestCase): - def _suite(self) -> str: - tasks = Path(self.tmp_dir) / "tasks" - tasks.mkdir() - _write_task(tasks, "0001_a", "alpha", 0) - _write_task(tasks, "0002_b", "beta", 3) - return str(self.tmp_dir) - - def test_train_dataset_columns_and_metadata(self): - ds = HarborSpec(self._suite()).train_dataset - assert len(ds) == 2 - assert ds[0]["prompt"] == [{"role": "user", "content": ""}] # env appends instruction at reset - assert ds[0]["task_dir"].endswith("0001_a") - assert ds[0]["task_index"] == 0 - assert ds[0]["gold_answer"] == "alpha" - assert ds[1]["difficulty_level"] == 3 - - def test_num_tasks_cap(self): - ds = HarborSpec(self._suite(), num_tasks=1).train_dataset - assert len(ds) == 1 - - def test_indices_selection(self): - ds = HarborSpec(self._suite(), indices=[1]).train_dataset - assert len(ds) == 1 and ds[0]["task_dir"].endswith("0002_b") - - def test_num_tasks_and_indices_mutually_exclusive(self): - with pytest.raises(ValueError): - HarborSpec(self._suite(), num_tasks=1, indices=[0]) - - def test_environment_factory_returns_fresh_envs(self): - factory = HarborSpec(self._suite(), agent="bash").environment_factory - e1, e2 = factory(), factory() - assert isinstance(e1, HarborBashEnv) and e1 is not e2 - - -class TestRewardFunc(TrlTestCase): - def test_outcome_reward_reads_env_reward(self): - class _Env: - def __init__(self, r): - self.reward = r - - assert _outcome_reward_func([_Env(1.0), _Env(0.0)]) == [1.0, 0.0] - - def test_outcome_reward_uses_environment_reward_when_passed(self): - # AsyncGRPOTrainer captures rewards in its rollout worker and passes them as a list, with no - # live env instances. The reward func must use them directly. - assert _outcome_reward_func(environment_reward=[0.25, 0.75]) == [0.25, 0.75] - - def test_fresh_env_reward_is_zero_without_backend(self): - # The trainer discovers tool methods via `inspect.getmembers`, which evaluates properties. A fresh - # env (never `reset`) must expose its tools and return 0.0 from `reward` WITHOUT starting the - # Harbor backend or importing `harbor` (not installed in the trainer env). - import inspect - - env = HarborBashEnv() - names = {n for n, _ in inspect.getmembers(env, predicate=inspect.ismethod)} - assert {"bash", "reset"} <= names - assert env.reward == 0.0 diff --git a/tests/testing_utils.py b/tests/testing_utils.py index a643843cd1c..2a7c746e6d9 100644 --- a/tests/testing_utils.py +++ b/tests/testing_utils.py @@ -34,7 +34,6 @@ from trl.chat_template_utils import _SUPPORTS_RESPONSE_TEMPLATE from trl.import_utils import ( - is_harbor_available, is_jmespath_available, is_joblib_available, is_liger_kernel_available, @@ -47,7 +46,6 @@ require_bitsandbytes = pytest.mark.skipif(not is_bitsandbytes_available(), reason="test requires bitsandbytes") require_comet = pytest.mark.skipif(not is_comet_available(), reason="test requires comet_ml") -require_harbor = pytest.mark.skipif(not is_harbor_available(), reason="test requires harbor") require_kernels = pytest.mark.skipif(not is_kernels_available(), reason="test requires kernels") require_liger_kernel = pytest.mark.skipif(not is_liger_kernel_available(), reason="test requires liger-kernel") require_math_latex = pytest.mark.skipif(not is_math_verify_available(), reason="test requires math_verify") diff --git a/trl/experimental/harbor/__init__.py b/trl/experimental/harbor/__init__.py deleted file mode 100644 index 54b8d243e40..00000000000 --- a/trl/experimental/harbor/__init__.py +++ /dev/null @@ -1,32 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Harbor × TRL integration (experimental). - -Train on Harbor agentic task suites with `GRPOTrainer` via `environment_factory`, with a pluggable base agent -(harness). Requires `harbor` installed in the same interpreter (`pip install trl[harbor]`, Python >= 3.12); `harbor` is -imported lazily so this module imports without it. - -```python ->>> from trl.experimental.harbor import HarborSpec - ->>> spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64) -``` -""" - -from ._env import AGENTS, HarborBashEnv, HarborEnv -from ._spec import HarborSpec - - -__all__ = ["AGENTS", "HarborBashEnv", "HarborEnv", "HarborSpec"] diff --git a/trl/experimental/harbor/_env.py b/trl/experimental/harbor/_env.py deleted file mode 100644 index fdef9800e26..00000000000 --- a/trl/experimental/harbor/_env.py +++ /dev/null @@ -1,231 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Harbor-backed environments for `GRPOTrainer(environment_factory=...)`. - -A `HarborEnv` wraps a Harbor sandbox + verifier. TRL drives the rollout loop: it calls the env's tool methods during -generation and reads `env.reward` afterwards. The pluggable "base agent" is the harness — the set of tool methods the -env exposes + how it submits. `HarborBashEnv` is the single-`bash`-tool harness (submit by writing -`/workdir/answer.txt`); subclass `HarborEnv` to add your own. - -Harbor's API is async and its sandbox client is bound to the event loop it was created on, so each env owns one loop -and drives start/exec/verify/stop through it synchronously (TRL's tool loop is sync). `harbor` is imported lazily, so -importing this module does not require it installed (install `trl[harbor]`, which needs Python >= 3.12). -""" - -import asyncio -import tempfile -import threading -import uuid -from pathlib import Path - - -_NO_REWARD = object() # sentinel: reward not computed yet (0.0 is a valid reward) - - -class HarborEnv: - """Base TRL environment backed by a Harbor sandbox + verifier. - - Subclasses define the tool methods (the harness). The lifecycle TRL drives per rollout: `reset(task_dir)` (start - the task's sandbox, return its instruction) -> tool methods (exec into the sandbox) -> `reward` (run the verifier - once, lazily, after the rollout). - - Args: - environment_type (`str`, *optional*, defaults to `"docker"`): - Harbor sandbox backend, passed through to Harbor (`"docker"`, `"e2b"`, `"daytona"`, ...). - """ - - #: Extra guidance appended to the task instruction by the harness subclass. - PROMPT_SUFFIX: str = "" - - def __init__(self, environment_type: str = "docker"): - self._environment_type = environment_type - # Harbor's async sandbox client is bound to the loop it was created on, so we run that loop on a - # dedicated daemon thread and submit coroutines to it via `run_coroutine_threadsafe`. This works - # whether the caller is on a plain thread (GRPOTrainer drives tools from the main thread) or - # already inside a running event loop (AsyncGRPOTrainer's rollout worker calls tool methods from - # its own loop, where `loop.run_until_complete` would raise "another loop is already running"). - self._loop = asyncio.new_event_loop() - self._loop_thread = threading.Thread(target=self._loop.run_forever, daemon=True) - self._loop_thread.start() - self._env = None # harbor BaseEnvironment for the current task - self._task = None - self._paths = None - self._reward = _NO_REWARD - - def _run(self, coro): - """Run a coroutine on this env's loop (which lives on its own thread) and block for the result.""" - return asyncio.run_coroutine_threadsafe(coro, self._loop).result() - - def reset(self, task_dir: str | None = None, **kwargs) -> str: - if task_dir is None: - raise ValueError("HarborEnv.reset requires `task_dir` (provided by the dataset row).") - instruction = self._run(self._start(task_dir)) - self._reward = _NO_REWARD - return instruction + self.PROMPT_SUFFIX - - def _exec(self, command: str, timeout: int = 180) -> str: - """Run a shell command in the sandbox; return combined stdout+stderr (truncated to 8k).""" - result = self._run(self._env.exec(command, timeout_sec=timeout)) - out = (result.stdout or "") + (result.stderr or "") - if len(out) > 8000: - out = out[:8000] + "\n... [truncated]" - return out or f"(empty output, rc={result.return_code})" - - @property - def reward(self) -> float: - # Submission = the agent wrote /workdir/answer.txt during the rollout; the verifier reads it. - # Computed once, lazily, on first read (TRL reads this after the rollout via reward_funcs). - # A fresh env that was never `reset` (e.g. the trainer probing tool methods via - # `inspect.getmembers`, which evaluates properties) has no sandbox/task to verify — return 0.0 - # without invoking the verifier, which would start the Harbor backend and import `harbor`. - if self._env is None: - return 0.0 - if self._reward is _NO_REWARD: - self._reward = self._run(self._verify()) - return self._reward - - # ── harbor lifecycle (async, run on this env's loop) ──────────────────── - - async def _start(self, task_dir: str) -> str: - from harbor.environments.factory import EnvironmentFactory - from harbor.models.task.task import Task - from harbor.models.trial.config import EnvironmentConfig as TrialEnvironmentConfig - from harbor.models.trial.paths import TrialPaths - - await self._stop() # tear down the previous task's sandbox - self._task = Task(task_dir=Path(task_dir)) - self._paths = TrialPaths(trial_dir=Path(tempfile.mkdtemp(prefix="harbor_trl_"))) - self._env = EnvironmentFactory.create_environment_from_config( - config=TrialEnvironmentConfig(type=self._environment_type), - environment_dir=self._task.paths.environment_dir, - environment_name=self._task.short_name, - session_id=uuid.uuid4().hex, - trial_paths=self._paths, - task_env_config=self._task.config.environment, - ) - await self._env.start(force_build=False) - await self._upload_build_files() # some sandbox builds (e.g. E2B from_dockerfile) drop COPY'd files - await self._env.run_healthcheck() # task pre-agent hook (e.g. pull data into /home/user/input) - await self._env.exec("mkdir -p /workdir /home/user/input") - await self._setup() # harness-specific sandbox prep (e.g. start a Jupyter kernel) - return self._task.instruction - - async def _upload_build_files(self) -> None: - """Replicate the task Dockerfile's `COPY` directives into the sandbox. - - E2B's remote `from_dockerfile` build honors `RUN` steps but silently drops files `COPY`'d from the build - context, which breaks healthchecks that run those files (e.g. a data-pull script). We re-create them at - runtime: `upload_file` writes as the sandbox `user`, so we stage each file in a user-writable tmp path and `mv` - it into place as root (destinations like `/opt` are root-owned). Idempotent. Handles the common ``COPY - `` form; flags / globs / ``--from`` are skipped. - """ - dockerfile = self._task.paths.environment_dir / "Dockerfile" - if not dockerfile.exists(): - return - for line in dockerfile.read_text().splitlines(): - s = line.strip() - if not s.upper().startswith("COPY ") or "--from" in s: - continue - parts = [p for p in s[len("COPY ") :].split() if not p.startswith("--")] - if len(parts) < 2: - continue - *srcs, dst = parts - for src in srcs: - local = self._task.paths.environment_dir / src - if not local.is_file(): - continue - remote = dst if (len(srcs) == 1 and not dst.endswith("/")) else dst.rstrip("/") + "/" + Path(src).name - parent = remote.rsplit("/", 1)[0] or "/" - tmp = "/tmp/" + uuid.uuid4().hex - await self._env.upload_file(local, tmp) - await self._env.exec(f"mkdir -p {parent} && mv {tmp} {remote}", user="root") - - async def _setup(self) -> None: - """Harness-specific sandbox preparation, run once per `reset` after the sandbox is up. - - Override to upload helper files (`await self._env.upload_file(...)`) or start servers in the sandbox. The - default is a no-op (the bash harness needs nothing beyond the base setup). - """ - - async def _verify(self) -> float: - from harbor.models.trial.config import VerifierConfig - from harbor.models.trial.paths import EnvironmentPaths - from harbor.verifier.factory import VerifierFactory - - # Pre-create the verifier dir (test.sh redirects stdout there; the shell can't mkdir the parent). - env_paths = EnvironmentPaths.for_os(self._env.os) - await self._env.empty_dirs([env_paths.verifier_dir], chmod=True) - # Carry the task's [verifier].env (e.g. expected-answer / judge-model settings) into the verifier, - # mirroring Harbor's trial runner (`override_env=`). A default trial `VerifierConfig()` is otherwise - # correct here — the task verifier has no trial-level import_path/kwargs to forward. - verifier = VerifierFactory.create_verifier_from_config( - VerifierConfig(), - task=self._task, - trial_paths=self._paths, - environment=self._env, - override_env=self._task.config.verifier.env or None, - ) - result = await verifier.verify() - rewards = result.rewards or {} - return float(rewards.get("reward", next(iter(rewards.values()), 0.0))) - - async def _stop(self) -> None: - if self._env is not None: - try: - await self._env.stop(delete=True) - finally: - self._env = None - - def __del__(self): - try: - self._run(self._stop()) - except Exception: # noqa: BLE001 — best-effort teardown - pass - finally: - self._loop.call_soon_threadsafe(self._loop.stop) - - -_BASH_PROMPT_SUFFIX = ( - "\n\nYou have a single `bash` tool: run a shell command in the sandbox and get its stdout+stderr. " - "The dataset files are in /home/user/input/. Python 3 + pandas + numpy + scikit-learn are " - "preinstalled. **Submit your final answer by writing it to /workdir/answer.txt via the `bash` " - 'tool**, e.g. `echo -n "" > /workdir/answer.txt`. Stating the answer in prose does NOT submit ' - "it; only writing the file counts. Keep the answer short, and do not end your turn without submitting." -) - - -class HarborBashEnv(HarborEnv): - """Single-`bash`-tool harness; submit by writing `/workdir/answer.txt`.""" - - PROMPT_SUFFIX = _BASH_PROMPT_SUFFIX - - def bash(self, command: str) -> str: - """ - Run a shell command in the sandbox and return its combined stdout+stderr. The shell is non-stateful between - calls. Use it to explore files (ls, head, cat), run Python (`python3 -c "..."`), and submit the answer (`echo - -n "" > /workdir/answer.txt`). - - Args: - command: The shell command to run. - - Returns: - The command's combined stdout and stderr. - """ - return self._exec(command) - - -#: Built-in harnesses, selectable by name in `HarborSpec(agent=...)`. Pass a `HarborEnv` subclass (or an -#: import path / file path resolving to one) for a custom harness. -AGENTS: dict[str, type[HarborEnv]] = {"bash": HarborBashEnv} diff --git a/trl/experimental/harbor/_spec.py b/trl/experimental/harbor/_spec.py deleted file mode 100644 index 47240607e0b..00000000000 --- a/trl/experimental/harbor/_spec.py +++ /dev/null @@ -1,216 +0,0 @@ -# Copyright 2020-2026 The HuggingFace Team. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""User-facing spec for the Harbor × TRL integration (mirror of ``OpenRewardSpec``). - -Construct **one** ``HarborSpec`` and read three properties off it — ``.train_dataset``, ``.environment_factory``, -``.reward_funcs`` — each plugging into the matching ``GRPOTrainer`` kwarg: - -```python ->>> from trl import GRPOConfig, GRPOTrainer ->>> from trl.experimental.harbor import HarborSpec - ->>> spec = HarborSpec("AdithyaSK/data_agent_rl_environment_train", agent="bash", num_tasks=64) - ->>> trainer = GRPOTrainer( -... model="Qwen/Qwen3.5-4B", -... args=GRPOConfig(num_generations=8, max_steps=50, max_tool_calling_iterations=25), -... train_dataset=spec.train_dataset, -... environment_factory=spec.environment_factory, -... reward_funcs=spec.reward_funcs, -... ) ->>> trainer.train() -``` - -A Harbor *task* is a directory (``instruction.md`` + ``task.toml`` + ``environment/`` + ``tests/``); the dataset is a -tree of them. The ``environment_factory`` env runs Harbor in-process (see ``_env.py``), so ``harbor`` must be installed -in the same interpreter (``pip install trl[harbor]``, Python >= 3.12). The **base agent** (harness/tool surface) is -selected by ``agent=`` — ``"bash"`` today, or a custom ``HarborEnv`` subclass. -""" - -from __future__ import annotations - -import os -from collections.abc import Callable -from functools import cached_property, partial -from pathlib import Path -from typing import Any - -from ._env import AGENTS, HarborEnv - - -def _outcome_reward_func(environments=None, environment_reward=None, **_) -> list[float]: - """Default reward: the Harbor verifier's scalar per rollout. - - `GRPOTrainer` passes the live env instances as `environments=` (read `env.reward`); `AsyncGRPOTrainer` runs envs in - its rollout worker and passes the already-captured per-rollout rewards as `environment_reward=`. Support both so - the same spec plugs into either trainer. - """ - if environment_reward is not None: - return [float(r) for r in environment_reward] - return [float(env.reward) for env in environments] - - -def _resolve_agent(agent: str | type[HarborEnv]) -> type[HarborEnv]: - """Resolve the `agent=` selector to a `HarborEnv` subclass. - - Accepts a `HarborEnv` subclass, a built-in name (`"bash"`), a module import path (`"pkg.module:Class"`), or a file - path (`"path/to/file.py:Class"`). - """ - if isinstance(agent, type): - cls = agent - elif agent in AGENTS: - cls = AGENTS[agent] - elif ":" in agent: - import importlib - import importlib.util - - target, _, cls_name = agent.rpartition(":") # rpartition: don't split a Windows drive (``D:\...``) - if target.endswith(".py") or os.path.sep in target: # file path -> load module from file - spec = importlib.util.spec_from_file_location(Path(target).stem, target) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - else: # dotted module path on sys.path - module = importlib.import_module(target) - cls = getattr(module, cls_name) - else: - raise ValueError( - f"Unknown agent {agent!r}; use a built-in name ({sorted(AGENTS)}), an import path " - "'pkg.module:Class', a file path 'path/to/file.py:Class', or a HarborEnv subclass." - ) - if not (isinstance(cls, type) and issubclass(cls, HarborEnv)): - raise TypeError(f"agent {agent!r} must resolve to a HarborEnv subclass, got {cls!r}") - return cls - - -def _read_task_meta(task_dir: Path) -> dict[str, Any]: - """Pull a few useful fields out of ``task.toml`` for the dataset rows / reward funcs.""" - try: - import tomllib # stdlib on Python 3.11+; lazy so the module imports on 3.10 (e.g. doc build) - - cfg = tomllib.loads((task_dir / "task.toml").read_text()) - except Exception: # noqa: BLE001 - return {} - meta = cfg.get("metadata", {}) - return { - "gold_answer": meta.get("gold_answer"), - "reward_mode": meta.get("reward_mode_initial"), - "difficulty_level": meta.get("difficulty_level"), - "kaggle_dataset": meta.get("kaggle_dataset_name"), - } - - -class HarborSpec: - """Single spec object that wires a Harbor task suite into a TRL trainer. - - Args: - dataset (`str`): - A Hugging Face dataset repo id holding a Harbor task tree (e.g. - `"AdithyaSK/data_agent_rl_environment_train"`), or a local path to a directory containing a `tasks/` - subtree. Each task is a dir with `instruction.md` / `task.toml` / `environment/` / `tests/`. - agent (`str` or `type`, *optional*, defaults to `"bash"`): - The base agent / harness — i.e. the tool surface the env exposes. One of: a built-in name (`"bash"`), an - import path `"package.module:ClassName"`, a file path `"path/to/file.py:ClassName"`, or a - [`~trl.experimental.harbor.HarborEnv`] subclass directly. - environment_type (`str`, *optional*, defaults to `"docker"`): - Harbor sandbox backend, passed through to Harbor (whatever it supports — `"docker"`, `"e2b"`, `"daytona"`, - `"gke"`, `"modal"`, `"runloop"`, ...). Not validated here; Harbor validates. `"docker"` is Harbor's own - default; pick `"e2b"` to offload sandboxing to the cloud. - num_tasks (`int`, *optional*): - Cap on the number of tasks pulled into the dataset. `None` uses every task in the tree. - indices (`list[int]`, *optional*): - Specific task indices (into the sorted task list). Mutually exclusive with `num_tasks`. - include_metadata (`bool`, *optional*, defaults to `True`): - Fold per-task `task.toml` metadata (gold_answer, difficulty, ...) into the dataset rows. - """ - - def __init__( - self, - dataset: str, - *, - agent: str | type[HarborEnv] = "bash", - environment_type: str = "docker", - num_tasks: int | None = None, - indices: list[int] | None = None, - include_metadata: bool = True, - ) -> None: - if num_tasks is not None and indices is not None: - raise ValueError("Provide num_tasks or indices, not both.") - self._dataset = dataset - self._environment_type = environment_type - self._num_tasks = num_tasks - self._indices = indices - self._include_metadata = include_metadata - self._env_cls = _resolve_agent(agent) - - # ── public surface ────────────────────────────────────────────── - - @cached_property - def _task_dirs(self) -> list[Path]: - """Resolve the dataset to a sorted list of local task directories (downloading if needed).""" - local = Path(self._dataset) - if (local / "tasks").is_dir(): - root = local / "tasks" - elif local.is_dir() and any(local.glob("*/task.toml")): - root = local - else: - # Treat as an HF dataset repo id; download the task tree. - from huggingface_hub import snapshot_download - - path = Path(snapshot_download(self._dataset, repo_type="dataset", allow_patterns=["tasks/**"])) - root = path / "tasks" - dirs = sorted(p.parent for p in root.glob("*/task.toml")) - if self._indices is not None: - dirs = [dirs[i] for i in self._indices] - elif self._num_tasks is not None: - dirs = dirs[: self._num_tasks] - if not dirs: - raise ValueError(f"No tasks (dir with task.toml) found under {root}") - return dirs - - @cached_property - def train_dataset(self): - """A `datasets.Dataset` of tasks. Plugs into TRL's `train_dataset=`. - - Columns: `prompt` (empty user message — TRL appends the env's instruction from `reset`), `task_dir` (passed to - `reset`), `task_index`, and per-task metadata when `include_metadata`. - """ - from datasets import Dataset - - dirs = self._task_dirs - rows: dict[str, list[Any]] = { - "prompt": [[{"role": "user", "content": ""}] for _ in dirs], - "task_dir": [str(d) for d in dirs], - # task_index is the position in the sorted suite, so it matches the `indices` selector. - "task_index": list(self._indices) if self._indices is not None else list(range(len(dirs))), - } - if self._include_metadata: - metas = [_read_task_meta(d) for d in dirs] - for key in ("gold_answer", "reward_mode", "difficulty_level", "kaggle_dataset"): - rows[key] = [m.get(key) for m in metas] - return Dataset.from_dict(rows) - - @cached_property - def environment_factory(self) -> Callable[[], HarborEnv]: - """Zero-arg callable returning a fresh harness env. Plugs into TRL's `environment_factory=`. - - Returns a `functools.partial` (not a closure) so it stays picklable — `AsyncGRPOTrainer` runs its rollout - worker in a separate process and pickles the factory to it (closures/lambdas would fail). - """ - return partial(self._env_cls, environment_type=self._environment_type) - - @property - def reward_funcs(self) -> Callable[..., list[float]]: - """Default outcome reward (Harbor verifier scalar). Plugs into TRL's `reward_funcs=`.""" - return _outcome_reward_func diff --git a/trl/import_utils.py b/trl/import_utils.py index 79766a6bbe4..b93c1cc257f 100644 --- a/trl/import_utils.py +++ b/trl/import_utils.py @@ -61,10 +61,6 @@ def is_deepspeed_available() -> bool: return _is_package_available("deepspeed") -def is_harbor_available() -> bool: - return _is_package_available("harbor") - - def is_jmespath_available() -> bool: return _is_package_available("jmespath")