chore: unify linting and formatting with Ruff

.github/workflows/lint.yml

@@ -26,7 +26,7 @@ jobs:
       - name: Install pre-commit
         run: |
           python -m pip install --upgrade pip
-          pip install pre-commit
+          pip install 'pre-commit==4.6.0'
 
       - name: Run pre-commit
-        run: pre-commit run --all-files
+        run: pre-commit run --all-files --show-diff-on-failure

.pre-commit-config.yaml

@@ -1,38 +1,10 @@
 repos:
-  - repo: https://github.com/pycqa/flake8.git
-    rev: 7.3.0
-    hooks:
-      - id: flake8
-        exclude: |
-            (?x)^(
-                examples/|
-                docs/|
-                tests/|
-                evalscope/utils/utils.py|
-                evalscope/third_party/|
-                evalscope/backend/rag_eval/clip_benchmark/tasks|
-                evalscope/backend/rag_eval/cmteb/tasks|
-                evalscope/metrics/vision/t2v_metrics
-            )
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.16.4
     hooks:
       - id: ruff-check
         args: [--fix]
-  - repo: https://github.com/google/yapf
-    rev: v0.43.0
-    hooks:
-      - id: yapf
-        exclude: |
-            (?x)^(
-                examples/|
-                docs/|
-                tests/|
-                evalscope/utils/utils.py|
-                evalscope/third_party/|
-                evalscope/backend/rag_eval/clip_benchmark/tasks|
-                evalscope/backend/rag_eval/cmteb/tasks
-            )
+      - id: ruff-format
   - repo: https://github.com/pre-commit/pre-commit-hooks.git
     rev: v6.0.0
     hooks:
@@ -44,8 +16,6 @@ repos:
         exclude: evalscope/third_party/|docs/|examples|.*\.json
       - id: requirements-txt-fixer
         exclude: evalscope/third_party/|docs/|examples
-      - id: double-quote-string-fixer
-        exclude: evalscope/third_party/|docs/|examples|.*\.json|cl_bench_adapter.py
       - id: check-merge-conflict
         exclude: evalscope/third_party/|docs/|examples
       - id: mixed-line-ending

AGENTS.md

@@ -14,7 +14,7 @@ Python ≥ 3.10 (3.10 / 3.11 / 3.12). Dependencies: `requirements/framework.txt`
 ## Build, lint, test
 
 ```bash
-make lint                                                                       # required before commit (yapf + Ruff + flake8 + basic pre-commit hooks)
+make lint                                                                       # apply Ruff fixes/formatting and run all pre-commit checks
 pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings        # CI smoke test
 pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s    # perf
 ```
@@ -63,9 +63,10 @@ run_task(TaskConfig(model='Qwen/Qwen2.5-0.5B-Instruct', datasets=['gsm8k'], limi
 ## Code style (enforced)
 
 - **Line width 120**, 4-space indent, LF endings, trailing newline at EOF.
-- **Quotes** governed by `double-quote-string-fixer` hook — follow existing file style; do not mix.
-- **f-strings** for formatting (no `%` or `.format()` unless necessary).
+- **Quotes**: single quotes, enforced by the Ruff formatter.
+- **Linting**: Ruff's `E`, `F`, and `W` rules for maintained source files.
 - **Imports**: Ruff's `I` rules, with `evalscope` detected as first-party and standard import sections.
+- **f-strings** for formatting (no `%` or `.format()` unless necessary).
 - **Type hints required** on every function signature.
 - **English only** for comments and docstrings.
 - **Public APIs need docstrings**; internal helpers only when intent is non-obvious.
@@ -80,7 +81,7 @@ run_task(TaskConfig(model='Qwen/Qwen2.5-0.5B-Instruct', datasets=['gsm8k'], limi
 | Handler function | `handle_` prefix |
 | Benchmark adapter file | `<name>_adapter.py` |
 
-**flake8 ignore list** (`setup.cfg`): `F401, F403, F405, F821, W503, E251, W504, F824, F541, E501, E226, E121-E129, E131, E741`. Do not expand — new ignores must be justified in the PR.
+**Ruff ignore list** (`pyproject.toml`): `E501, E741, F401, F403, F405, F541, F821`. Do not expand — new ignores must be justified in the PR.
 
 ## Design rules
 
@@ -164,6 +165,6 @@ An adapter must **never** call `self.llm_judge.judge()` or parse a judge reply i
 
 ```bash
 make dev      # once
-make lint     # before every commit
+make lint     # apply fixes and run all checks before every commit
 pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings
 ```

CONTRIBUTING.md

@@ -306,18 +306,21 @@ evalscope service
 
 This project uses **pre-commit** with the following hooks:
 
-- **flake8** — Python style checker
-- **Ruff** — Import sorting (`I` rules)
-- **yapf** — Code formatting
-- Trailing whitespace, YAML checks, line ending fixes
+- **Ruff check** — Python linting (`E`, `F`, and `W`) and import sorting (`I`)
+- **Ruff format** — Python code formatting with 120-character lines and single quotes
+- Trailing whitespace, YAML checks, and line ending fixes
+
+Ruff's lint hook runs before its formatter so that any automatic fixes are formatted consistently. Pre-commit is installed by `make dev` with the version pinned in `requirements/dev.txt`.
 
 ```bash
-# Run all checks
+# Apply safe fixes, format maintained Python files, and run all repository checks
 make lint
 # or
 pre-commit run --all-files
 ```
 
+If pre-commit modifies files, review and stage those changes, then run `make lint` again. The configured Ruff scope and exclusions are defined in `pyproject.toml`.
+
 ### Testing
 
 ```bash
@@ -342,9 +345,9 @@ pytest tests/benchmark/test_xxx.py
    git commit -m "feat: add MyBenchmark adapter"
    ```
 
-3. **Run quality checks** before pushing:
+3. **Run quality checks before pushing:**
    ```bash
-   pre-commit run --all-files
+   make lint
    pytest tests/
    ```
 

Makefile

@@ -150,7 +150,6 @@ lint:
 .PHONY: dev
 dev:
 	pip install -e '.[dev,perf,docs]'
-	pip install pre-commit
 
 .PHONY: install
 install:

docs/en/advanced_guides/add_benchmark.md

@@ -611,8 +611,8 @@ make docs
 ```
 
 ## 6. Submitting PR
-After completing the implementation of these methods and document generation, your benchmark evaluation is ready! You can submit a [PR](https://github.com/modelscope/evalscope/pulls). Before submitting, please run the following command, which will automatically format the code:
+After completing the implementation and documentation generation, run all repository checks before submitting a [PR](https://github.com/modelscope/evalscope/pulls). This command applies safe Ruff fixes and formatting before validating the remaining hooks:
 ```bash
 make lint
 ```
-Ensure there are no formatting issues, and we will merge your contribution as soon as possible, allowing more users to use the benchmark evaluation you contributed. If you don't know how to submit a PR, you can check our [Guide](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md). Give it a try 🚀
+Once the checks pass, your contribution is ready for review. For the complete development workflow, see the [Contributing Guide](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md). Give it a try 🚀

docs/zh/advanced_guides/add_benchmark.md

@@ -603,8 +603,8 @@ make docs
 ```
 
 ## 6. 提交PR
-完成这些方法的实现和文档生成后，您的基准评测就准备就绪了！可以提交[PR](https://github.com/modelscope/evalscope/pulls)了。在提交之前请运行如下命令，将自动格式化代码：
+完成实现和文档生成后，请在提交 [PR](https://github.com/modelscope/evalscope/pulls) 前运行仓库的全部检查。该命令会先应用 Ruff 的安全修复和格式化，再验证其余 hooks：
 ```bash
 make lint
 ```
-确保没有格式问题后，我们将尽快合并你的贡献，让更多用户来使用你贡献的基准评测。如果你不知道如何提交PR，可以查看我们的[指南](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md)，快来试一试吧🚀
+检查通过后即可提交评审。完整开发流程请参考[贡献指南](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md)，快来试一试吧🚀

evalscope/agent/environments/enclave.py

@@ -214,8 +214,7 @@ async def _ensure_sandbox(self) -> SandboxHandle:
                 manager_config=self._manager_config or None,
             )
             logger.debug(
-                f'EnclaveAgentEnvironment: sandbox {self._handle.sandbox_id} ready '
-                f'(engine={self._engine.value}).'
+                f'EnclaveAgentEnvironment: sandbox {self._handle.sandbox_id} ready (engine={self._engine.value}).'
             )
         return self._handle
 

evalscope/agent/external/bridge/_sse_common.py

@@ -23,4 +23,4 @@
 def iter_chunks(text: str, max_len: int) -> List[str]:
     """Slice ``text`` into ``max_len`` segments; always yields at least one
     entry (an empty string when ``text`` is empty)."""
-    return [text[i:i + max_len] for i in range(0, len(text), max_len)] or ['']
+    return [text[i : i + max_len] for i in range(0, len(text), max_len)] or ['']

evalscope/agent/external/bridge/server.py

@@ -279,8 +279,8 @@ async def _catchall(request: web.Request) -> web.Response:
                     'type': 'error',
                     'error': {
                         'type': 'not_found',
-                        'message': f'no handler for {request.path}'
-                    }
+                        'message': f'no handler for {request.path}',
+                    },
                 },
                 status=404,
             )
@@ -429,8 +429,8 @@ async def _handle_anthropic_messages(self, request: web.Request) -> web.StreamRe
                     'type': 'error',
                     'error': {
                         'type': 'authentication_error',
-                        'message': str(exc)
-                    }
+                        'message': str(exc),
+                    },
                 },
                 status=401,
             )
@@ -488,10 +488,12 @@ async def _respond_json_openai(
         except Exception as exc:  # pragma: no cover - upstream-dependent
             _log_upstream_failure(session, exc, mode='json')
             return web.json_response(
-                {'error': {
-                    'type': 'api_error',
-                    'message': repr(exc)
-                }},
+                {
+                    'error': {
+                        'type': 'api_error',
+                        'message': repr(exc),
+                    },
+                },
                 status=502,
             )
 
@@ -537,10 +539,13 @@ async def _respond_streaming_openai(
         except Exception as exc:  # pragma: no cover - upstream-dependent
             failure_handled = True
             _log_upstream_failure(session, exc, mode='stream')
-            error_event = (
-                f'data: {json.dumps({"error": {"type": "api_error", "message": repr(exc)}})}\n\n'
-                f'data: [DONE]\n\n'
-            ).encode('utf-8')
+            error_payload = {
+                'error': {
+                    'type': 'api_error',
+                    'message': repr(exc),
+                },
+            }
+            error_event = f'data: {json.dumps(error_payload)}\n\ndata: [DONE]\n\n'.encode('utf-8')
             try:
                 await response.write(error_event)
             except ConnectionResetError:
@@ -592,10 +597,12 @@ async def _respond_json_responses(
         except Exception as exc:  # pragma: no cover - upstream-dependent
             _log_upstream_failure(session, exc, mode='json')
             return web.json_response(
-                {'error': {
-                    'type': 'api_error',
-                    'message': repr(exc)
-                }},
+                {
+                    'error': {
+                        'type': 'api_error',
+                        'message': repr(exc),
+                    },
+                },
                 status=502,
             )
 
@@ -719,11 +726,13 @@ async def _respond_json_gemini(
         except Exception as exc:
             _log_upstream_failure(session, exc, mode='json')
             return web.json_response(
-                {'error': {
-                    'code': 502,
-                    'message': repr(exc),
-                    'status': 'UNAVAILABLE',
-                }},
+                {
+                    'error': {
+                        'code': 502,
+                        'message': repr(exc),
+                        'status': 'UNAVAILABLE',
+                    },
+                },
                 status=502,
             )
 
@@ -797,8 +806,8 @@ async def _respond_json(
                     'type': 'error',
                     'error': {
                         'type': 'api_error',
-                        'message': repr(exc)
-                    }
+                        'message': repr(exc),
+                    },
                 },
                 status=502,
             )
@@ -846,11 +855,14 @@ async def _respond_streaming(
         except Exception as exc:  # pragma: no cover - upstream-dependent
             failure_handled = True
             _log_upstream_failure(session, exc, mode='stream')
-            error_event = (
-                f'event: error\ndata: '
-                f'{json.dumps({"type": "error", "error": {"type": "api_error", "message": repr(exc)}})}'
-                f'\n\n'
-            ).encode('utf-8')
+            error_payload = {
+                'type': 'error',
+                'error': {
+                    'type': 'api_error',
+                    'message': repr(exc),
+                },
+            }
+            error_event = f'event: error\ndata: {json.dumps(error_payload)}\n\n'.encode('utf-8')
             try:
                 await response.write(error_event)
             except ConnectionResetError:
@@ -875,11 +887,13 @@ async def _auth_check_openai(self, request: web.Request) -> 'TrialSession | web.
         except _BridgeAuthError as exc:
             logger.debug(f'bridge: auth failed — {exc}')
             return web.json_response(
-                {'error': {
-                    'type': 'invalid_request_error',
-                    'code': 'invalid_api_key',
-                    'message': str(exc),
-                }},
+                {
+                    'error': {
+                        'type': 'invalid_request_error',
+                        'code': 'invalid_api_key',
+                        'message': str(exc),
+                    },
+                },
                 status=401,
             )
 
@@ -902,7 +916,7 @@ async def _lookup_session(self, request: web.Request) -> TrialSession:
         token = _extract_bearer_token(request)
         if not token or not token.startswith(_TRIAL_TOKEN_PREFIX):
             raise _BridgeAuthError(f'missing or malformed bridge token (expected {_TRIAL_TOKEN_PREFIX}<id>)')
-        trial_id = token[len(_TRIAL_TOKEN_PREFIX):]
+        trial_id = token[len(_TRIAL_TOKEN_PREFIX) :]
         async with self._sessions_lock:
             session = self._sessions.get(trial_id)
         if session is None:
@@ -917,19 +931,21 @@ class _BridgeAuthError(Exception):
 #: Exception class names treated as "upstream business error" (rate
 #: limit, auth, model-side failure). Matched by class name so we don't
 #: take a hard dependency on the ``anthropic`` package at import time.
-_UPSTREAM_BUSINESS_ERRORS = frozenset({
-    'APIError',
-    'APIStatusError',
-    'APIConnectionError',
-    'APITimeoutError',
-    'RateLimitError',
-    'AuthenticationError',
-    'PermissionDeniedError',
-    'NotFoundError',
-    'BadRequestError',
-    'UnprocessableEntityError',
-    'InternalServerError',
-})
+_UPSTREAM_BUSINESS_ERRORS = frozenset(
+    {
+        'APIError',
+        'APIStatusError',
+        'APIConnectionError',
+        'APITimeoutError',
+        'RateLimitError',
+        'AuthenticationError',
+        'PermissionDeniedError',
+        'NotFoundError',
+        'BadRequestError',
+        'UnprocessableEntityError',
+        'InternalServerError',
+    }
+)
 
 
 def _log_upstream_failure(session: 'TrialSession', exc: BaseException, *, mode: str) -> None: