Skip to content

fix(model): make model download pause race-safe - #497

Draft
Jinghao-coding wants to merge 1 commit into
raids-lab:mainfrom
Jinghao-coding:fix/md-pause
Draft

fix(model): make model download pause race-safe#497
Jinghao-coding wants to merge 1 commit into
raids-lab:mainfrom
Jinghao-coding:fix/md-pause

Conversation

@Jinghao-coding

Copy link
Copy Markdown
Member

中文说明

背景

关联 Issue:Closes #472

模型下载任务在暂停请求与 Reconciler 检查之间存在竞态:Kubernetes Job 可能已被删除,但数据库状态仍未及时更新,导致暂停中的任务被错误标记为失败。

修改内容

  • 使用条件状态转换,确保只有 Downloading 状态可以转为 Paused
  • 在暂停事务中释放下载配额。
  • 暂停时先持久化 Paused 状态,再删除 Kubernetes Job。
  • Reconciler 遇到 Paused 状态时只清理 Job,不再根据 Job 状态覆盖数据库状态。
  • Job 不存在时重新读取并加锁,仅将 PendingDownloading 任务标记为失败。
  • 增加暂停竞态、暂停任务保留状态和活动任务失败路径的测试。

验证

  • make lint
  • go vet ./pkg/reconciler ./internal/handler
  • 后端 Handler/Reconciler 测试包编译检查
  • Controller 和 migrate 二进制构建检查
  • git diff --check

本地测试二进制未直接执行,因为测试初始化会连接当前配置的 PostgreSQL 环境;完整测试将由 CI 执行。

English Description

Background

Related issue: Closes #472

There is a race between a model download pause request and the Reconciler: the Kubernetes Job may already be deleted while the database state has not been updated, causing a paused download to be incorrectly marked as failed.

Changes

  • Added a conditional state transition so only Downloading downloads can become Paused.
  • Released the download quota within the pause transaction.
  • Persisted the Paused state before deleting the Kubernetes Job.
  • Updated the Reconciler to clean up the Job without overwriting a persisted Paused state.
  • Re-read and lock the record when a Job is missing, and only fail Pending or Downloading downloads.
  • Added tests covering the pause race, paused-state preservation, and active-download failure paths.

Validation

  • make lint
  • go vet ./pkg/reconciler ./internal/handler
  • Handler and Reconciler test-package compilation
  • Controller and migration binary builds
  • git diff --check

The local test binaries were not executed because package initialization connects to the PostgreSQL environment configured for this checkout; the complete test suite will run in CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 暂停模型下载后任务可能被错误标记为失败

1 participant