Skip to content

fix(knowledge): accept SQL uploads and route them through MarkItDown - #3499

Open
dvd233 wants to merge 3 commits into
Tencent:mainfrom
dvd233:codex/fix/docreader-sql-text-3494
Open

dvd233 wants to merge 3 commits into
Tencent:mainfrom
dvd233:codex/fix/docreader-sql-text-3494

Conversation

@dvd233

@dvd233 dvd233 commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Accept SQL files throughout the knowledge-upload path and parse them as text with the existing MarkItDown engine. The Go upload whitelist previously rejected .sql before DocReader was reached; DocReader also lacked the SQL routing entry.

  • Add SQL to the shared Go upload/import whitelist and the frontend fallback whitelist used before parser metadata loads.
  • Route empty or builtin engine requests for SQL through MarkItDown and allow explicit MarkItDown requests.
  • Cover the upload service through storage and document-task enqueueing, dynamic frontend whitelist behavior, parser routing and actual SQL text extraction.

SQL statements are read as document text, not executed.

Type of Change

  • Bug fix
  • Test
  • Breaking change

Related Issue

Fixes #3494

Testing

  • Focused Go upload/import regressions: 45 tests and subtests passed after merging main bccb4b1 (Go 1.26.5, desktop build tag, GCC 15 UCRT toolchain), including .sql and .SQL uploads into DocumentProcess without a spreadsheet-summary task.
  • Frontend file-type regressions: 4 passed (including the current-main XMind regression); changed utility strict TypeScript check passed.
  • Python parser routing and actual SQL text extraction: 10 passed.
  • Changed-scope golangci-lint 2.13.2: 0 issues (CI uses 2.12.2); gofmt and git diff --check passed.
  • The earlier full DocReader run exercised 199 tests with one environment error because the Playwright WebKit executable was absent. It was not rerun for the Go/frontend whitelist follow-up. Full frontend vue-tsc --build and a deployed HTTP/gRPC/database end-to-end environment were not run locally.

Checklist

  • Changes are limited to SQL file admission, parser routing and regressions.
  • Changed source files are formatted and targeted tests pass.
  • Local verification limits are documented above.
  • Self-reviewed the final diff and verification evidence.
  • Breaking changes (none).

This PR was prepared with AI assistance. Screenshots are not applicable to this file-admission and parser-routing change.

Copilot AI lite review requested due to automatic review settings September 21, 2026 02:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@lyingbug

Copy link
Copy Markdown
Collaborator

Go 侧的上传白名单里没有 sql,上传直接被拒,这个改动端到端不起作用

@dvd233 dvd233 changed the title fix(docreader): route SQL files through MarkItDown fix(knowledge): accept SQL uploads and route them through MarkItDown Sep 29, 2026
@dvd233

dvd233 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

已在 8fd9535 补齐 Go 共享上传/导入白名单中的 sql,并同步前端 parser 元数据尚未加载时的默认白名单。新增回归从 CreateKnowledgeFromFile 实际入口验证 .sql / .SQL 可保存并进入 DocumentProcess 队列,同时保留 Python 的实际 SQL 文本提取测试。Go 相关 44 项测试/子用例、前端 3 项、Python 10 项通过,差异范围 golangci-lint 为 0 issues。SQL 仅作为文本知识文档解析,不执行其中的语句。

@dvd233

dvd233 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Current-head CI follow-up for cd674830: the DocReader job is cancelled, not passed. In run 36898338243, dependency installation and Python compilation succeeded, but Install Playwright WebKit spent almost the entire 30-minute job budget installing system packages. The Python and Go client test steps were never reached. Other applicable current-head checks succeeded.

The completed local frontend, parser and Go checks remain recorded in the test plan; they do not replace this incomplete integration job. Could a maintainer rerun the cancelled DocReader job? I have no upstream Actions write permission and have not changed assertions or workflow time limits in this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: docreader supports .sql text format

3 participants