feat(query-coordinator): Add query job lifecycle handling - #2513
feat(query-coordinator): Add query job lifecycle handling#2513Bill-hbrhbr wants to merge 55 commits into
Conversation
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
…env` into a shared `task::utils` module.
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
LinZhihao-723
left a comment
LinZhihao-723
left a comment
There was a problem hiding this comment.
You should ask codex/claude to do an adversay code review: instead of telling it to code by following the guideline from https://github.com/LinZhihao-723/claude-instruction/tree/main/rust, ask it to:
- The coding subagent programs by following the guideline.
- The review subagent reviews the code by checking against the guideline.
That should bring the delivered code much closer to the desirable shape.
| # Upgrade query-job tables created before Spider lifecycle support was added. | ||
| query_jobs_table_upgrades = ( | ||
| ( | ||
| f""" | ||
| ALTER TABLE `{QUERY_JOBS_TABLE_NAME}` | ||
| ADD COLUMN `status_msg` VARCHAR(512) NOT NULL DEFAULT '' AFTER `status` | ||
| """, | ||
| ER_DUP_FIELDNAME, | ||
| ), | ||
| ( | ||
| f""" | ||
| ALTER TABLE `{QUERY_JOBS_TABLE_NAME}` | ||
| ADD COLUMN `spider_id` BIGINT UNSIGNED NULL DEFAULT NULL AFTER `job_config` | ||
| """, | ||
| ER_DUP_FIELDNAME, | ||
| ), | ||
| ( | ||
| f""" | ||
| ALTER TABLE `{QUERY_JOBS_TABLE_NAME}` | ||
| ADD INDEX `JOB_SPIDER_ID` (`spider_id`) USING BTREE | ||
| """, | ||
| ER_DUP_KEYNAME, | ||
| ), | ||
| ) | ||
| for upgrade_query, duplicate_error_code in query_jobs_table_upgrades: | ||
| try: | ||
| scheduling_db_cursor.execute(upgrade_query) | ||
| except Exception as err: | ||
| if not (hasattr(err, "errno") and err.errno == duplicate_error_code): | ||
| raise |
There was a problem hiding this comment.
Similar to compression job table: we do not need to support backward compatibility.
| /// Drives one already-planned query job through submission and terminal persistence. | ||
| pub struct QueryJobHandle<SubmitterType: QueryJobSubmitter> { |
There was a problem hiding this comment.
Type parameter docstring is missing.
| } | ||
|
|
||
| impl<SubmitterType: QueryJobSubmitter> QueryJobHandle<SubmitterType> { | ||
| /// Constructs a handle for an already-planned query job. |
There was a problem hiding this comment.
This docstring doesn't match the convention for factories.
| result | ||
| } | ||
|
|
||
| async fn submit_and_wait(&self) -> Result<(), Error> { |
There was a problem hiding this comment.
I will stop reading since the docstrings are missing for all private methods.
| /// # Errors | ||
| /// | ||
| /// Returns an error if submission, submission persistence, polling, or terminal persistence | ||
| /// fails. |
There was a problem hiding this comment.
The docstring does not match the coding guideline.
| /// # Errors | ||
| /// | ||
| /// Returns an error if polling or terminal persistence fails. |
There was a problem hiding this comment.
The docstring does not match the coding guideline.
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
2 participants
Description
Add the query-job lifecycle foundation to the query coordinator:
QueryJobHandleand a coordinator-preparedQueryPlan.spider_idandstatus_msgquery-job columns, including idempotent upgrades for existing tables.Spider task-graph construction remains intentionally deferred;
submit_query_jobretains its existing skeleton.Checklist
breaking change.
Validation performed
Not run yet. This is a draft PR, and repository checks are deferred until it is prepared for review.