diff --git a/.ai-agents/skills/inkless-changelog/SKILL.md b/.ai-agents/skills/inkless-changelog/SKILL.md new file mode 100644 index 00000000000..fbbe563c177 --- /dev/null +++ b/.ai-agents/skills/inkless-changelog/SKILL.md @@ -0,0 +1,73 @@ +--- +name: inkless-changelog +description: Generate or update the Inkless changelog and GitHub release notes for an Inkless release increment. Diffs conventional-commit history and the auto-generated configs.rst/metrics.rst between two inkless-release tags, categorizes changes, and produces a detailed changelog entry plus a curated release-notes summary. Use when cutting an Inkless release, refreshing docs/inkless/CHANGELOG.md, or preparing GitHub release notes for an inkless-release- tag. +--- + +# Inkless Changelog & Release Notes + +Produce two artifacts for an Inkless increment (`inkless-release-` -> `inkless-release-`): + +1. A **detailed changelog entry** added to `docs/inkless/CHANGELOG.md` (newest first; see the placement step below). +2. A **curated release-notes summary** for the GitHub Release of `inkless-release-`. + +Both derive from the same sources; the summary is a filtered subset of the detailed entry. + +## Sources and their reliability + +| Source | Command | Reliability | +| --- | --- | --- | +| Commits | `git log --first-parent --no-merges` between the two tags | High. `--first-parent` excludes upstream commits dragged in by `apache/kafka` merge commits, leaving inkless PR squash-merges. | +| Config changes | diff of `docs/inkless/configs.rst` between tags | High. Config keys are stable. | +| Metric changes | diff of `docs/inkless/metrics.rst` between tags | Low. `metrics.rst` is produced by a hand-maintained registry list in `MetricsDocs.main()`, so a delta may be documentation catch-up, not a shipped metric (e.g. the 3->16 mbean jump at 0.39). ALWAYS curate. | +| Upstream sync | `apache/kafka` merge commits in range + `gradle.properties` version delta | High. Emitted as a blockquote note under the heading when `main`'s Kafka base moves (e.g. 4.1.0 -> 4.2.0-SNAPSHOT at 0.35). | + +## Steps + +1. **Locate the repo and tags.** Run from the inkless worktree. Confirm the target tag exists: + ```bash + git tag | grep '^inkless-release-' | sort -V | tail -5 + ``` + +2. **Generate the draft** with the helper (run from repo root): + ```bash + .ai-agents/skills/inkless-changelog/gen-changelog.py --version + ``` + Options: + - `--version ` -> uses `inkless-release-..inkless-release-`. + - `--from --to ` -> explicit range. + - no args -> latest two `inkless-release-*` tags. + - `--summary` -> emit only the curated release-notes summary (features + fixes + config changes). + +3. **Curate the metrics block.** The draft marks metric deltas with a + `` comment. Cross-check each entry against the + feature/fix commits in the same range. Keep metrics that a commit introduced; + collapse documentation catch-up into a short note (see the 0.39 entry in + `CHANGELOG.md` as the reference pattern). Remove the REVIEW comment before publishing. + +4. **Prepend the entry** to `docs/inkless/CHANGELOG.md` directly under the + `---` separator (newest first). Keep the Kafka-version list in the heading + accurate: + ```bash + git tag | grep -E "^inkless-4\.[0-9]+\.[0-9]+-$" + ``` + +5. **Produce the GitHub release notes** from `--summary` output. Drop + `chore`/`test`/`docs`/`refactor`; keep features, fixes, and config/metric + changes an operator cares about. Tighten wording; strip PR numbers only if + the release UI already links them. + +## Constraints + +- The only file this skill writes in the repo is `docs/inkless/CHANGELOG.md`. + Release notes are handed to the operator / GitHub Release UI, not committed. +- Do not invent metric or config names; every entry must trace to a commit or a + `configs.rst`/`metrics.rst` diff. +- Never publish raw metric deltas without the curation step. + +## Related + +- `docs/inkless/RELEASES.md` -- release process; the "Cutting a Release" section references this skill. +- `docs/inkless/VERSIONING-STRATEGY.md` -- what an increment is. +- `storage/inkless/src/main/java/io/aiven/inkless/doc/MetricsDocs.java` -- the + registry list behind `metrics.rst`; incomplete coverage here is the root cause + of noisy metric deltas. diff --git a/.ai-agents/skills/inkless-changelog/gen-changelog.py b/.ai-agents/skills/inkless-changelog/gen-changelog.py new file mode 100755 index 00000000000..3b436fe17da --- /dev/null +++ b/.ai-agents/skills/inkless-changelog/gen-changelog.py @@ -0,0 +1,277 @@ +#!/usr/bin/env python3 +"""Generate an Inkless changelog draft between two inkless-release tags. + +Sources, in order of reliability: + 1. Conventional-commit log (backbone) -- categorized by type/scope. + 2. configs.rst delta -- RELIABLE (config keys are stable). + 3. metrics.rst delta -- NOISY, marked REVIEW (see caveat below). + +Caveat on metrics: docs/inkless/metrics.rst is produced by MetricsDocs.main(), +a hand-maintained list of metric registries. When a registry is added to that +list, its metrics appear as "added" in the diff even though the code shipped +earlier (e.g. the 3->16 mbean jump at 0.39 was backfilled tooling, not 13 new +metrics). Always curate the metrics block before publishing. + +Usage: + gen-changelog.py # latest two inkless-release tags + gen-changelog.py --version 0.44 # 0.43 -> 0.44 + gen-changelog.py --from inkless-release-0.43 --to inkless-release-0.44 +""" +import argparse +import re +import subprocess +import sys + +TYPE_LABELS = [ + ("feat", "Features"), + ("fix", "Fixes"), + ("refactor", "Refactors"), + ("test", "Tests"), + ("docs", "Docs"), + ("chore", "Chores"), +] +# Types surfaced in the curated GH-release-notes summary (feat/fix only). +SUMMARY_TYPES = {"feat", "fix"} + + +def git(*args): + r = subprocess.run(["git", *args], capture_output=True, text=True) + if r.returncode != 0: + if r.stderr.strip(): + print(f"warning: git {' '.join(args)}: {r.stderr.strip()}", file=sys.stderr) + return None + return r.stdout + + +def release_tags(): + out = git("tag") or "" + tags = [t for t in out.split() if re.match(r"^inkless-release-0\.\d+$", t)] + return sorted(tags, key=lambda t: int(t.rsplit(".", 1)[-1])) + + +def kafka_base_tags(version): + out = git("tag") or "" + tags = [t for t in out.split() + if re.match(rf"^inkless-4\.\d+\.\d+-{re.escape(version)}$", t)] + # extract the kafka version portion "4.1.2" + return sorted({t[len("inkless-"):-(len(version) + 1)] for t in tags}) + + +def show(tag, path): + return git("show", f"{tag}:{path}") + + +def gradle_version(tag): + """Return (kafka_base, scala) from gradle.properties at tag. + + The version string is like `4.2.0-inkless-SNAPSHOT`; `-inkless` is stripped so + the Kafka base reads `4.2.0-SNAPSHOT`. + """ + text = show(tag, "gradle.properties") or "" + ver = scala = None + for line in text.splitlines(): + if line.startswith("version="): + ver = line[len("version="):].strip().replace("-inkless", "") + elif line.startswith("scalaVersion="): + scala = line[len("scalaVersion="):].strip() + return ver, scala + + +def upstream_syncs(frm, to): + """Merge commits in range that merged apache/kafka trunk (an upstream sync).""" + out = git("log", "--merges", "--format=%s", f"{frm}..{to}") or "" + return [s.strip() for s in out.splitlines() + if re.search(r"apache/kafka|sync/upstream", s) or s.startswith("merge: apache")] + + +def upstream_note(frm, to): + """One-line note on upstream syncs / main base version move in the range, or None.""" + fv, fs = gradle_version(frm) + tv, ts = gradle_version(to) + synced = bool(upstream_syncs(frm, to)) + if tv and fv and tv != fv: + note = f"Upstream sync: main development base moved to Kafka {tv} (from {fv})" + if ts and fs and ts != fs: + note += f", Scala {fs} -> {ts}" + return note + "." + if synced: + return "Upstream sync: merged apache/kafka trunk (no base version change)." + return None + + +def parse_configs(text): + keys, prefix = set(), "" + if not text: + return keys + for line in text.splitlines(): + m = re.match(r"^Under ``([^`]*)``", line) + if m: + prefix = m.group(1) + continue + m = re.match(r"^``([^`]+)``\s*$", line) + if m: + keys.add(prefix + m.group(1)) + return keys + + +def parse_metrics(text): + mbeans, attrs, cur = set(), set(), None + if not text: + return mbeans, attrs + for line in text.splitlines(): + m = re.match(r"^(io\.aiven\.inkless[^\s]+)", line) + if m: + cur = m.group(1) + mbeans.add(cur) + continue + m = re.match(r"^([A-Za-z][a-zA-Z0-9\-\._]+)\s{2,}\S", line) + if m and cur: + attrs.add(f"{cur} :: {m.group(1)}") + return mbeans, attrs + + +def parse_commit(subject): + """Return (type, scope, description) for a conventional commit, else None.""" + m = re.match(r"^(\w+)(?:\(([^)]*)\))?!?:\s*(.+)$", subject) + if not m: + return None + return m.group(1).lower(), (m.group(2) or "").lower(), m.group(3).strip() + + +def collect_commits(frm, to): + # --first-parent follows only the mainline of PR squash-merges, excluding the + # thousands of individual upstream commits pulled in by apache/kafka merge commits. + out = git("log", "--first-parent", "--no-merges", "--format=%s", f"{frm}..{to}") or "" + buckets = {t: [] for t, _ in TYPE_LABELS} + other = [] + for line in out.splitlines(): + line = line.strip() + if not line: + continue + parsed = parse_commit(line) + if not parsed: + continue # skip Merge/KAFKA-/MINOR upstream noise + typ, scope, desc = parsed + # keep only inkless-relevant scopes; drop pure sync bookkeeping + if scope.startswith("sync") or scope == "sync": + continue + entry = (scope, desc) + if typ in buckets: + buckets[typ].append(entry) + else: + other.append(entry) + return buckets, other + + +def fmt_entry(scope, desc): + return f"- {'(' + scope + ') ' if scope else ''}{desc}" + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--version") + ap.add_argument("--from", dest="frm") + ap.add_argument("--to") + ap.add_argument("--summary", action="store_true", + help="emit only the curated GH-release-notes summary (feat/fix + config " + "changes; metric deltas are omitted -- too noisy to auto-publish, see " + "module docstring)") + args = ap.parse_args() + + tags = release_tags() + if args.frm and args.to: + frm, to = args.frm, args.to + elif args.version: + to = f"inkless-release-{args.version}" + if to not in tags: + ap.error(f"tag {to} not found; known: {', '.join(tags) or '(none)'}") + idx = tags.index(to) + if idx == 0: + ap.error(f"{to} is the earliest inkless-release tag; nothing precedes it") + frm = tags[idx - 1] + else: + if len(tags) < 2: + ap.error(f"need at least two inkless-release tags, found {len(tags)}") + frm, to = tags[-2], tags[-1] + + version = to.rsplit("-", 1)[-1] + kafka = kafka_base_tags(version) + + ck_add = sorted(parse_configs(show(to, "docs/inkless/configs.rst")) + - parse_configs(show(frm, "docs/inkless/configs.rst"))) + ck_rm = sorted(parse_configs(show(frm, "docs/inkless/configs.rst")) + - parse_configs(show(to, "docs/inkless/configs.rst"))) + mb_to, ma_to = parse_metrics(show(to, "docs/inkless/metrics.rst")) + mb_fr, ma_fr = parse_metrics(show(frm, "docs/inkless/metrics.rst")) + mb_add, mb_rm = sorted(mb_to - mb_fr), sorted(mb_fr - mb_to) + ma_add, ma_rm = sorted(ma_to - ma_fr), sorted(ma_fr - ma_to) + + buckets, other = collect_commits(frm, to) + + p = print + header = f"## {version}" + if kafka: + header += f" (Kafka {', '.join(kafka)})" + p(header) + p("") + + note = upstream_note(frm, to) + if note: + p(f"> {note}") + p("") + + if args.summary: + for typ, label in TYPE_LABELS: + if typ not in SUMMARY_TYPES: + continue + if buckets[typ]: + p(f"### {label}") + for scope, desc in buckets[typ]: + p(fmt_entry(scope, desc)) + p("") + if ck_add or ck_rm: + p("### Configuration") + for c in ck_add: + p(f"- Added `{c}`") + for c in ck_rm: + p(f"- Removed `{c}`") + p("") + return + + for typ, label in TYPE_LABELS: + if buckets[typ]: + p(f"### {label}") + for scope, desc in buckets[typ]: + p(fmt_entry(scope, desc)) + p("") + if other: + p("### Other") + for scope, desc in other: + p(fmt_entry(scope, desc)) + p("") + + p("### Config & metric changes") + if ck_add or ck_rm: + for c in ck_add: + p(f"- config added: `{c}`") + for c in ck_rm: + p(f"- config removed: `{c}`") + else: + p("- no config changes") + if mb_add or mb_rm or ma_add or ma_rm: + p("") + p("") + for m in mb_add: + p(f"- metric mbean added: `{m}`") + for m in mb_rm: + p(f"- metric mbean removed: `{m}`") + for a in ma_add: + p(f"- metric attr added: `{a}`") + for a in ma_rm: + p(f"- metric attr removed: `{a}`") + p("") + + +if __name__ == "__main__": + main() diff --git a/.ai-agents/skills/inkless-release-prep/SKILL.md b/.ai-agents/skills/inkless-release-prep/SKILL.md new file mode 100644 index 00000000000..63ba2dcd108 --- /dev/null +++ b/.ai-agents/skills/inkless-release-prep/SKILL.md @@ -0,0 +1,188 @@ +--- +name: inkless-release-prep +description: Prepare a new Inkless increment by aligning the active release branches (inkless-4.1, inkless-4.2, ...) with the inkless commits on main. Covers checking which inkless commits are missing from a release branch, cherry-picking (backporting) them main->release, and creating a new release branch for a new Kafka minor. This is the manual prep that precedes the automated GitHub release. Use when asked to prepare/cut an inkless increment, backport or cherry-pick inkless commits to release branches, check release-branch consistency with main, or create a new inkless-4.x release branch. For merging upstream apache/kafka changes, use the inkless-upstream-sync skill instead. +--- + +# Inkless Release Prep + +Assemble a new Inkless increment (`inkless-release-`). Inkless features land on +`main` first; before a release, each active release branch must be brought level +by cherry-picking the missing inkless commits. This skill covers that prep; the +release itself (tags, Docker images, binaries, GitHub Release) is automated -- see +[RELEASES.md](../../../docs/inkless/RELEASES.md). + +The tooling lives in [`inkless-sync/`](../../../inkless-sync/). This skill is the +agent entry point; run scripts from the repo root. + +## Worktree layout + +Prep runs across dedicated git worktrees, one per branch you touch. The active +set is `main` plus each active release branch (currently 4.1 and 4.2 -- released +minors like 4.0 are excluded): + +| Worktree | Branch | Role | +| ------------------ | ------------- | ---------------------------------------- | +| `../inkless-main` | `main` | Cherry-pick source (features land here). | +| `../inkless-4.1` | `inkless-4.1` | Active release branch (pick target). | +| `../inkless-4.2` | `inkless-4.2` | Active release branch (pick target). | + +Run `cherry-pick-to-release.sh ` from the worktree that already has +`` checked out. The script checks out the target branch if HEAD isn't +already on it (`cherry-pick-to-release.sh:294`); using the matching worktree +avoids branch-switching churn in a shared checkout and the "branch already +checked out in another worktree" error, and keeps `main` available in +`../inkless-main` as the pick source. + +### Prepare/update worktrees (pre-req) + +Before cherry-picking, ensure every active worktree exists and is level with +`origin`. Offer to create missing ones and fast-forward the rest -- the tooling +diffs `origin/*` refs, so a stale local branch yields misleading results. + +```bash +git fetch origin --prune + +# branch -> worktree (one per active branch) +update_worktree() { + local branch="$1" wt="$2" + if [ -d "$wt" ]; then + git -C "$wt" merge --ff-only "origin/$branch" # update existing + else + git worktree add "$wt" "$branch" # create missing + fi +} + +update_worktree main ../inkless-main +update_worktree inkless-4.1 ../inkless-4.1 +update_worktree inkless-4.2 ../inkless-4.2 +``` + +If a worktree has local commits that block a fast-forward, stop and surface it +rather than forcing the update. + +| Step | Script | Guide | +| --- | --- | --- | +| Which inkless commits are missing from a release branch? | `branch-consistency.sh` | -- | +| Backport (cherry-pick) missing commits main->release | `cherry-pick-to-release.sh` | [CHERRY-PICK-SYNC-GUIDE.md](../../../inkless-sync/CHERRY-PICK-SYNC-GUIDE.md) | +| Create a release branch for a new Kafka minor | `create-release-branch.sh` | -- | + +Conflict handling: [CONFLICT-RESOLUTION-STRATEGY.md](../../../inkless-sync/CONFLICT-RESOLUTION-STRATEGY.md) +plus the divergence tables in the cherry-pick guide. Full tooling reference: +[inkless-sync/README.md](../../../inkless-sync/README.md). + +### Aligning back to branch expectations (cherry-pick + revert) + +Some main commits should not take effect on a release branch (e.g. a JDK/CI change +scoped to a newer Kafka line -- 4.1 mirrors apache/4.1 on JDK 23, 4.2 on JDK 25). +Do NOT skip them with a side list. Instead keep the decision in history: + +1. **Cherry-pick the commit** so the branch stays in sync (it is then "present" by + PR number for `branch-consistency.sh`). +2. **Add a follow-up commit that reverts or adjusts it** back to the branch's + expectation, prefixed `sync(revert):` (clean revert) or `sync(align):` (partial + adjust). Explain why in the message. + +`sync(...)` commits are excluded by `branch-consistency.sh` (`is_excluded_commit`), +so they are never treated as missing elsewhere and never re-picked. Net result: the +branch reads as in sync, and the rollback is explicit and auditable -- no hidden +skip file. Example: `sync(revert): keep JDK 23 on inkless-4.1 (revert #502; 4.1 +mirrors apache/4.1)`. + +For the specific case of version strings (`gradle.properties`/`Makefile`/`.env`), +`cherry-pick-to-release.sh` already automates this: it restores version-owned files +to the branch value after every pick, so version-bump commits need no manual revert. + +## Increment workflow + +0. **Log the previous increment** (if not already done). The changelog entry is + diffed from `inkless-release-*` tags, so the just-released increment can only + be written *after* its tag exists -- i.e. at the start of the next prep. + Generate it and land it on `main` via a normal PR (the increment number lives + only in tags, never in `gradle.properties`): + ```bash + .ai-agents/skills/inkless-changelog/gen-changelog.py --version + # prepend the output under the '---' in docs/inkless/CHANGELOG.md, then PR to main + ``` + See the [`inkless-changelog`](../inkless-changelog/SKILL.md) skill for curation. + +1. **Find what's missing** on each active release branch: + ```bash + ./inkless-sync/branch-consistency.sh inkless-4.1 --missing + ``` +2. **Track progress** in a session file: + ```bash + cp inkless-sync/CHERRY-PICK-SESSION-TEMPLATE.md .inkless-sync/CHERRY-PICK-SESSION-$(date +%Y-%m-%d).md + ``` +3. **Cherry-pick** (oldest-first; interactive prompts before each commit). Run + from that branch's worktree (e.g. `../inkless-4.1`), not `../inkless-main`: + ```bash + ./inkless-sync/cherry-pick-to-release.sh inkless-4.1 --dry-run + ./inkless-sync/cherry-pick-to-release.sh inkless-4.1 + ``` +4. **Resolve conflicts** using the divergence tables in the cherry-pick guide + (main vs release-branch API differences: `TopicPartition` vs `TopicIdPartition`, + share coordinator presence, `Option` vs `Optional`, etc.). +5. **Compile per commit** (fast path clean picks, full path after conflicts): + ```bash + ./gradlew :storage:inkless:compileJava :core:compileScala + ``` + Run a full `./gradlew compileJava compileScala compileTestJava compileTestScala` + every 5-10 picks to catch cross-module regressions. +6. **Verify:** `make build` then `make test`. +7. **Archive the session:** + ```bash + mv .inkless-sync/CHERRY-PICK-SESSION-*.md inkless-sync/sessions/ + ``` + +If branches diverge such that a commit cannot be cleanly backported, record it in +the session file and defer it rather than forcing it (branches may end up one +increment apart -- see VERSIONING-STRATEGY.md). + +## Creating a new release branch + +When Apache ships a new minor (e.g. 4.3.0): + +```bash +./inkless-sync/create-release-branch.sh 4.3 --dry-run +./inkless-sync/create-release-branch.sh 4.3 +``` + +Then set the release version by merging the upstream tag with the +`inkless-upstream-sync` skill (`release-sync.sh inkless-4.3 --to-tag 4.3.0`). + +## Then: push, release, changelog + +Prep ends once branches build+test clean. The rest is the release ceremony, +which is automated -- do NOT create tags by hand (the workflow owns tag +creation, version math, and pushing). Order matters: + +1. **Push the release branches** to `origin` (`git push origin inkless-4.1`, + etc.). This is the gate: the release workflow validates against `origin/*` + and aborts if a branch is behind. Nothing is released until branches are + pushed. +2. **Trigger the release** per [RELEASES.md](../../../docs/inkless/RELEASES.md) + (GitHub Actions -> Inkless Release). It validates, creates+pushes tags, builds + images/binaries, and publishes the GitHub Release. +3. **Changelog** is generated FROM the new `inkless-release-` tag, so it + happens *after* the tag exists -- not during prep. The release workflow now + auto-injects the curated summary into the Release body via the + [`inkless-changelog`](../inkless-changelog/SKILL.md) generator. The detailed + `docs/inkless/CHANGELOG.md` entry is committed to `main` via a normal PR -- + done at the START of the next increment (step 0 above), since the tag must + exist first. + +## Guardrails + +- Cherry-pick in batches; compile between batches. Reset and defer a commit whose fix is non-trivial. +- Keep cherry-picked commit messages intact; add `sync(compile):` / `sync(test):` commits (or amend) for adaptation fixes. +- `branch-consistency.sh` is also the gate the release CI uses -- a branch must report zero missing commits before release. +- **Versions are set only by upstream syncs.** A release branch's version string + (`gradle.properties` `version=`, `Makefile` `VERSION`, docker `.env` + `KAFKA_VERSION`) is owned by `release-sync.sh` (`--to-tag`), never by a + cherry-pick from main. A main version-bump commit (e.g. `MINOR: update Kafka + version variables`) still gets cherry-picked so the branch stays in sync, but + `cherry-pick-to-release.sh` restores those version-owned files to the branch's + value afterward (the bump can apply with no conflict, so this runs after every + pick, not just on conflict). Net effect: the commit lands (present for + branch-consistency) without changing the branch's output version. For other + branch-specific divergences, use the cherry-pick + `sync(revert):` pattern above. diff --git a/.ai-agents/skills/inkless-upstream-sync/SKILL.md b/.ai-agents/skills/inkless-upstream-sync/SKILL.md new file mode 100644 index 00000000000..661e33ee054 --- /dev/null +++ b/.ai-agents/skills/inkless-upstream-sync/SKILL.md @@ -0,0 +1,81 @@ +--- +name: inkless-upstream-sync +description: Keep the Inkless fork current with upstream Apache Kafka by merging upstream changes. Covers main sync (merge apache/kafka trunk into main) and release sync (merge an upstream patch release such as 4.1.2 into an inkless-4.x release branch), plus checking how far behind a branch is. Use when asked to sync main with apache/kafka trunk, merge upstream, catch a release branch up to a Kafka patch release, or check how far behind upstream a branch is. For backporting inkless commits to release branches or cutting an increment, use the inkless-release-prep skill instead. +--- + +# Inkless Upstream Sync + +Merge upstream Apache Kafka into the fork. Two workflows, both merge-based (never +rebase), following [VERSIONING-STRATEGY.md](../../../docs/inkless/VERSIONING-STRATEGY.md). + +The tooling lives in [`inkless-sync/`](../../../inkless-sync/) (scripts, guides, +`lib/common.sh`, session templates, archived sessions). This skill is the agent +entry point; run scripts from the repo root. + +| Workflow | When | Script | Guide | +| --- | --- | --- | --- | +| Main sync | Catch `main` up to apache/kafka trunk (every 2-3 months, or before a new Kafka minor) | `main-sync.sh` | [MAIN-SYNC-ACTION-PLAN.md](../../../inkless-sync/MAIN-SYNC-ACTION-PLAN.md) | +| Release sync | Merge an upstream patch (e.g. 4.1.2) into `inkless-4.1` | `release-sync.sh` | [RELEASE-SYNC-GUIDE.md](../../../inkless-sync/RELEASE-SYNC-GUIDE.md), [RELEASE-SYNC-ACTION-PLAN.md](../../../inkless-sync/RELEASE-SYNC-ACTION-PLAN.md) | +| Status check | How far behind upstream is a branch? | `sync-status.sh` | -- | + +Conflict handling for both: [CONFLICT-RESOLUTION-STRATEGY.md](../../../inkless-sync/CONFLICT-RESOLUTION-STRATEGY.md). +Full tooling reference: [inkless-sync/README.md](../../../inkless-sync/README.md). + +## Prerequisite + +```bash +git remote add apache https://github.com/apache/kafka.git # once +git fetch apache +``` + +## Main sync + +```bash +./inkless-sync/main-sync.sh --dry-run # preview +./inkless-sync/main-sync.sh # to apache/trunk HEAD +./inkless-sync/main-sync.sh --before-version 4.3 +``` + +The script fetches upstream, creates a `sync/upstream-YYYYMMDD` branch, merges, +and categorizes conflicts: OWNED inkless paths auto-resolve "ours"; INTERLEAVED +files stop for manual review. Follow the action plan's file-by-file playbook. + +## Release sync + +```bash +./inkless-sync/release-sync.sh inkless-4.1 --list-tags +./inkless-sync/release-sync.sh inkless-4.1 --to-tag 4.1.2 --dry-run +./inkless-sync/release-sync.sh inkless-4.1 --to-tag 4.1.2 +``` + +Resolve version-file conflicts with the `{upstream_version}-inkless` pattern +(keep upstream versions in `streams/quickstart` POMs). See the guide. + +## Status check + +```bash +./inkless-sync/sync-status.sh main +./inkless-sync/sync-status.sh --all +``` + +## Every sync + +1. Track progress in a session file (templates in `inkless-sync/`): + ```bash + cp inkless-sync/MAIN-SYNC-SESSION-TEMPLATE.md .inkless-sync/SESSION-$(date +%Y-%m-%d).md + cp inkless-sync/RELEASE-SYNC-SESSION-TEMPLATE.md .inkless-sync/RELEASE-SESSION--$(date +%Y-%m-%d).md + ``` +2. Resolve conflicts per the strategy doc; show diffs and wait for approval on + INTERLEAVED files before committing. +3. Verify: `make build` then `make test`. +4. Commit with sync prefixes (`merge:`, `sync(compile):`, `sync(test):`, `sync(config):`, `sync(verify):`). +5. Archive the session: + ```bash + mv .inkless-sync/SESSION-*.md inkless-sync/sessions/ + ``` + +## Guardrails + +- Merge, never rebase. Commit incrementally; keep behavioral and test-only fixes separate. +- When restoring a file upstream removed for inkless reasons, restore its test too and add an `INKLESS NOTE` comment. +- If blocked on a conflict, record it in the session file and ask rather than forcing it. diff --git a/.github/workflows/inkless-publish.yml b/.github/workflows/inkless-publish.yml index ebaadca1c66..84003d21768 100644 --- a/.github/workflows/inkless-publish.yml +++ b/.github/workflows/inkless-publish.yml @@ -306,6 +306,9 @@ jobs: uses: actions/checkout@v4 with: ref: inkless-release-${{ needs.prepare.outputs.inkless_version }} + # Full history + tags so the changelog generator can diff this release + # tag against the previous inkless-release-* tag for the release notes. + fetch-depth: 0 persist-credentials: false - name: Setup Gradle @@ -362,44 +365,83 @@ jobs: path: distributions merge-multiple: true + - name: Generate curated changelog summary + id: changelog + env: + INKLESS_VERSION: ${{ needs.prepare.outputs.inkless_version }} + run: | + # Best-effort: the curated summary is diffed from the previous + # inkless-release-* tag. If it can't be produced (e.g. first release, + # or the previous tag is missing), fall back to boilerplate-only notes + # rather than failing the release. + SUMMARY="" + if SUMMARY=$(python3 .ai-agents/skills/inkless-changelog/gen-changelog.py \ + --version "$INKLESS_VERSION" --summary 2>/tmp/changelog.err); then + echo "Generated changelog summary ($(printf '%s' "$SUMMARY" | wc -l) lines)" + else + echo "::warning::Changelog summary generation failed; using boilerplate notes only." + cat /tmp/changelog.err || true + SUMMARY="" + fi + # Persist via multiline output for the next step. + { + echo "summary<> "$GITHUB_OUTPUT" + - name: Create or update GitHub Release env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} INKLESS_VERSION: ${{ needs.prepare.outputs.inkless_version }} KAFKA_BASE_MATRIX: ${{ needs.prepare.outputs.kafka_base_matrix }} GITHUB_REPOSITORY: ${{ github.repository }} + CHANGELOG_SUMMARY: ${{ steps.changelog.outputs.summary }} run: | + set -eo pipefail RELEASE_TAG="inkless-release-${INKLESS_VERSION}" - RELEASE_NOTES="Inkless version ${INKLESS_VERSION} - - ## Docker Images - - \`\`\`bash - docker pull ghcr.io/aiven/inkless:${INKLESS_VERSION} - docker pull ghcr.io/aiven/inkless:latest - \`\`\` - - ## Kafka Versions" - - TAGS=$(echo "$KAFKA_BASE_MATRIX" | jq -r '.include[].tag // empty' 2>/dev/null | sort -V | uniq) - while IFS= read -r tag; do - if [[ "$tag" =~ ^inkless-([0-9]+\.[0-9]+\.[0-9]+)-(.+)$ ]]; then - KAFKA_VER="${BASH_REMATCH[1]}" - RELEASE_NOTES="${RELEASE_NOTES} - - Kafka ${KAFKA_VER}: [\`${tag}\`](https://github.com/${GITHUB_REPOSITORY}/tree/${tag})" + # Build the notes in a file to avoid the YAML block-scalar indentation + # leaking into the markdown (leading spaces would render headings and + # bullets as an indented code block). + NOTES_FILE="$(mktemp)" + trap 'rm -f "$NOTES_FILE"' EXIT + { + printf 'Inkless version %s\n\n' "$INKLESS_VERSION" + printf '## Docker Images\n\n' + printf '```bash\n' + printf 'docker pull ghcr.io/aiven/inkless:%s\n' "$INKLESS_VERSION" + printf 'docker pull ghcr.io/aiven/inkless:latest\n' + printf '```\n\n' + printf '## Kafka Versions\n' + + if ! TAGS=$(echo "$KAFKA_BASE_MATRIX" | jq -r '.include[].tag // empty' | sort -V | uniq); then + echo "::error::Could not parse kafka_base_matrix as JSON: $KAFKA_BASE_MATRIX" >&2 + exit 1 + fi + while IFS= read -r tag; do + if [[ "$tag" =~ ^inkless-([0-9]+\.[0-9]+\.[0-9]+)-(.+)$ ]]; then + KAFKA_VER="${BASH_REMATCH[1]}" + printf -- '- Kafka %s: [`%s`](https://github.com/%s/tree/%s)\n' \ + "$KAFKA_VER" "$tag" "$GITHUB_REPOSITORY" "$tag" + fi + done <<< "$TAGS" + + # Append the curated changelog summary when available. + if [ -n "${CHANGELOG_SUMMARY//[$'\n\r\t ']/}" ]; then + printf '\n%s\n' "$CHANGELOG_SUMMARY" fi - done <<< "$TAGS" + } > "$NOTES_FILE" if ! gh release view "$RELEASE_TAG" --repo "$GITHUB_REPOSITORY" > /dev/null 2>&1; then gh release create "$RELEASE_TAG" \ --repo "$GITHUB_REPOSITORY" \ --title "Inkless ${INKLESS_VERSION}" \ - --notes "$RELEASE_NOTES" + --notes-file "$NOTES_FILE" else gh release edit "$RELEASE_TAG" \ --repo "$GITHUB_REPOSITORY" \ - --notes "$RELEASE_NOTES" + --notes-file "$NOTES_FILE" fi if [ -d distributions ] && [ "$(ls -A distributions/ 2>/dev/null)" ]; then diff --git a/.github/workflows/inkless-release.yml b/.github/workflows/inkless-release.yml index b5413f66126..61ce9c5972c 100644 --- a/.github/workflows/inkless-release.yml +++ b/.github/workflows/inkless-release.yml @@ -36,6 +36,11 @@ on: description: 'Space-separated list of additional release branches to include (e.g. "inkless-4.3"). Used when onboarding a new branch for the first time.' required: false type: string + resume: + description: 'Resume a partially-completed release whose tags already exist but whose GitHub Release was never published. Skips tag creation and re-runs build+publish. Requires inkless_version to be set explicitly.' + required: false + default: false + type: boolean concurrency: group: inkless-release @@ -56,6 +61,7 @@ jobs: main_commit: ${{ steps.resolve.outputs.main_commit }} release_tag: ${{ steps.resolve.outputs.release_tag }} active_branches: ${{ steps.resolve.outputs.active_branches }} + resume: ${{ steps.resolve.outputs.resume }} kafka_versions: ${{ steps.resolve-kafka.outputs.kafka_versions }} build_matrix: ${{ steps.resolve-kafka.outputs.build_matrix }} kafka_base_matrix: ${{ steps.resolve-kafka.outputs.kafka_base_matrix }} @@ -73,9 +79,19 @@ jobs: INPUT_VERSION: ${{ inputs.inkless_version }} INPUT_COMMIT: ${{ inputs.main_commit }} INPUT_EXTRA_BRANCHES: ${{ inputs.extra_branches }} + INPUT_RESUME: ${{ inputs.resume }} run: | + RESUME="${INPUT_RESUME:-false}" + # --- Inkless version --- - if [ -z "$INPUT_VERSION" ]; then + if [ "$RESUME" = "true" ]; then + # Resume reuses an EXISTING version; auto-increment would be wrong. + if [ -z "$INPUT_VERSION" ]; then + echo "::error::resume=true requires inkless_version to be set explicitly (the version whose tags already exist)." + exit 1 + fi + INKLESS_VERSION="$INPUT_VERSION" + elif [ -z "$INPUT_VERSION" ]; then LATEST=$(git tag -l "inkless-release-*" | sort -V | tail -n1) if [ -z "$LATEST" ]; then echo "::error::No existing inkless-release-* tags found; specify inkless_version explicitly" @@ -91,10 +107,19 @@ jobs: RELEASE_TAG="inkless-release-${INKLESS_VERSION}" - # Ensure the version doesn't already exist - if git rev-parse "$RELEASE_TAG" >/dev/null 2>&1; then - echo "::error::Tag $RELEASE_TAG already exists. If the release was never finalized (no GitHub Release published), delete the tag and retry. Otherwise use a higher version number." - exit 1 + # --- Tag existence guard (inverted under resume) --- + if [ "$RESUME" = "true" ]; then + # Resuming: the release tag MUST already exist (tagging is skipped). + if ! git rev-parse --verify --quiet "refs/tags/$RELEASE_TAG" >/dev/null; then + echo "::error::resume=true but $RELEASE_TAG does not exist. Run a normal release (resume=false) to create tags." + exit 1 + fi + echo "Resuming release $RELEASE_TAG (tags already exist; will skip tagging, re-run build+publish)." + else + if git rev-parse --verify --quiet "refs/tags/$RELEASE_TAG" >/dev/null; then + echo "::error::Tag $RELEASE_TAG already exists. To finish a release whose tags exist but was never published, re-run with resume=true. Otherwise use a higher version number." + exit 1 + fi fi # --- Main commit --- @@ -107,19 +132,25 @@ jobs: fi # --- Active release branches --- - # Discover from the previous release's Kafka-base tags. - PREV_VERSION=$(git tag -l "inkless-release-*" | sort -V | tail -n1) - PREV_VERSION="${PREV_VERSION#inkless-release-}" + # Discover active branches from Kafka-base tags. Normally these come + # from the PREVIOUS release; when resuming, the target version's own + # tags already exist, so discover from those. + if [ "$RESUME" = "true" ]; then + DISCOVER_VERSION="$INKLESS_VERSION" + else + DISCOVER_VERSION=$(git tag -l "inkless-release-*" | sort -V | tail -n1) + DISCOVER_VERSION="${DISCOVER_VERSION#inkless-release-}" + fi DISCOVERED="" - if [ -n "$PREV_VERSION" ]; then - DISCOVERED=$(git tag -l "inkless-*-${PREV_VERSION}" \ + if [ -n "$DISCOVER_VERSION" ]; then + DISCOVERED=$(git tag -l "inkless-*-${DISCOVER_VERSION}" \ | grep -E '^inkless-[0-9]+\.[0-9]+\.[0-9]+-' \ | while read -r tag; do if [[ "$tag" =~ ^inkless-([0-9]+\.[0-9]+)\.[0-9]+-(.+)$ ]]; then echo "inkless-${BASH_REMATCH[1]}" fi done | sort -u) - echo "Discovered active branches from ${PREV_VERSION} tags: $(echo "$DISCOVERED" | tr '\n' ' ')" + echo "Discovered active branches from ${DISCOVER_VERSION} tags: $(echo "$DISCOVERED" | tr '\n' ' ')" fi # Merge with extra_branches input @@ -135,6 +166,7 @@ jobs: echo "inkless_version=$INKLESS_VERSION" >> $GITHUB_OUTPUT echo "main_commit=$MAIN_COMMIT" >> $GITHUB_OUTPUT echo "release_tag=$RELEASE_TAG" >> $GITHUB_OUTPUT + echo "resume=$RESUME" >> $GITHUB_OUTPUT echo "active_branches=$(echo "$ALL_BRANCHES" | tr '\n' ' ' | sed 's/ $//')" >> $GITHUB_OUTPUT - name: Resolve Kafka versions for active branches @@ -142,8 +174,13 @@ jobs: env: ACTIVE_BRANCHES: ${{ steps.resolve.outputs.active_branches }} INKLESS_VERSION: ${{ steps.resolve.outputs.inkless_version }} + RESUME: ${{ steps.resolve.outputs.resume }} run: | - # For each active branch, determine its current Kafka version from gradle.properties + # For each active branch, determine its Kafka version from gradle.properties. + # Normal release reads the branch HEAD (about to be tagged). On resume the + # tags already exist and the branch HEAD may have moved on, so read the + # gradle.properties captured at the existing kafka-base tag instead by + # discovering the tag directly. BUILD_MATRIX_ENTRIES="" KAFKA_MATRIX_ENTRIES="" LATEST_TAGS_JSON="{" @@ -151,8 +188,23 @@ jobs: KAFKA_VERSIONS="" for BRANCH in $ACTIVE_BRANCHES; do - KAFKA_VER=$(git show "origin/${BRANCH}:gradle.properties" \ - | grep '^version=' | head -1 | sed 's/version=\(.*\)-inkless.*/\1/') + if [ "$RESUME" = "true" ]; then + # Find the existing kafka-base tag for this branch+version and use it + # as-is: on resume the tag is authoritative (it was validated at cut + # time). Do NOT recompute the tag name from gradle.properties -- that + # would assume the embedded Kafka version still matches the tag. + KAFKA_MINOR="${BRANCH#inkless-}" + TAG=$(git tag -l "inkless-${KAFKA_MINOR}.*-${INKLESS_VERSION}" | sort -V | tail -n1) + if [ -z "$TAG" ]; then + echo "::error::resume: no existing tag inkless-${KAFKA_MINOR}.*-${INKLESS_VERSION} for branch $BRANCH" + exit 1 + fi + KAFKA_VER=$(git show "${TAG}:gradle.properties" \ + | grep '^version=' | head -1 | sed 's/version=\(.*\)-inkless.*/\1/') + else + KAFKA_VER=$(git show "origin/${BRANCH}:gradle.properties" \ + | grep '^version=' | head -1 | sed 's/version=\(.*\)-inkless.*/\1/') + fi if [ -z "$KAFKA_VER" ]; then echo "::error::Cannot determine Kafka version for branch $BRANCH (gradle.properties version= not found)" @@ -160,7 +212,11 @@ jobs: fi KAFKA_MAJOR_MINOR=$(echo "$KAFKA_VER" | cut -d. -f1-2) - TAG="inkless-${KAFKA_VER}-${INKLESS_VERSION}" + # On resume TAG is the discovered tag (kept above); on a normal release + # it is the tag about to be created for this branch's HEAD. + if [ "$RESUME" != "true" ]; then + TAG="inkless-${KAFKA_VER}-${INKLESS_VERSION}" + fi echo "$BRANCH -> Kafka $KAFKA_VER -> tag $TAG" KAFKA_VERSIONS="${KAFKA_VERSIONS} ${TAG}" @@ -185,11 +241,17 @@ jobs: echo "kafka_base_matrix={\"include\":[${KAFKA_MATRIX_ENTRIES}]}" >> $GITHUB_OUTPUT echo "latest_tags=${LATEST_TAGS_JSON}" >> $GITHUB_OUTPUT - # Assert that each active release branch is ready to release: - # - the target main commit is present on the branch - # - branch-consistency.sh reports zero missing commits + # Assert that each active release branch is ready to release: it contains every + # actionable inkless commit from main (by PR presence, via branch-consistency.sh + # --check). We do NOT assert the exact main commit SHA is an ancestor: release + # branches are built by CHERRY-PICKING from main, so the commits are present + # under different SHAs. Skip-listed and old commits do not fail the check. validate: needs: prepare + # On resume the tags already exist and were validated when first cut; the + # branches (and main) may have moved on since, so re-checking against current + # main would spuriously fail. Skip validation when resuming. + if: needs.prepare.outputs.resume != 'true' runs-on: ubuntu-latest steps: - name: Checkout @@ -201,38 +263,35 @@ jobs: - name: Validate all active branches env: ACTIVE_BRANCHES: ${{ needs.prepare.outputs.active_branches }} - MAIN_COMMIT: ${{ needs.prepare.outputs.main_commit }} run: | FAILED=0 for BRANCH in $ACTIVE_BRANCHES; do - git fetch origin "$BRANCH" - - # Assert main commit is present - if ! git merge-base --is-ancestor "$MAIN_COMMIT" "origin/$BRANCH"; then - echo "::error::Commit $MAIN_COMMIT is NOT present on $BRANCH. Cherry-pick it first." - FAILED=1 - continue - fi - echo "✓ $MAIN_COMMIT is present on $BRANCH" - - # Assert no missing inkless commits - MISSING=$(./inkless-sync/branch-consistency.sh "$BRANCH" --missing 2>&1 \ - | grep -E '^- [0-9a-f]{10}' | wc -l | tr -d ' ') - if [ "$MISSING" -gt 0 ]; then - echo "::error::$BRANCH has $MISSING missing inkless commit(s) from main. Run cherry-pick sync first." - ./inkless-sync/branch-consistency.sh "$BRANCH" --missing + # Populate the remote-tracking refs the check reads (origin/main and + # origin/). A bare `git fetch origin ` only updates + # FETCH_HEAD, so branch-consistency.sh's origin/ lookup fails. + git fetch origin \ + "+refs/heads/main:refs/remotes/origin/main" \ + "+refs/heads/$BRANCH:refs/remotes/origin/$BRANCH" + + # branch-consistency.sh --check exits non-zero when the branch has + # actionable missing inkless commits. It matches by PR number, which + # is correct for cherry-picked history (SHAs differ from main). + if ./inkless-sync/branch-consistency.sh "$BRANCH" --check; then + echo "✓ $BRANCH is in sync with main" + else + echo "::error::$BRANCH is missing actionable inkless commits from main. Run cherry-pick sync first." FAILED=1 - continue fi - echo "✓ $BRANCH is in sync with main" done exit $FAILED # Create and push all tags: # - inkless-release- on the resolved main commit # - inkless-- on the HEAD of each active release branch + # Skipped when resuming a release whose tags already exist. tag: needs: [prepare, validate] + if: needs.prepare.outputs.resume != 'true' runs-on: ubuntu-latest permissions: contents: write @@ -278,8 +337,11 @@ jobs: # Build Docker images and binary distributions for all Kafka-base tags, # then finalize the GitHub Release with all artifacts. + # Runs after tagging on a normal release, or directly after validate when + # resuming (the tag job is skipped because tags already exist). publish: - needs: [prepare, tag] + needs: [prepare, validate, tag] + if: ${{ always() && !cancelled() && needs.prepare.result == 'success' && (needs.validate.result == 'success' || needs.validate.result == 'skipped') && (needs.tag.result == 'success' || needs.tag.result == 'skipped') }} uses: ./.github/workflows/inkless-publish.yml with: inkless_version: ${{ needs.prepare.outputs.inkless_version }} diff --git a/AGENTS.md b/AGENTS.md index acd6b31dc6e..ef050079b51 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,8 +54,10 @@ Read these only when a task touches the relevant area. All docs are under | `configs.rst`, `topic_configs.rst`, `metrics.rst` | Auto-generated config/metrics reference. | Upstream Kafka build, test, and tooling commands are in the root -[`README.md`](README.md). Upstream sync procedures live in -[`inkless-sync/`](inkless-sync/). +[`README.md`](README.md). Upstream sync tooling lives in +[`inkless-sync/`](inkless-sync/), driven by the +[`inkless-upstream-sync`](.ai-agents/skills/inkless-upstream-sync/SKILL.md) and +[`inkless-release-prep`](.ai-agents/skills/inkless-release-prep/SKILL.md) skills. ### Engineering Rules diff --git a/docs/inkless/CHANGELOG.md b/docs/inkless/CHANGELOG.md new file mode 100644 index 00000000000..7ad59df5264 --- /dev/null +++ b/docs/inkless/CHANGELOG.md @@ -0,0 +1,385 @@ +# Inkless Changelog + +Detailed, per-increment log of Inkless changes. Each section covers one Inkless +iteration (`inkless-release-` -> `inkless-release-`) and lists the +mainline commits plus operator-facing config and metric changes. + +- For release artifacts (Docker images, binaries), see [RELEASES.md](RELEASES.md). +- For how versions work, see [VERSIONING-STRATEGY.md](VERSIONING-STRATEGY.md). +- GitHub release notes are a **curated summary** of these entries (features and + fixes plus config/metric changes; chores, tests, and docs are dropped). + +## How this file is produced + +Entries are generated from git and the auto-generated docs, then curated: + +- **Commits** come from `git log --first-parent --no-merges` between the two + release tags, categorized by conventional-commit type. `--first-parent` + excludes the individual upstream commits pulled in by `apache/kafka` merge + commits. +- **Config changes** are diffed from `docs/inkless/configs.rst` (reliable). +- **Metric changes** are diffed from `docs/inkless/metrics.rst`. This file is + produced by a hand-maintained registry list in `MetricsDocs.main()`, so a + delta can reflect documentation catch-up rather than a newly shipped metric + (see the 0.39 note). Metric deltas are always curated before publishing. +- **Upstream syncs** are detected from `apache/kafka` merge commits in the range + and a `gradle.properties` version delta; when `main`'s Kafka development base + moves, it is noted as a blockquote under the increment heading. + +Regenerate a draft with the `inkless-changelog` skill (or run it directly): + +```bash +.ai-agents/skills/inkless-changelog/gen-changelog.py --version # detailed entry +.ai-agents/skills/inkless-changelog/gen-changelog.py --version --summary # release-notes summary +``` + +--- + +## 0.44 (Kafka 4.1.2, 4.2.1) + +### Features +- (inkless:switch) auto-enable remote.storage atomically on diskless switch (#678) +- (inkless:consolidation) [KC-298] add JMX metrics for the cross-tier log start offset (#682) +- (inkless) [KC-298] track and serve the cross-tier earliest offset for consolidated topics (#670) +- (inkless:switch) implement AlterDisklessSwitch and tooling [KC-97] (#665) +- (inkless) resolve client AZ from listener map in metadata transformer (#684) +- (inkless) add inkless.client.az.listener.map config (#683) +- (inkless:consolidation) route consolidation fetcher through coldpath to avoid cache pollution [KC-171] (#679) +- (inkless) coalesce contiguous batches into one row in commit_file_v2 (#671) +- (inkless) add partition fan-in commit metrics (#675) + +### Fixes +- (inkless:consolidation) serve cross-tier earliest offset promptly after startup (#688) +- (inkless) find_batches rejects offset below log start offset (#666) +- (inkless:consolidation) don't fence a consolidating leader below the seal (#677) +- (inkless:controller) fix leader skew for managed diskless after rolling restart (#643) +- (inkless:consolidation) recover a switched consolidated leader after local-log loss (#673) + +### Tests +- (inkless:consolidation) add dependency-outage system test for consolidation pipeline (#669) +- (inkless:consolidation) add read-from-remote system tests, harness support (#654) +- (inkless:consolidation) add born-consolidated pipeline + WAL-prune system test (#674) + +### Docs +- (inkless) document the interceptors introduced for CREATE_TOPIC (#680) + +### Config & metric changes +- config added: `inkless.client.az.listener.map` +- config added: `inkless.consume.cross.tier.log.start.cache.enabled` +- config added: `inkless.consume.cross.tier.log.start.cache.ttl.ms` +- config added: `inkless.control.plane.batch.coalescing.enabled` +- config added: `inkless.cross.tier.log.start.report.interval.ms` +- metric mbean added: `io.aiven.inkless.cache:type=CrossTierLogStartCache` +- metric mbean added: `io.aiven.inkless.delete:type=CrossTierLogStartReporter` +- metric attrs added: `CrossTierLogStartCache :: CacheHits, CacheMisses, CacheSize` +- metric attrs added: `CrossTierLogStartReporter :: PartitionsReported, PendingPartitions, ReportErrors` +- metric attrs added: `PostgresControlPlane :: AdvanceCrossTierLogStartQueryRate, AdvanceCrossTierLogStartQueryTime` +- metric attrs added: `FileCommitter :: BatchesPerPartitionPerCommit, PartitionsPerCommit` + +## 0.43 (Kafka 4.1.2, 4.2.1) + +### Fixes +- (inkless:consolidation) start consolidation when remote storage is enabled on a diskless topic (#672) +- (inkless:consolidation) complete the classic-to-consolidated switch by always sealing and registering (#651) +- (inkless) Add PostgreSQL socket timeouts for Inkless control plane (#668) +- (inkless:metrics) exclude consolidating partitions from URP metrics (#640) +- (inkless) truncate above the seal only if those messages are not consolidated (#667) + +### Config & metric changes +- config added: `inkless.control.plane.connection.pool.timeout.ms` +- config added: `inkless.control.plane.read.connection.pool.timeout.ms` +- config added: `inkless.control.plane.read.socket.timeout.ms` +- config added: `inkless.control.plane.read.tcp.connect.timeout.ms` +- config added: `inkless.control.plane.socket.timeout.ms` +- config added: `inkless.control.plane.tcp.connect.timeout.ms` +- config added: `inkless.control.plane.write.connection.pool.timeout.ms` +- config added: `inkless.control.plane.write.socket.timeout.ms` +- config added: `inkless.control.plane.write.tcp.connect.timeout.ms` + +## 0.42 (Kafka 4.1.2, 4.2.1) + +### Features +- (inkless:tools) add operator tooling for topic type switch [KC-97] (#661) +- (inkless:consolidation) supplement local log with diskless data on fetch [KC-168] (#638) +- (inkless:consolidation) serve consolidated reads from remote via OFFSET_MOVED_TO_TIERED_STORAGE (#650) +- (inkless) ensure unclean leader election is disabled on classic-to-diskless switch [KC-129] (#647) +- (inkless) KC-156 diskless leader epoch for consolidation truncation (#631) +- (inkless) add controller guards for classic-to-diskless switch [POD-2464] (#634) +- (inkless:test) implement system tests for classic to diskless switch (#581) +- (inkless:consolidation) make fetch quota config dynamic (#637) +- (inkless) add dedicated rate limit for consolidation fetch [KC-145] (#636) +- (inkless:consolidation) add separate wiring for consolidation fetcher [KC-160] (#635) + +### Fixes +- (inkless:switch) Fix stale HW on switched leader promotion (#660) +- (inkless:systest) fix sigstop and slow consumer giving false negatives (#659) +- (inkless) create tiered topic when diskless.enable=false and system is disabled (#641) +- (inkless) reject topic creation with config conflicting with diskless regex (#642) +- (inkless) Remove consolidating fetchers unconditionally (#629) + +### Refactors +- (inkless) clarify fetch routing for consolidating partitions (#633) + +### Tests +- (inkless:systest) run ducktape system tests against consolidated topics (#645) +- (inkless) Integration test for diskless + consolidate separately (#639) + +### Docs +- (inkless) Enrich documentation of classicToDisklessStartOffset (#658) +- (inkless) fix rendering of release sync prompt (#648) +- (inkless:switch) Document classic to diskless switch (#632) + +### Config & metric changes +- no config changes + +## 0.41 (Kafka 4.1.2, 4.2.0) + +### Features +- (inkless) KC-72 Reconcile stale records after diskless switch (#612) +- (inkless:consume) Add request hedging for storage reads (#582) + +### Fixes +- (inkless) initialize diskless switch for legacy alter configs (#630) +- (inkless:consolidation) prevent ConsolidationFetcherThread crash on topic deletion (#627) +- (inkless:switch) bump leader epoch on classic-to-diskless switch (#626) +- (inkless:build) pass commitId to Gradle for worktree builds (#628) +- (inkless) retry describeTopics in InklessManagedReplicasClusterTest for metadata propagation (#625) +- (inkless) ensure ReplicaManager shutdown in Inkless tests (#623) + +### Refactors +- (inkless) SharedState to own StorageBackend lifecycle (#562) +- (inkless) extract Inkless tests from ReplicaManagerTest into standalone file (#624) + +### Tests +- (inkless:switch) add consolidated diskless integration tests with transition matrix (#617) + +### Docs +- (inkless) add system test documentation (#588) + +### Config & metric changes +- config added: `inkless.fetch.hedge.total.time.threshold.ms` +- config added: `inkless.fetch.hedge.ttfb.threshold.ms` +- metric attrs added: `InklessFetchMetrics :: HedgeRequestRate, HedgeWonRate, HedgeTtfbTriggeredRate, HedgeTotalTimeTriggeredRate` (request hedging, #582) + +## 0.40 (Kafka 4.1.2, 4.2.0) + +### Features +- (inkless) Introduce CREATE_TOPICS config interceptor framework and DisklessForceCreateTopicInterceptor (#614) +- (inkless:switch) auto-enable remote.storage.enable on diskless topic creation (#619) +- (inkless:switch) validate diskless requires remote storage when consolidation enabled (#616) +- (inkless:switch) expose DisklessWithoutRemoteStorageCount metric for legacy topics (#618) +- (inkless) Support OffsetsForLeaderEpoch for partitions switched to diskless (#613) +- (inkless) POD-2395 Prune consolidated diskless offsets (#587) +- (inkless:switch) support DELETE_RECORDS for hybrid partitions (#611) +- (inkless) POD-2456 Enable offset fetch in consolidating partitions (#594) +- (inkless) POD-2457 Allow transactional offset commits for Diskless sources (#596) + +### Fixes +- (inkless) add KafkaConfigTest validation for diskless force topic regexes (#622) +- (inkless) allow searching offsets in UnifiedLog only if switched topics are in sync (#620) +- (inkless) correctly migrate all producer states after a diskless switch (#615) +- (inkless:fetch) properly propagate exception back on failing FetchOffset (#608) +- (inkless:switch) Advance HW past stale checkpoint for sealed leader after restart (#605) +- (inkless:switch) Init Diskless Log on Control Plane on leader changes (#603) +- (inkless:migration) reschedule fetcher on leader change during pending migration (#600) +- (inkless) POD-1965 Use local log start in DisklessLeaderEndPoint (#591) +- (inkless:migration) catch up replicas that are below the seal (#598) +- (inkless) Ensure additional system topics created as classic when log.diskless.enable=true [POD-1312] (#586) + +### Refactors +- (inkless:consolidation) rename and improve ConsolidationMetrics (#621) +- (inkless:switch) drop sealing and registering from BrokerMetadataPublisher (#604) +- (inkless) Rename diskless migration to diskless switch (#601) + +### Tests +- (inkless:switch) Add invariant tests for sealed partition recovery (#609) +- (inkless) add test for verifying consolidating partition reassignment (#599) + +### Chores +- (inkless) remove FileMerger component and all related infrastructure (#607) + +### Config & metric changes +- config added: `inkless.consolidation.cleanup.interval.ms` +- config removed: `inkless.control.plane.file.merge.lock.period.ms` +- config removed: `inkless.control.plane.file.merge.size.threshold.bytes` +- config removed: `inkless.control.plane.read.file.merge.lock.period.ms` +- config removed: `inkless.control.plane.read.file.merge.size.threshold.bytes` +- config removed: `inkless.control.plane.write.file.merge.lock.period.ms` +- config removed: `inkless.control.plane.write.file.merge.size.threshold.bytes` +- config removed: `inkless.file.merger.interval.ms` +- config removed: `inkless.file.merger.temp.dir` +- metric mbean removed: `io.aiven.inkless.merge:type=FileMerger` (FileMerger component removed, #607) +- metric attrs removed: `FileMerger :: FileMergeErrorRate, FileMergeFilesRate, FileMergeRate, FileMergeTotalTime, FileUploadTime` (#607) +- metric attrs removed: `PostgresControlPlane :: {Commit,Get,Release}FileMergeWorkItemQuery{Rate,Time}` (#607) + +## 0.39 (Kafka 4.1.2) + +### Features +- (inkless) fast fail producing during classic-to-diskless migration (#595) +- (inkless) Always allow fetching from replicas for migrated partitions (#593) +- (inkless) extended metrics for diskless migration states tracking (#589) +- (inkless) add metrics for consolidated topic partitions (#590) + +### Config & metric changes +- New metrics for diskless migration state tracking (#589) and consolidated + topic partitions (#590). +- Documentation catch-up: `docs/inkless/metrics.rst` expanded from 3 to 16 mbeans + in this release (KC-1, #592). Most of the mbeans that appear "new" in the doc + diff (fetch, produce, delete, cache, control-plane, thread-pool, AZ-awareness) + were already shipping earlier; they became documented here. Do not read the + raw metrics diff for this release as newly introduced metrics. + +## 0.38 (Kafka 4.1.2) + +### Features +- (inkless) Support ListOffsets for all cases of diskless partitions (#584) +- (inkless) POD-2394 Implement read path for consolidated logs (#583) +- (inkless) Support basic ListOffset for hybrid topics (#577) +- (inkless) POD-2398 Integration test for consolidation produce path (#569) +- (inkless) POD-2393 Add DisklessLeaderEndPoint to handle fetch requests (#568) +- (inkless) Set migrating partitions in KRaft metadata (#580) +- (inkless) Abort transactions on partition sealing (#573) +- (inkless) POD-2392 Implement consolidating partition tracking (#567) +- (inkless:consume) Add time-to-first-byte (TTFB) metric for storage reads (#575) +- (inkless) Init diskless log on Control Plane (#563) +- (inkless) Add TS Consolidation configs (#556) +- (inkless:config) enforce diskless feature flag dependency chain (#560) +- (inkless:docker) add diskless tiered storage unification demo (#559) +- (inkless) allow reading from UnifiedLog for diskless topics (#553) +- (inkless:metadata) Remove topic already diskless from InitDisklessLog Controller API (#547) +- (inkless) Orchestrate classic-to-diskless migration (#536) +- (inkless) Parse request and response of InitDisklessLog (#545) +- (inkless:config) Enforce diskless.allow.from.classic.enable (#546) + +### Fixes +- (inkless) Stop replicating after migration to diskless is completed (#585) +- (cache) respect max-idle=-1 (disable idle eviction) (#470) +- (inkless) Create Partition object even when diskless is enabled (#574) +- (inkless:migration) Avoid deadlock in InitDisklessLogBatchQueue (#578) +- (inkless) Add InitDisklessLog JSON conversion to RequestConvertToJson (#570) +- (inkless:migration) Fix InitDisklessLogManager gaps during partition registration (#566) +- (inkless:migration) fix classic-to-diskless migration validation and add LogConfigTest to CI (#558) +- (inkless:migration) bypass diskless/remote-storage mutual exclusion for classic-to-diskless migration (#555) +- (inkless:consume) handle error path in lastOffsetForLeaderEpoch for diskless topics (#554) +- (inkless:metrics) Fix double-counting of diskless topic and partition metrics (#552) + +### Refactors +- (inkless) Rename disklessStartOffset to classicToDisklessStartOffset (#572) +- (inkless) Split InitDisklessLogManager logic into separate components (#561) +- (inkless) Refactor InitDisklessLog flow and add integration tests (#549) + +### Other +- Fix potential leak when closing resources (#565) +- update Kafka version variables to 4.2.0-inkless-SNAPSHOT (#548) + +### Config & metric changes +- config added: `inkless.consolidation.*` (TS Consolidation configs, #556) +- `diskless.allow.from.classic.enable` feature flag enforced (#546, #560). + +## 0.37 (Kafka 4.0.2, 4.1.2) + +### Features +- (inkless) add bytes limit to cache (#544) +- (inkless) Expose InitDisklessLog controller API (#541) +- (controller:diskless) add partitions support for diskless topics (#540) +- (controller:diskless) enable immediate partition reassignment (#537) +- (storage:inkless) add control plane method for getting the producer states (#530) +- (inkless) Add metrics for sealed partitions (#534) +- (inkless:sync) add upstream sync tooling and documentation (#498) +- (diskless) add managed replicas routing to metadata transformer (#504) +- (inkless) Seal the local log when a topic is migrated from classic to diskless (#533) +- (inkless) implement InitDisklessLog Controller API (#531) +- (metadata:diskless) add controller metrics for diskless topics (#503) +- (metadata:diskless) implement managed replicas for diskless topics (#492) +- (storage:inkless) InitDisklessLog Diskless Controller API (#528) +- (produce) optimize AppendCompleter to complete futures first (#529) + +### Refactors +- (inkless:consume) replace ByteRange.coalesce with BoundingRangeAlignment strategy (#532) +- (inkless) improve thread pool lifecycle management (#475) +- (metadata:diskless) preserve leader epoch in metadata transformer (#539) +- (inkless) cache LogConfig in InklessMetadataView (#474) + +### Tests +- (metadata:diskless) add integration tests for managed replicas (#542) + +### Docs +- (inkless) add managed replicas documentation (#535) +- (docker:inkless) add managed replicas demo with test procedure (#543) + +### Config & metric changes +- config added: `inkless.consume.cache.max.bytes` + +## 0.36 (Kafka 4.0.0, 4.0.2, 4.1.1, 4.1.2) + +### Features +- (inkless:config) disallow setting diskless.enable if diskless storage system is disabled (#520) + +### Fixes +- (inkless:test) Fix KafkaConfigTest for CLASSIC_REMOTE_STORAGE_FORCE_EXCLUDE_TOPIC_REGEXES_CONFIG (#521) +- (inkless) Fix error message when diskless.enable and remote.storage.enable are set (#516) + +### Chores +- (build) Set Docker API version 1.44 in Gradle config (#519) + +### Config & metric changes +- no config changes + +## 0.35 (Kafka 4.0.0, 4.1.1) + +> Upstream sync: main development base moved to Kafka 4.2.0-SNAPSHOT (from 4.1.0-SNAPSHOT), Scala 2.13.16 -> 2.13.17. + +### Features +- (metadata) Introduce ClassicTopicRemoteStorageForcePolicy (#514) +- Disallow setting remote.storage.enable when diskless.enable is set to true (#511) + +### Chores +- (ci) use Docker API version 1.44 (#509) +- (ci) replace usage of JDK 23 with 25 (#502) +- (inkless) Update demo and documentation for GHCR images (#497) +- (inkless:release) add gh workflows to release inkless artifacts (#489) + +### Other +- storage: add metrics constructor to InMemoryStorage (#510) + +### Config & metric changes +- no config changes + +## 0.34 (Kafka 4.0.0, 4.1.1) + +### Features +- Allow switching diskless.enable from false to true (#486) + +### Fixes +- (storage:metrics) eagerly initialize meters with only topicType tag (#493) + +### Refactors +- (metadata:diskless) fail on topic creation with replica assignment (#488) + +### Docs +- (inkless) update architecture diagram (#487) +- (inkless) update readme with new sections and glossary (#483) +- (inkless) fix performance docs (#482) +- (inkless) fix relation with kips on client-az awareness (#485) +- (inkless) add az-alignment feature documentation (#481) + +### Chores +- (storage:inkless) add jooq classes (#490) + +### Config & metric changes +- no config changes + +## 0.33 (Kafka 4.0.0, 4.1.1) + +First release under the global Inkless iteration counter (previously +`inkless-4.0.0-rc32` / `inkless-4.1.1-rc1`; see +[VERSIONING-STRATEGY.md](VERSIONING-STRATEGY.md)). + +### Refactors +- (inkless:metrics) only add topic-type tag on all topic stats (#472) + +### Docs +- (inkless) add versioning strategy (#479) + +### Config & metric changes +- no config changes diff --git a/docs/inkless/README.md b/docs/inkless/README.md index b87e6b48205..eaf3b1f5452 100644 --- a/docs/inkless/README.md +++ b/docs/inkless/README.md @@ -47,6 +47,7 @@ See [Releases](RELEASES.md) for active versions and release history. ### Reference - [Versioning Strategy](VERSIONING-STRATEGY.md) - Version format and release workflow - [Releases](RELEASES.md) - Artifacts released (binaries and docker images) +- [Changelog](CHANGELOG.md) - Per-increment changes (features, fixes, config and metric changes) - [Glossary](GLOSSARY.md) - Definitions of Inkless-specific terms and concepts ### Development & Maintenance diff --git a/docs/inkless/RELEASES.md b/docs/inkless/RELEASES.md index 3f4cf1955db..2e0825483de 100644 --- a/docs/inkless/RELEASES.md +++ b/docs/inkless/RELEASES.md @@ -133,18 +133,87 @@ Go to **GitHub Actions → Inkless Release → Run workflow** and fill in: | `inkless_version` | Version to release (e.g. `0.44`) | Auto-increments from latest tag | | `main_commit` | Commit on `main` to release | `main` HEAD | | `extra_branches` | Space-separated branches to add (e.g. `inkless-4.3`) | Empty — use for new branches only | +| `resume` | Finish a release whose tags already exist but was never published. Skips tagging, re-runs build+publish. Requires `inkless_version`. | `false` | The workflow then: 1. **Resolves** the version and discovers active branches from the previous release's tags -2. **Validates** each branch — asserts the target commit is present and no inkless commits are missing -3. **Creates and pushes tags** — `inkless-release-` on main and `inkless--` on each branch + (from the target version's own tags when resuming) +2. **Validates** each branch — asserts every actionable inkless commit is present (by PR + number, via `branch-consistency.sh --check`; cherry-picked commits have different SHAs + than main, so presence is matched by PR, not by SHA ancestry) +3. **Creates and pushes tags** — `inkless-release-` on main and `inkless--` on each branch (**skipped when `resume=true`**) 4. **Builds** Docker images (amd64 + arm64) and binary distributions for each Kafka version in parallel 5. **Finalizes** — creates the GitHub Release, attaches all artifacts, publishes `latest` and `X.Y-latest` Docker aliases If validation fails (a branch is missing commits), the workflow aborts before touching any tags. Fix the branch with cherry-pick sync and re-trigger. +**Resuming a partial release:** if a run created the tags but never published the GitHub Release +(e.g. it failed after step 3), re-run with the same `inkless_version` and `resume=true`. It re-validates, +skips tag creation, and re-runs build + finalize. Do not delete tags or bump the version. + +### Break-glass: build and publish by hand + +Only if the workflow is unavailable. This reproduces steps 4-5 using the same `make` targets the +workflow calls (`inkless-publish.yml`). Do it per **Kafka-base tag** (one per active branch), building +each architecture. `VERSION`/`DIST_VERSION` are derived from `gradle.properties` at the checked-out +tag, so no version override is needed. The image tag is `--` +(e.g. `4.1.2-0.45-amd64`) -- NOT `-`. + +```bash +# Example for 0.45: tags inkless-4.1.2-0.45 and inkless-4.2.1-0.45. +# Run each from a checkout of that tag (a worktree is convenient). +export IMAGE=ghcr.io/aiven/inkless + +for pair in "inkless-4.1.2-0.45 4.1.2" "inkless-4.2.1-0.45 4.2.1"; do + set -- $pair; TAG="$1"; KVER="$2"; INC="0.45" + git -C ../inkless-work checkout "$TAG" # detached checkout at the kafka-base tag + ( cd ../inkless-work && make build_release ) # VERSION comes from gradle.properties at the tag + + # Per-arch build + push (loads/pushes the arch-suffixed tag) + for ARCH in amd64 arm64; do + ( cd ../inkless-work && make docker_build \ + PLATFORM="linux/${ARCH}" \ + DOCKER_TAGS="${IMAGE}:${KVER}-${INC}-${ARCH}" \ + PUSH=true ) + done + + # Multi-arch manifest for the version tag (and the X.Y-latest alias for the highest patch) + docker buildx imagetools create -t "${IMAGE}:${KVER}-${INC}" \ + "${IMAGE}:${KVER}-${INC}-amd64" "${IMAGE}:${KVER}-${INC}-arm64" +done +``` + +Attach the `.tgz` distributions and publish the Release as in the changelog section below +(the workflow's `finalize-release` normally does this). Prefer `resume=true` over this whenever +the workflow is available. + +### Changelog and release notes + +Every release increment gets an entry in [CHANGELOG.md](CHANGELOG.md) and a curated GitHub +release-notes summary. Both are generated from the commit history plus the auto-generated +`configs.rst`/`metrics.rst` diffs between the previous and new `inkless-release-*` tags, then curated +(metric deltas in particular need review -- see the changelog header for why). + +The **curated summary is injected into the GitHub Release body automatically** by the release +workflow's `finalize-release` job (it runs the generator against the freshly created tag; if +generation fails it falls back to boilerplate notes). You do **not** need to paste it by hand, +and re-running publish reproduces the same notes rather than clobbering them. + +The **detailed CHANGELOG.md entry is a manual commit to `main` via a normal PR** (there is no +automation and no auto-PR). Because it is diffed from the new `inkless-release-` tag, it can +only be produced **after** the release workflow creates that tag -- in practice at the start of +the next increment's prep. Generate it and prepend to `docs/inkless/CHANGELOG.md`: + +```bash +# Detailed CHANGELOG entry for inkless-release- (prepend to CHANGELOG.md, then PR to main) +.ai-agents/skills/inkless-changelog/gen-changelog.py --version + +# Curated summary (same text the workflow injects) -- for preview/manual override +.ai-agents/skills/inkless-changelog/gen-changelog.py --version --summary +``` + ### Onboarding a new release branch When a new Kafka minor branch is ready (e.g. `inkless-4.3`), add it to the first release via the @@ -155,6 +224,9 @@ in future runs. For details on how Inkless versions work, see [Versioning Strategy](VERSIONING-STRATEGY.md). +The increment number is carried only by tags, never by `gradle.properties`, so a release involves +no per-branch version bump commit. + **Quick summary:** - Same Inkless version (e.g., `0.33`) across Kafka versions = same Inkless features - Higher Inkless version = newer features diff --git a/inkless-sync/MAIN-SYNC-PROMPT.md b/inkless-sync/MAIN-SYNC-PROMPT.md deleted file mode 100644 index 477fc7de5f1..00000000000 --- a/inkless-sync/MAIN-SYNC-PROMPT.md +++ /dev/null @@ -1,257 +0,0 @@ -# Agent Prompts for Upstream Sync - -This document provides prompt templates for guiding an AI agent through the upstream sync process. - -## Initial Sync Prompt - -Use this prompt to start a new sync session: - -``` -I need to sync inkless with upstream Apache Kafka. - -Target: [commit SHA, tag, or "trunk" for latest] - -Please follow the sync process documented in inkless-sync/: -1. Create a worktree with branch sync/upstream-YYYYMMDD -2. Preview and categorize conflicts -3. Resolve conflicts following CONFLICT-RESOLUTION-STRATEGY.md -4. Track progress in .inkless-sync/SESSION-YYYY-MM-DD.md using the template -5. Fix compilation errors -6. Run tests - -Reference files: -- inkless-sync/CONFLICT-RESOLUTION-STRATEGY.md -- inkless-sync/MAIN-SYNC-ACTION-PLAN.md -- inkless-sync/MAIN-SYNC-SESSION-TEMPLATE.md - -Start by reading these files and setting up the worktree. -``` - -## Phase-Specific Prompts - -### Phase 1: Setup - -``` -Set up a new sync session targeting [TARGET]. - -1. Create worktree: ../inkless-sync-YYYYMMDD with branch sync/upstream-YYYYMMDD -2. Create .inkless-sync/ directory -3. Copy MAIN-SYNC-SESSION-TEMPLATE.md to .inkless-sync/SESSION-$(date +%Y-%m-%d).md -4. Fetch upstream and identify target commit - -Report the setup status and target commit SHA. -``` - -### Phase 2: Preview Conflicts - -``` -Preview the merge and categorize conflicts. - -1. Run: git merge --no-commit [TARGET] -2. List all conflicting files -3. Categorize each conflict using CONFLICT-RESOLUTION-STRATEGY.md: - - Category 1: Protected (pure inkless files) - - Category 2: Configuration files - - Category 3: Core files with inkless modifications - - Category 4: Files deleted by upstream - - Category 5: Import-only conflicts - -Update the session file in .inkless-sync/ with the conflict summary. -Do NOT complete the merge yet - just preview and categorize. -``` - -### Phase 3: Resolve Protected Files - -``` -Resolve all Category 1 (Protected) conflicts. - -For files matching these patterns, use "ours" (inkless version): -- storage/inkless/** -- docs/inkless/** -- config/inkless/** -- .github/workflows/inkless*.yml - -Commands: -git checkout --ours [file] -git add [file] - -Update the session file in .inkless-sync/ with resolutions. -``` - -### Phase 4: Resolve Configuration Files - -``` -Resolve Category 2 (Configuration) conflicts. - -For each config file (gradle.properties, build.gradle, gradle/dependencies.gradle): -1. Show both versions (ours and theirs) -2. Explain what each side has -3. Propose a merged version that: - - Keeps inkless version string - - Keeps inkless module configuration - - Accepts upstream dependency/plugin versions - -Wait for approval before applying changes. -``` - -### Phase 5: Resolve Core Files - -``` -Resolve Category 3 (Core files with inkless modifications). - -For [FILE_NAME], follow the playbook in MAIN-SYNC-ACTION-PLAN.md: -1. Take upstream version as base: git checkout --theirs [FILE] -2. Identify inkless additions needed (from strategy doc) -3. Apply inkless additions: - - Add imports - - Add constructor parameters - - Add fields - - Add methods -4. Show the diff of changes made -5. Verify syntax compiles - -Wait for approval before moving to next file. -``` - -### Phase 6: Complete Merge - -``` -Complete the merge commit. - -1. Verify all conflicts are resolved: git diff --name-only --diff-filter=U -2. If clean, create merge commit: - git commit -m "merge: apache/kafka trunk [TARGET_INFO]" -3. Show commit summary - -Report any remaining issues. -``` - -### Phase 7: Fix Compilation - -``` -Fix compilation errors. - -1. Run: make build -2. Collect all compilation errors -3. For each error: - - Identify the cause (missing import, API change, etc.) - - Propose a fix - - Apply the fix -4. Re-run build until clean -5. Commit fixes: git commit -m "sync(compile): [description]" - -Update the session file in .inkless-sync/ with error log and fixes. -``` - -### Phase 8: Fix Tests - -``` -Fix test failures. - -1. Run: make test -2. Collect failing tests -3. For each failure: - - Analyze the error - - Identify if it's due to API changes or missing config - - Propose and apply fix -4. Re-run tests until green -5. Commit fixes: git commit -m "sync(test): [description]" - -Update the session file in .inkless-sync/ with test log and fixes. -``` - -### Phase 9: Verify and Report - -``` -Complete verification and generate report. - -1. Run verification checklist: - - make build passes - - make test passes - - Key inkless files exist - - Version preserved in gradle.properties - -2. Generate summary: - - Total conflicts resolved - - Commits created - - Time spent (if tracked) - - Any blockers or issues - -3. Update the session file in .inkless-sync/ with final status - -4. Archive the session file: - mv .inkless-sync/SESSION-*.md inkless-sync/sessions/ - -5. If successful, provide PR description draft. -``` - -## Error Recovery Prompts - -### When Blocked on Conflict - -``` -I'm blocked on resolving [FILE]. - -The conflict is: -[paste conflict markers] - -Inkless needs: [what inkless adds] -Upstream changed: [what upstream changed] - -Please help resolve this by: -1. Explaining what both sides are doing -2. Proposing a merged solution -3. Showing the exact code to use -``` - -### When Compilation Fails - -``` -Compilation failed with these errors: -[paste errors] - -Please: -1. Identify the root cause for each error -2. Explain the upstream API change that caused it -3. Provide the fix for each error -4. Show me the exact edits needed -``` - -### When Tests Fail - -``` -These tests are failing: -[paste test failures] - -Please: -1. Analyze each failure -2. Determine if it's a real bug or test infrastructure issue -3. Propose fixes -4. Show the exact changes needed -``` - -## Continuation Prompt - -If a session is interrupted: - -``` -Continue the sync session on branch [sync/upstream-YYYYMMDD]. - -Current status: -- Phase: [current phase] -- Last completed step: [step] -- Blockers: [if any] - -Read the session file in .inkless-sync/ for full context and continue from where we left off. -``` - -## Best Practices for Agent Usage - -1. **Be specific about targets** - Always provide exact commit SHA or tag -2. **Work incrementally** - Complete one phase before moving to next -3. **Document everything** - Keep the session file in .inkless-sync/ updated -4. **Wait for approval** - On complex merges, show diff before committing -5. **Track blockers** - If stuck, clearly describe the issue -6. **Commit often** - Small, focused commits are easier to review/revert -7. **Restore tests with restored classes** - When restoring a class removed by upstream for inkless functionality, also restore its associated test file to maintain test coverage -8. **Add INKLESS NOTE** - When restoring removed files, add a Javadoc/comment explaining why the file was retained for inkless, with a TODO for future migration diff --git a/inkless-sync/README.md b/inkless-sync/README.md index 3437d3c78ea..b69da6b6a39 100644 --- a/inkless-sync/README.md +++ b/inkless-sync/README.md @@ -11,9 +11,14 @@ The sync process follows the [Versioning Strategy](../docs/inkless/VERSIONING-ST ## AI-Assisted Sync (Recommended) -Start a sync session with Claude Code using the appropriate prompt: -- **Main Sync** → [MAIN-SYNC-PROMPT.md](MAIN-SYNC-PROMPT.md) — weekly/biweekly sync with Apache Kafka trunk -- **Release Sync** → [RELEASE-SYNC-PROMPT.md](RELEASE-SYNC-PROMPT.md) — sync release branches with upstream patches +These scripts are driven by two agent skills (in `.ai-agents/skills/`), which route +the workflows and reference the guides here: +- **[`inkless-upstream-sync`](../.ai-agents/skills/inkless-upstream-sync/SKILL.md)** — main sync (apache/kafka trunk → `main`) and release sync (upstream patch → `inkless-4.x`). +- **[`inkless-release-prep`](../.ai-agents/skills/inkless-release-prep/SKILL.md)** — build a new inkless increment: cherry-pick to release branches, create release branches. + +Reference guides in this directory: +- **Main Sync** → [MAIN-SYNC-ACTION-PLAN.md](MAIN-SYNC-ACTION-PLAN.md) — weekly/biweekly sync with Apache Kafka trunk +- **Release Sync** → [RELEASE-SYNC-GUIDE.md](RELEASE-SYNC-GUIDE.md) — sync release branches with upstream patches - **Cherry-pick Sync** → [CHERRY-PICK-SYNC-GUIDE.md](CHERRY-PICK-SYNC-GUIDE.md) — backport inkless features from main to release branches Context for the agent: @@ -33,6 +38,12 @@ Context for the agent: | `create-release-branch.sh` | Create new inkless release branches | | `cherry-pick-to-release.sh` | Cherry-pick inkless commits to release branches | +Aligning a release branch back to its own expectations (e.g. a main commit that +should not take effect there) is done in history, not via a skip list: cherry-pick +the commit so the branch stays in sync, then add a `sync(revert):` / `sync(align):` +follow-up commit to roll it back. `sync(...)` commits are excluded by +`branch-consistency.sh`, so the branch reads as in sync and the rollback is explicit. + ## Three Types of Sync | Type | Branch | Script | Use Case | @@ -181,11 +192,9 @@ inkless-sync/ ├── cherry-pick-to-release.sh # Cherry-pick commits to releases ├── README.md # This file ├── RELEASE-SYNC-GUIDE.md # Release sync documentation -├── RELEASE-SYNC-PROMPT.md # AI prompt for release syncs ├── CHERRY-PICK-SYNC-GUIDE.md # Cherry-pick sync documentation ├── CHERRY-PICK-SESSION-TEMPLATE.md # Session template for cherry-pick syncs ├── CONFLICT-RESOLUTION-STRATEGY.md # Conflict resolution guidance -├── MAIN-SYNC-PROMPT.md # AI prompt for main branch syncs ├── MAIN-SYNC-SESSION-TEMPLATE.md # Session template for main syncs ├── MAIN-SYNC-ACTION-PLAN.md # Action plan for main syncs ├── RELEASE-SYNC-SESSION-TEMPLATE.md # Session template for release syncs @@ -307,4 +316,5 @@ The structured commit approach makes it easy to: - [Release Sync Guide](RELEASE-SYNC-GUIDE.md) - [Cherry-pick Sync Guide](CHERRY-PICK-SYNC-GUIDE.md) - [Conflict Resolution Strategy](CONFLICT-RESOLUTION-STRATEGY.md) -- [Main Sync Prompt](MAIN-SYNC-PROMPT.md) +- [inkless-upstream-sync skill](../.ai-agents/skills/inkless-upstream-sync/SKILL.md) +- [inkless-release-prep skill](../.ai-agents/skills/inkless-release-prep/SKILL.md) diff --git a/inkless-sync/RELEASE-SYNC-GUIDE.md b/inkless-sync/RELEASE-SYNC-GUIDE.md index 2015b18290f..5a73911195f 100644 --- a/inkless-sync/RELEASE-SYNC-GUIDE.md +++ b/inkless-sync/RELEASE-SYNC-GUIDE.md @@ -313,4 +313,4 @@ make test ## AI-Assisted Sync -For AI-assisted release syncs, see [RELEASE-SYNC-PROMPT.md](RELEASE-SYNC-PROMPT.md) for a ready-to-use prompt. +For AI-assisted release syncs, use the [`inkless-upstream-sync`](../.ai-agents/skills/inkless-upstream-sync/SKILL.md) skill, which routes the release-sync workflow. diff --git a/inkless-sync/RELEASE-SYNC-PROMPT.md b/inkless-sync/RELEASE-SYNC-PROMPT.md deleted file mode 100644 index 4fb354046a6..00000000000 --- a/inkless-sync/RELEASE-SYNC-PROMPT.md +++ /dev/null @@ -1,149 +0,0 @@ -# Release Sync AI Prompt - -Use this prompt with Claude Code to sync an inkless release branch with upstream Apache Kafka releases. - -## Quick Start - -Copy and paste this prompt to start a release sync session: - ---- - -**PROMPT:** - -```` -I need to sync the inkless release branch with an upstream Apache Kafka release. - -## Context -- Release branch: inkless-4.0 (or inkless-4.1, etc.) -- Target: Sync to latest upstream release tag (e.g., 4.0.1, 4.0.2) - -## Steps - -1. **Discovery**: Run `./inkless-sync/release-sync.sh inkless-4.0 --list-tags` to see available upstream tags - -2. **Create worktree**: Create a dedicated worktree for this sync: - ```bash - git worktree add ../inkless-sync-4.0.1 -b inkless-4.0-sync-4.0.1 origin/inkless-4.0 - ``` - -3. **Copy scripts**: Copy sync scripts to worktree: - ```bash - cp -r inkless-sync ../inkless-sync-4.0.1/ - ``` - -4. **Dry run**: Preview conflicts: - ```bash - cd ../inkless-sync-4.0.1 - ./inkless-sync/release-sync.sh inkless-4.0 --to-tag 4.0.1 --branch inkless-4.0-sync-4.0.1 --dry-run - ``` - -5. **Execute sync**: Run the actual sync: - ```bash - ./inkless-sync/release-sync.sh inkless-4.0 --to-tag 4.0.1 --branch inkless-4.0-sync-4.0.1 - ``` - -6. **Resolve conflicts**: When conflicts occur, resolve them following these patterns: - - **Version files**: Use `{upstream_version}-inkless` pattern (e.g., `4.0.1-inkless`) - - **gradle/dependencies.gradle**: Add upstream new deps, verify inkless deps are actually used - - **streams/quickstart POMs**: Keep upstream version (no -inkless suffix) - - **Test files**: Accept upstream changes, keep inkless-specific code - - **.gitignore**: Keep both inkless and upstream entries - -7. **Create session file**: Copy template and track progress: - ```bash - mkdir -p .inkless-sync - cp inkless-sync/RELEASE-SYNC-SESSION-TEMPLATE.md .inkless-sync/RELEASE-SESSION-{branch}-{date}.md - ``` - -8. **Verify build**: - ```bash - make build - make test - ``` - -9. **Archive session**: Move session file to committed history: - ```bash - mv .inkless-sync/RELEASE-SESSION-*.md inkless-sync/sessions/ - ``` - -10. **Push**: When verified, push the sync branch for PR review - -Please help me execute this release sync process. -```` - ---- - -## Conflict Resolution Patterns - -### Version Files - -Files that need `-inkless` suffix: -- `gradle.properties` → `version=4.0.1-inkless` -- `tests/kafkatest/__init__.py` → `__version__ = '4.0.1.inkless'` -- `tests/kafkatest/version.py` → `DEV_VERSION = KafkaVersion("4.0.1-inkless-SNAPSHOT")` -- `docs/js/templateData.js` → `"fullDotVersion": "4.0.1-inkless"` -- `committer-tools/kafka-merge-pr.py` → `DEFAULT_FIX_VERSION = "4.0.1-inkless"` - -### POM Files (Keep Upstream Version) - -These files use standard Apache Kafka versioning: -- `streams/quickstart/pom.xml` -- `streams/quickstart/java/pom.xml` -- `streams/quickstart/java/src/main/resources/archetype-resources/pom.xml` - -### Dependencies - -In `gradle/dependencies.gradle`: -- Add new upstream dependencies -- Verify inkless-specific dependencies are actually used (search for imports) -- Remove unused dependencies that upstream removed - -### Import Organization - -Scala/Java files may have import reorganization conflicts: -- Accept upstream import ordering -- Remove duplicate imports after merge - -## Session File Template - -Use the template at `inkless-sync/RELEASE-SYNC-SESSION-TEMPLATE.md`: - -```bash -mkdir -p .inkless-sync -cp inkless-sync/RELEASE-SYNC-SESSION-TEMPLATE.md .inkless-sync/RELEASE-SESSION-inkless-4.0-$(date +%Y-%m-%d).md -``` - -The template includes sections for: -- Session info (branch, target, status) -- Conflict summary and resolution tracking -- Version file resolutions -- Build verification checklist - -## Verification - -After merge, run: - -```bash -# Build core inkless components -make build - -# Run inkless tests -make test -``` - -Expected: All builds and tests pass. - -## Common Issues - -### Tag not found -```bash -git fetch apache --tags -``` - -### Merge conflicts in test files -Usually import reorganization - accept upstream, remove duplicates. - -### Build fails after merge -Check for: -- Missing dependencies (add to gradle/dependencies.gradle) -- API changes (may need inkless-specific fixes) diff --git a/inkless-sync/branch-consistency.sh b/inkless-sync/branch-consistency.sh index 111964e9805..e2e369ea537 100755 --- a/inkless-sync/branch-consistency.sh +++ b/inkless-sync/branch-consistency.sh @@ -16,6 +16,12 @@ RELEASE_BRANCH="" VERBOSE=false SHOW_MISSING=false SHOW_ALL=false +CHECK=false + +# Set by check_consistency: number of actionable missing commits (newer than the +# last cherry-pick; old below-window commits do not count). Used by --check to +# derive the process exit code. +ACTIONABLE_COUNT=0 usage() { cat <; it does not need + # the upstream (apache/kafka) remote. Skip fetch_upstream in that mode so the + # check runs in CI checkouts that only have 'origin' (no apache/upstream remote). + if [[ "$CHECK" != "true" ]]; then + fetch_upstream + fi git fetch origin --prune check_consistency "$RELEASE_BRANCH" + + # --check turns the actionable-missing count into a process exit code so CI + # can gate on it without parsing the human-readable output. Old (below-window) + # commits are NOT actionable and do not fail the check. + if [[ "$CHECK" == "true" ]]; then + if [[ "$ACTIONABLE_COUNT" -gt 0 ]]; then + echo "" + echo "CHECK FAILED: $RELEASE_BRANCH has $ACTIONABLE_COUNT actionable missing commit(s)." + exit 1 + fi + echo "" + echo "CHECK OK: $RELEASE_BRANCH has no actionable missing commits." + fi } main "$@" diff --git a/inkless-sync/cherry-pick-to-release.sh b/inkless-sync/cherry-pick-to-release.sh index 4fe3eac6ed4..891ff90b850 100755 --- a/inkless-sync/cherry-pick-to-release.sh +++ b/inkless-sync/cherry-pick-to-release.sh @@ -137,6 +137,93 @@ is_applicable_commit() { return 0 } +# Version-carrying files whose content is OWNED by the release branch (set by +# release-sync.sh), never by a cherry-pick from main. A version-bump commit from +# main must still land on the branch (so it is "present" for branch-consistency), +# but it must NOT change the branch's output version. +# +# Note: a bump can apply either WITH a conflict (branch version differs from the +# commit's expected "from" line) or cleanly (they happen to match) -- in the clean +# case there is no conflict to resolve, yet the version is silently changed. So we +# do not rely on conflict resolution; instead we RESTORE these files to the +# branch's pre-pick content after the pick, covering both cases. +# +# Keep this list in sync with what release-sync.sh's update_version_files touches. +VERSION_OWNED_FILES=( + "gradle.properties" + "Makefile" + "docker/examples/docker-compose-files/inkless/.env" +) + +# Print the version-owned files (if any) that a commit modifies. Used by --dry-run +# to report which picks would trigger version restoration, and is purely +# informational (a static diff inspection, no working-tree changes). +commit_touches_version_owned_files() { + local commit="$1" + local changed + changed=$(git show --name-only --format="" "$commit" 2>/dev/null) + local vf + for vf in "${VERSION_OWNED_FILES[@]}"; do + grep -qxF "$vf" <<< "$changed" && echo "$vf" + done +} + +# Restore version-owned files to their pre-pick (HEAD-before-pick) content and, if +# that changed the tree, amend the just-created cherry-pick commit. Args: the +# pre-pick commit SHA (HEAD before the pick). Emits a note when it restores. +restore_version_owned_files() { + local prepick="$1" + local restored=false + local f + for f in "${VERSION_OWNED_FILES[@]}"; do + # Only touch files that exist at the pre-pick HEAD. + git cat-file -e "${prepick}:${f}" 2>/dev/null || continue + # If the current file differs from the pre-pick version, restore it. + if ! git diff --quiet "${prepick}" -- "$f" 2>/dev/null; then + git checkout "$prepick" -- "$f" + git add -- "$f" + echo " ↳ kept branch version (restored $f)" + restored=true + fi + done + if [[ "$restored" == "true" ]]; then + # Fold the restore into the cherry-pick commit; keep its message. + GIT_EDITOR=true git commit --amend --no-edit >/dev/null 2>&1 + fi +} + +# If the in-progress cherry-pick's ONLY conflicts are version-owned files, resolve +# them to "ours", stage, and continue. Returns 0 if fully resolved, 1 otherwise +# (leaving the conflict in place for manual handling). +try_autoresolve_version_conflict() { + local conflicted + conflicted=$(git diff --name-only --diff-filter=U) + + # No conflicts recorded -> nothing to auto-resolve here. + [[ -z "$conflicted" ]] && return 1 + + # Every conflicted path must be version-owned; otherwise there is a real + # conflict and we must not mask it. + local f vf owned + while IFS= read -r f; do + owned=false + for vf in "${VERSION_OWNED_FILES[@]}"; do + [[ "$f" == "$vf" ]] && { owned=true; break; } + done + [[ "$owned" != "true" ]] && return 1 + done <<< "$conflicted" + + # All conflicts are version-owned: keep the branch's version (ours). + while IFS= read -r f; do + git checkout --ours -- "$f" + git add -- "$f" + echo " ↳ kept branch version for version-owned file: $f" + done <<< "$conflicted" + + # Continue the cherry-pick with the resolved tree (keep the original message). + GIT_EDITOR=true git cherry-pick --continue >/dev/null 2>&1 +} + # Cherry-pick a single commit with error handling cherry_pick_commit() { local commit="$1" @@ -178,31 +265,44 @@ cherry_pick_commit() { esac fi + local prepick + prepick=$(git rev-parse HEAD) + if git cherry-pick "$commit"; then + # Versions are branch-owned (set only by upstream sync / release-sync.sh). + # A pick may change a version-owned file with NO conflict, so restore + # unconditionally after a clean apply too. + restore_version_owned_files "$prepick" echo " ✅ Successfully cherry-picked" return 0 else + # Version-owned files (gradle.properties/Makefile/.env) are set by + # release-sync, not by picks. If they are the ONLY conflict, keep the + # branch's version and continue so the commit still lands. + if try_autoresolve_version_conflict; then + restore_version_owned_files "$prepick" + echo " ✅ Cherry-picked (version kept at branch value)" + return 0 + fi + echo " ⚠️ Cherry-pick failed (conflict or other issue)" echo "" - echo " Options:" + echo " The conflicted state has been left in place so you can resolve it in" + echo " dependency order. Do NOT skip ahead: applying later commits before this" + echo " one is resolved can silently break ordering." + echo "" + echo " To continue:" echo " 1. Resolve conflicts, then: git cherry-pick --continue" - echo " 2. Abort this cherry-pick (you can re-run this script to continue): git cherry-pick --abort" + echo " 2. Re-run this script to cherry-pick the remaining commits." + echo " (auto-detect mode skips the resolved commit automatically; with" + echo " explicit hashes, pass only the ones not yet applied.)" echo "" - - if [[ "$INTERACTIVE" == "true" ]]; then - read -p " Abort and continue with next? [Y/n] " -n 1 -r - echo "" - if [[ ! "$REPLY" =~ ^[Nn]$ ]]; then - git cherry-pick --abort 2>/dev/null || true - return 1 - else - echo " Leaving in conflicted state for manual resolution" - exit 1 - fi - else - git cherry-pick --abort 2>/dev/null || true - return 1 - fi + echo " Or, to bail out entirely: git cherry-pick --abort" + echo "" + # Signal the caller to STOP the run (return code 2), leaving the conflict + # in place. Never auto-abort-and-continue: that reorders the remaining + # picks relative to their dependencies. + return 2 fi } @@ -211,7 +311,11 @@ main() { parse_args "$@" require_git_repo - require_clean_worktree + # Dry-run is read-only: it inspects commits and prints a plan but never + # touches the working tree, so it does not require a clean worktree. + if [[ "$DRY_RUN" != "true" ]]; then + require_clean_worktree + fi fetch_upstream git fetch origin --prune @@ -276,6 +380,13 @@ main() { applicable="⚠ (merge resolution - may not apply)" fi echo "- $commit $msg $applicable" + # Flag commits that touch version-owned files: on a real run their version + # changes are restored to the branch value after the pick. + local touched + touched=$(commit_touches_version_owned_files "$commit") + if [[ -n "$touched" ]]; then + echo " ↳ touches version-owned file(s); branch version will be kept: $(echo "$touched" | tr '\n' ' ')" + fi done if [[ "$DRY_RUN" == "true" ]]; then @@ -303,9 +414,15 @@ main() { # Cherry-pick each commit local success_count=0 local skip_count=0 - local fail_count=0 + local stopped_on="" + local remaining=0 for commit in "${ordered[@]}"; do + if [[ -n "$stopped_on" ]]; then + remaining=$((remaining + 1)) + continue + fi + if ! is_applicable_commit "$commit"; then echo "" echo "Skipping merge resolution commit: $commit" @@ -313,10 +430,13 @@ main() { continue fi - if cherry_pick_commit "$commit"; then + cherry_pick_commit "$commit" && rc=0 || rc=$? + if [[ $rc -eq 0 ]]; then success_count=$((success_count + 1)) else - fail_count=$((fail_count + 1)) + # rc 2 (conflict) or any other failure: stop immediately, leaving the + # conflict in place, so the remaining picks keep their dependency order. + stopped_on="$commit" fi done @@ -327,11 +447,14 @@ main() { echo "|--------|-------|" echo "| Successfully cherry-picked | $success_count |" echo "| Skipped (merge resolution) | $skip_count |" - echo "| Failed (conflicts) | $fail_count |" + echo "| Stopped at (conflict) | ${stopped_on:-none} |" + echo "| Remaining (not attempted) | $remaining |" echo "" - if [[ $fail_count -gt 0 ]]; then - echo "⚠️ Some commits failed to cherry-pick. Manual resolution may be needed." + if [[ -n "$stopped_on" ]]; then + echo "⛔ Stopped at $stopped_on due to a conflict. Resolve it (git cherry-pick" + echo " --continue), then re-run this script to apply the $remaining remaining commit(s)." + return 1 elif [[ $success_count -gt 0 ]]; then echo "✅ Cherry-pick complete" echo ""