Add code_source block in ai_runtime_task to provide cleaner UX

[ben-hansen-db]

Changes

Adds a DABs-native code_source block to ai_runtime_task, giving train.yaml
parity for code delivery inside a bundle:

ai_runtime_task:
  experiment: my-exp
  code_source:
    root_path: ./src
    include_paths: [src, configs]
    git: { commit: abc123 }   # or { branch: main }

At bundle deploy, the CLI packages the directory into a content-addressed tarball
(honoring .gitignore + sync include/exclude, narrowing to include_paths, or
archiving a pinned git branch/commit), uploads it via normal file sync, and sets
the SDK field ai_runtime_task.code_source_path. No proto/API change.

How it works

code_source is not a field on the SDK jobs.AiRuntimeTask, and config
normalization drops unknown fields. A pre-normalize rewrite
(rewriteAiRuntimeCodeSource in bundle/config/root.go) extracts each block into an
internal Root.AiRuntimeExtras stash keyed by (job, task_key) so it survives the
task reordering MergeJobTasks does. The build-phase aicode mutator lowers it into
code_source_path, reusing the existing overlay-on-sync-root packaging. The string
form code_source_path: ./dir is unchanged; the two are mutually exclusive per task.

Scope / follow-ups

• remote_volume is rejected as not-yet-supported (needs a deploy-phase Volume upload
  • a bundle destroy cleanup decision). Fast-follow, not a blocker.
• code_source under a targets.* override is rejected (stash can't represent
  per-target overrides).
• Updating air convert-to-dabs to emit this block is a separate PR.

Testing

Tested e2e via

bundle:
  name: ai-runtime-code-source-demo

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://dbc-559ffd80-2bfc.cloud.databricks.com

resources:
  jobs:
    train:
      name: "[${bundle.target}] AI Runtime training (code_source demo)"
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: my-training
            code_source:
              root_path: ./src
              git:
                branch: master          # <-- or `commit: <sha>`
            deployments:
              - command_path: src/command.sh
                compute:
                  accelerator_type: GPU_1xA10
                  accelerator_count: 1
      environments:
        - environment_key: default
          spec:
            environment_version: "5"
            dependencies:
              - torch>=2.0.0

Unit tests (rewrite, validation, filterIncludePaths) plus acceptance tests under
acceptance/bundle/ai_runtime_task/ (code_source_block, code_source_include_paths,
code_source_git, code_source_errors) â�� passing on both terraform and direct.

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

22 files changed
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

/bundle/ - needs approval

10 files changed
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/ai-runtime-code-source-block.md, pyrefly.toml, python/codegen/codegen/jsonschema.py
Based on git history:

• @janniklasrose -- recent work in bundle/config/, bundle/schema/, .nextchanges/bundles/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: 0bb32f3

Run: 31432857349

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	294	1124	5:21
💚​	aws windows	4	4	296	1122	3:41
💚​	azure linux	4	4	293	1124	5:09
💚​	azure windows	4	4	295	1122	4:02
💚​	gcp linux	1	5	294	1124	5:38
💚​	gcp windows	1	5	296	1122	4:25

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:20	gcp windows	TestAccept
3:56	azure windows	TestAccept
3:32	aws windows	TestAccept
3:09	gcp linux	TestAccept
3:08	azure linux	TestAccept
3:00	aws linux	TestAccept