-
Notifications
You must be signed in to change notification settings - Fork 93
Support dynamic IaC for integrations #1209
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jeniawhite
wants to merge
2
commits into
elastic:main
Choose a base branch
from
jeniawhite:evgb-DynIaC
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| ## | ||
| ## Describes the layout of the blueprints/ payload folder. | ||
| ## | ||
| ## Files are laid out to mirror the blueprint `id` (<provider>/<trust-model>/<scope>) | ||
| ## with the `format` and file extension appended: | ||
| ## | ||
| ## blueprints/<provider>/<trust-model>/<scope>.<format>.<ext> | ||
| ## | ||
| ## e.g. blueprints/aws/federated-identity/account.cloudformation.json | ||
| ## | ||
| spec: | ||
| additionalContents: false | ||
| contents: | ||
| - description: Provider directory (e.g. aws, azure, gcp) | ||
| type: folder | ||
| pattern: '^[a-z0-9][a-z0-9-]*$' | ||
| required: true | ||
| additionalContents: false | ||
| contents: | ||
| - description: Trust-model directory (e.g. federated-identity, workload-identity, service-principal) | ||
| type: folder | ||
| pattern: '^[a-z0-9][a-z0-9-]*$' | ||
| required: true | ||
| additionalContents: false | ||
| contents: | ||
| - description: A canonical IaC blueprint file, named <scope>.<format>.<ext> | ||
| type: file | ||
| pattern: '^[a-z0-9][a-z0-9_-]*\.[a-z0-9-]+\.[a-z0-9]+$' | ||
| required: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| ## | ||
| ## Describes the specification for the blueprint package's main manifest.yml file | ||
| ## | ||
| spec: | ||
| # Everything under here follows JSON schema (https://json-schema.org/), written as YAML for readability | ||
| type: object | ||
| additionalProperties: false | ||
| properties: | ||
| format_version: | ||
| description: The version of the package specification format used by this package. | ||
| $ref: "../integration/manifest.spec.yml#/definitions/version" | ||
| name: | ||
| description: The name of the package. | ||
| type: string | ||
| pattern: '^[a-z0-9_]+$' | ||
| examples: | ||
| - elastic_iac_blueprints | ||
| title: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/title" | ||
| description: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/description" | ||
| version: | ||
| description: The version of the package. | ||
| $ref: "../integration/manifest.spec.yml#/definitions/version" | ||
| source: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/source" | ||
| type: | ||
| description: The type of package. | ||
| type: string | ||
| enum: | ||
| - blueprint | ||
| examples: | ||
| - blueprint | ||
| conditions: | ||
| description: Conditions under which this package can be installed. | ||
| type: object | ||
| additionalProperties: false | ||
| properties: | ||
| elastic: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/conditions/properties/elastic" | ||
| kibana: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/conditions/properties/kibana" | ||
| owner: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/owner" | ||
| deprecated: | ||
| $ref: "../integration/manifest.spec.yml#/definitions/deprecated" | ||
| required: | ||
| - format_version | ||
| - name | ||
| - title | ||
| - description | ||
| - version | ||
| - type | ||
| - owner | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| ## | ||
| ## Entrypoint of "blueprint packages" specification. | ||
| ## | ||
| ## A blueprint package is a versioned bundle of canonical Infrastructure-as-Code (IaC) | ||
| ## base templates ("blueprints"). Unlike other package types it is not installed into | ||
| ## Kibana, Elasticsearch, or the Elastic Agent — it is fetched by the cloud-iac-provisioner | ||
| ## renderer, which applies each enabled integration's RFC 6902 `iac_blueprints` patches on | ||
| ## top of the canonical blueprint to produce one deployable artifact. | ||
| ## | ||
| ## Describes the folders and files that make up a package. | ||
| ## | ||
| spec: | ||
| additionalContents: false | ||
| totalContentsLimit: 65535 | ||
| totalSizeLimit: 250MB | ||
| sizeLimit: 150MB | ||
| configurationSizeLimit: 5MB | ||
| relativePathSizeLimit: 3MB | ||
| contents: | ||
| - description: The main package manifest file | ||
| type: file | ||
| contentMediaType: "application/x-yaml" | ||
| sizeLimit: 5MB | ||
| name: "manifest.yml" | ||
| required: true | ||
| $ref: "./manifest.spec.yml" | ||
| - description: The package's CHANGELOG file | ||
| type: file | ||
| contentMediaType: "application/x-yaml" | ||
| name: "changelog.yml" | ||
| required: true | ||
| $ref: "../integration/changelog.spec.yml" | ||
| - description: The package's license file | ||
| type: file | ||
| contentMediaType: "text/plain" | ||
| name: "LICENSE.txt" | ||
| required: false | ||
| - description: Folder containing documentation for the package | ||
| type: folder | ||
| name: docs | ||
| required: true | ||
| $ref: "../integration/docs/spec.yml" | ||
| - description: Folder containing the canonical IaC base blueprints served to the renderer | ||
| type: folder | ||
| name: blueprints | ||
| required: true | ||
| $ref: "./blueprints/spec.yml" | ||
| - description: Configuration file to process the results returned from the package validation. This file is just for package validation and it should be ignored when installing or using the package. | ||
| type: file | ||
| contentMediaType: "application/x-yaml" | ||
| name: "validation.yml" | ||
| required: false | ||
| $ref: "../integration/validation.spec.yml" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you need a separate blueprint package type? Isn't it enough to allow input and integration packages to define blueprints?
I would expect that the permissions and other definitions in a blueprint would need to be tied to specific integration versions, so it seems preferable to bundle them into the existing package types, rather than have a separate package the likely introduces the need for version compatibility rules.
It seems much nicer to be able to test the AWS package as a complete unit for example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CC @mrodm @teresaromero
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmacknz How do we deal if there are like multiple, like 20 odd AWS integrations that need same blueprint , If we limit the blueprint inside one integration then probably we would have to duplicate it for other similar integrations?
The purpose was to have one place of truth for all these blueprints that could be used by multiple packages. Reusing
input/integrationtypes might causes problems in Fleet UI if we do not have any Kibana/ES assets probably?If the naming needs to be generic , like to reuse the package type for any general purpose template / prompt / text / asset storage mechanism then we can use more generic / base package naming convention.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needing to reuse these is valid, but creating a separate package type creates a compatibility check problem as I mentioned so if reuse is the only reason for a separate package we maybe need another way to reuse configuration in packages.
It should be possible for us to create a way to reuse configuration across integrations without creating a new releasable user artifact. We should not be publishing packages whose main purpose is just config reuse in other packages.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i was thinking on this re-use and had the links idea since the last meeting, so i went ahead and checked this out.
The proposal: use elastic-package .link files to share the canonical blueprint asset across integration packages.
The .link mechanism already exists — a .link file is a plain text pointer ( ) that elastic-package resolves transparently at build time. The built package zip contains the real file content, not the pointer. elastic-package links update/check keeps checksums in sync and fails the build on drift.
I ran a POC on the elastic/integrations repo to verify this approach: elastic/integrations#20721
Where to store the shared source:
What this avoids: no new type: blueprint in the spec, no EPR changes, no new elastic-package support for a new package type. The canonical template is just a file — shared via a mechanism the toolchain already understands.
@bhapas The iac folder will contain "blueprint" + patches; would the service that patches permissions be able to grab from this folder all the files?
@mrodm will this be a challenge for backporting a package? having a linked file source on root repo? would it be better to have it on a package and link the other from there?
from this we could also update elastic-package so it can simulate the permissions patching and have this asset tested during development; as we do now with policies with fleet (thinking out loud)
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@teresaromero Yes, I think linking files between different packages is going to be problematic when doing backports.
Currently, we suggest that new backport branches should be created just with the required package and remove the others. If the linked file is going to be located in another package, when creating the backport branch it is likely that the linked file does not exist. In that case one option is that the owners should copy the source of the linked file from the commit where the backport branch was created.
In order to support that, scripts that copy the contents of the backport branches should also to take into account the source of the linked files to ensure that any other necessary package is kept. And this is not considered currently in this process.
If the linked file is in the root of the repository could also cause some issues. For instance, when a change in that blueprint folder in the root of the repository is introduced, it is not going to be easy to know to which packages affect and which branches should be backported. If possible, I would try to avoid this (thinking in backports).
EDIT:
And for instance, changes in the packages containing the source of the links will not be suggested as part of the backport process automation.
It is difficult to ensure that the backport process automation would take care of all these scenarios.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created issue to ensure that the source packages of the linked files are copied too elastic/integrations#20726
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created issue to ensure that the source packages of the linked files are copied too elastic/integrations#20726
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmacknz Agree that we should not be publishing packages that just package a reusable config for other [ specific set of ] packages.
@teresaromero
This sounds good. Except that I have a few questions around this.
Yes , it should be possible if we can read the specific resources from the gcloud bucket.
For backport branches , specific package versions can have specific base template instead of a link. That way we work with the specific template manually in that backport branch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The link concept solves this, the backporting issues can be solved through the tooling updates Mario suggested hopefully. One day might just have to commit to maintaining backport branches of the entire integrations repository if this is troublesome.
I don't think the link concept prevents anything related to breaking changes or multiple versions of a blueprint at once. You can change what the links point to, have current and next versions of a blueprint etc. they would just all coexist on the main branch. Potentially you might want a version
Maybe we need to iterate on the ergonomics of this a bit but conceptually this direction makes sense to me.