ValueReferences to resources created by dependent Fluxtomizations cannot be resolved

[teeuwen]

I have a Fluxtomization referencing a Kustomization creating a ConfigMap. Then I have another Fluxtomization depending on the first which creates a HelmRelease. In that HelmRelease, I am unable to reference the aforementioned ConfigMap with valuesFrom .
For context, what I'm trying to achieve here is a setup where I have multiple clusters running the same apps, but with different Helm chart values per cluster (also see example below).

I believe the reason this is not working is because the lookup flux-local does only uses the Fluxtomization it is currently "reconciling" as context, as can be seen here.

I have tried to fiddle with fixing this myself, but have not found a clean solution yet. I think ideally flux-local should walk through the tree of dependent Fluxtomization resources and resolve references to ConfigMaps and Secrets that way. I was not able to determine if flux-local already builds a tree of Fluxtomization dependencies. I was hoping someone else could have a look at this, also because of unfamiliarity with the codebase. Then again, maybe I am missing something really obvious here. Thanks :)

Tested using flux-local:main docker image (commit 1bb1330).

Debug Logs

$ flux-local --log-level DEBUG build hr -A
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:flux_local.tool.selector:Building HelmRelease selector from args: {'log_level': 'DEBUG', 'command': 'build', 'cls': <class 'flux_local.tool.build.BuildHelmReleaseAction'>, 'helmrelease': None, 'path': None, 'sources': None, 'all_namespaces': True, 'namespace': 'flux-system', 'label_selector': None, 'skip_crds': True, 'skip_secrets': True, 'skip_kinds': None, 'skip_invalid_kustomization_paths': False, 'kustomize_build_flags': '', 'kube_version': None, 'api_versions': None, 'registry_config': None, 'skip_invalid_helm_release_paths': True}
DEBUG:flux_local.git_repo:Processing cluster with selector ResourceSelector(path=PathSelector(path=None, sources=None), cluster=MetadataSelector(enabled=True, name='flux-system', namespace='flux-system', label_selector=None, skip_crds=True, skip_secrets=True, skip_kinds=None, visitor=None), kustomization=MetadataSelector(enabled=True, name=None, namespace=None, label_selector=None, skip_crds=True, skip_secrets=True, skip_kinds=None, visitor=ResourceVisitor(func=<bound method ContentOutput.call_async of <flux_local.visitor.ContentOutput object at 0x7143941d8d70>>)), helm_repo=MetadataSelector(enabled=True, name=None, namespace=None, label_selector=None, skip_crds=True, skip_secrets=True, skip_kinds=None, visitor=ResourceVisitor(func=<function HelmVisitor.repo_visitor.<locals>.add_repo at 0x7143941f3ba0>)), helm_release=MetadataSelector(enabled=True, name=None, namespace=None, label_selector=None, skip_crds=True, skip_secrets=True, skip_kinds=None, visitor=ResourceVisitor(func=<function HelmVisitor.release_visitor.<locals>.add_release at 0x7143941f3ce0>)), oci_repo=MetadataSelector(enabled=True, name=None, namespace=None, label_selector=None, skip_crds=True, skip_secrets=True, skip_kinds=None, visitor=ResourceVisitor(func=<function HelmVisitor.repo_visitor.<locals>.add_repo at 0x7143941f3b00>)), doc_visitor=None)
DEBUG:flux_local.context:[Trace] > Cluster 'None'
DEBUG:git.util:sys.platform='linux', git_executable='git'
DEBUG:git.cmd:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/teeuwen/minimalrepro, stdin=None, shell=False, universal_newlines=False)
DEBUG:flux_local.git_repo:Visiting path (None) .
DEBUG:flux_local.context:[Trace] > Cluster 'None' > Kustomization '.'
DEBUG:flux_local.command:Running command: (. (abs)) kustomize cfg grep 'kind=^(Kustomization|ConfigMap|Secret)$' .
DEBUG:flux_local.command:Running command: kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(ConfigMap|Secret)$'
DEBUG:flux_local.context:[Trace] < Cluster 'None' > Kustomization '.' (0.02s)
DEBUG:flux_local.git_repo:Found 2 new Kustomizations
DEBUG:flux_local.git_repo:Kustomization 'podinfo-config' sourceRef.kind 'GitRepository' of 'flux-system'
DEBUG:flux_local.git_repo:Kustomization 'podinfo' sourceRef.kind 'GitRepository' of 'flux-system'
DEBUG:flux_local.git_repo:Visiting path (None) clusters/prod/podinfo-config
DEBUG:flux_local.context:[Trace] > Cluster 'None' > Kustomization 'flux-system/podinfo-config'
DEBUG:flux_local.git_repo:Visiting path (None) apps/podinfo
DEBUG:flux_local.context:[Trace] > Cluster 'None' > Kustomization 'flux-system/podinfo'
DEBUG:flux_local.command:Running command: flux build ks podinfo --dry-run --kustomization-file /dev/stdin --path /home/teeuwen/minimalrepro/apps/podinfo --namespace flux-system
DEBUG:flux_local.command:Running command: flux build ks podinfo-config --dry-run --kustomization-file /dev/stdin --path /home/teeuwen/minimalrepro/clusters/prod/podinfo-config --namespace flux-system
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(Kustomization|ConfigMap|Secret)$'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(Kustomization|ConfigMap|Secret)$'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'spec.sourceRef.kind=GitRepository|OCIRepository'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(ConfigMap|Secret)$'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(ConfigMap|Secret)$'
DEBUG:flux_local.context:[Trace] < Cluster 'None' > Kustomization 'flux-system/podinfo' (0.04s)
DEBUG:flux_local.context:[Trace] < Cluster 'None' > Kustomization 'flux-system/podinfo-config' (0.05s)
DEBUG:flux_local.git_repo:Found 0 new Kustomizations
DEBUG:flux_local.git_repo:Kustomization flux-system/podinfo waiting for {'flux-system/podinfo-config'}
DEBUG:flux_local.git_repo:Processing kustomization 'podinfo-config': clusters/prod/podinfo-config
DEBUG:flux_local.context:[Trace] > Cluster 'None' > Build 'flux-system/podinfo-config'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(CustomResourceDefinition|Secret)$' --invert-match
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(ConfigMap|HelmRepository|OCIRepository|HelmRelease|Secret)$'
DEBUG:flux_local.context:[Trace] < Cluster 'None' > Build 'flux-system/podinfo-config' (0.01s)
DEBUG:flux_local.git_repo:Processing kustomization 'podinfo': apps/podinfo
DEBUG:flux_local.context:[Trace] > Cluster 'None' > Build 'flux-system/podinfo'
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(CustomResourceDefinition|Secret)$' --invert-match
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(ConfigMap|HelmRepository|OCIRepository|HelmRelease|Secret)$'
DEBUG:flux_local.context:[Trace] < Cluster 'None' > Build 'flux-system/podinfo' (0.01s)
DEBUG:flux_local.values:Expanding value reference ValuesReference(kind='ConfigMap', name='podinfo-values', values_key='values.yaml', target_path=None, optional=False)
DEBUG:flux_local.values:Expanding value reference ValuesReference(kind='ConfigMap', name='podinfo-values', values_key='values.yaml', target_path=None, optional=False)
WARNING:flux_local.values:Unable to find ConfigMap podinfo/podinfo-values referenced
DEBUG:flux_local.context:[Trace] < Cluster 'None' (0.09s)
DEBUG:flux_local.visitor:Inflating Helm charts in cluster
DEBUG:flux_local.helm:Updating 1 repositories
DEBUG:flux_local.visitor:Waiting for inflate tasks to complete
DEBUG:flux_local.command:Running command: helm template podinfo oci://ghcr.io/stefanprodan/charts/podinfo --namespace podinfo --repository-cache /tmp/tmprim4cnyy --repository-config /tmp/tmptworu52i/repository-config.yaml --registry-config /tmp/tmpag2ftd_i.json --skip-crds --skip-tests --version 6.5.0
DEBUG:flux_local.command:Running command: kustomize cfg grep 'kind=^(CustomResourceDefinition|Secret)$' --invert-match

[Rendered manifest output omitted for brevity]

Minimal setup to reproduce

# /apps/podinfo/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo
resources:
  - helm-repository.yaml
  - helm-release.yaml

# /apps/podinfo/helm-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: podinfo
spec:
  type: oci
  interval: 5m
  url: oci://ghcr.io/stefanprodan/charts

# /apps/podinfo/helm-release.yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
spec:
  chart:
    spec:
      chart: podinfo
      version: 6.5.0
      sourceRef:
        kind: HelmRepository
        name: podinfo
  releaseName: podinfo
  valuesFrom:
    - kind: ConfigMap
      name: podinfo-values

# /clusters/prod/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo-config
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./clusters/prod/podinfo-config
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  dependsOn:
    - name: podinfo-config
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/podinfo
  prune: true
  wait: true

# /clusters/prod/podinfo-config/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo
resources:
  - namespace.yaml
  - values.yaml

# /clusters/prod/podinfo-config/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: podinfo

# /clusters/prod/podinfo-config/values.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: podinfo-values
data:
  values.yaml: |
    redis:
      enabled: true
      repository: public.ecr.aws/docker/library/redis
      tag: 7.0.6

EDIT:

I have also tried to see what happens with flux-local shell, and it indeed is able to resolve the ConfigMap (as I guess it simulates the behavior of Flux's kustomize-controller):

flux-local --log-level DEBUG shell
...
INFO:flux_local.helm_controller.controller:Waiting for dependencies: ['HelmRepository/podinfo/podinfo exists', 'ConfigMap/podinfo/podinfo-values exists']
...
INFO:flux_local.helm_controller.controller:All dependencies of podinfo/podinfo are ready: {NamedResource(kind='HelmRepository', namespace='podinfo', name='podinfo'), NamedResource(kind='ConfigMap', namespace='podinfo', name='podinfo-values')}
...

However, it doesn't actually seem to use the values from the ConfigMap in the generated resources, as (with the podinfo example above), no Deployment is created for Redis. When I paste the values directly into the HelmRelease (under spec.values), the Redis Deployment is created.

[allenporter]

Yeah i see what you mean. I see when we do post build substitutions we do reference all clusters:

flux-local/flux_local/git_repo.py

Line 792 in 1bb1330

values.ks_cluster_config(cluster.kustomizations),

The flux-local shell is a rewrite that is meant to fix the gaps like this in the original version. It does not yet have feature parity with the original command line.