Skip to content

troubleshooting.md

Latest commit

 

History

History
195 lines (139 loc) · 5.91 KB

troubleshooting.md

File metadata and controls

195 lines (139 loc) · 5.91 KB
Raw

Troubleshooting

Common problems

Permission denied

Almost every command needs root.

sudo solo-provisioner block node install --profile=local

Exceptions: solo-provisioner version, and block node reconcile-shaper --check.

profile flag is required

Every block node command needs --profile.

sudo solo-provisioner block node check --profile=mainnet

kube cluster install is the exception — it takes no profile. See Deployment profiles.

invalid base path

The storage path must exist and be writable.

sudo mkdir -p /mnt/storage
sudo solo-provisioner block node install --profile=mainnet --base-path=/mnt/storage

To change a storage path on an existing install you also need --purge-storage — a local PV's hostPath.path is immutable, so the PV/PVCs are deleted and recreated at the new paths. It is accepted by reconfigure (re-applies chart values at the current version), upgrade (same move plus a version bump) and reset (no Helm operation at all). Without it, a path change is rejected. See reconfigure.

Helm chart problems

Pin the version explicitly instead of resolving the latest:

sudo solo-provisioner block node install --profile=mainnet --chart-version=0.22.1

If the install times out, raise --timeout. The default is 5m0s, and exceeding it rolls the operation back.

Alloy fails: cluster not reachable

Alloy deploys into an existing cluster; it never creates one. Install the cluster first:

sudo solo-provisioner kube cluster install

Locked out of SSH after enabling the firewall

The host firewall is default-drop. An empty management allowlist admits nobody.

  • block node install --firewall-enabled with an empty --mgmt-cidrs skips the firewall on purpose, exactly to avoid this.
  • network firewall create with no --mgmt-cidrs does not skip it — it renders an empty allowlist. Always pass --mgmt-cidrs.

Recovery needs console access. Then:

sudo solo-provisioner network firewall add --name mgmt --cidr <your-cidr>

Corrupt firewall config

Every apply keeps the previous generation. Restore it and re-apply:

sudo cp /etc/solo-provisioner/network-weaver-host-firewall.yaml.prev \
        /etc/solo-provisioner/network-weaver-host-firewall.yaml
sudo solo-provisioner network firewall reapply

Full detail: Recovering a corrupt config.

Traffic shaping is on but nothing is being shaped

Check in this order:

  1. Is the daemon running?

    sudo solo-provisioner daemon service check

  2. Do the policies exist, and do their sets have members?

    sudo solo-provisioner network policy show

    Sets start empty and are filled by the daemon from the block node's statusz. An empty set has two different causes, and they are not the same problem:

    • Statusz has not answered yet — first boot, or a reboot before the first poll. nft set membership is not boot-persistent, so it is rebuilt on the first successful poll.

    • Statusz reported peers, but none in this category. A call that returns endpoints but no usable one for a category clears that category's set, and the daemon logs it:

      journalctl -u solo-provisioner-daemon -g TrafficShaperMembershipCleared

    A call that returns 200 with nothing in it (activeEndpoints: []) does not empty anything — every set that call feeds is left alone. Neither does a poll that fails (statusz unreachable, or an apply error): it leaves the last-good membership in place and retries on the next tick. So an empty set in steady state points at a statusz response that reports peers but not that category, not at a connectivity problem or an empty response. Check the endpoint the daemon is actually polling with --statusz-base-url, then read it yourself:

    solo-provisioner block node reconcile-shaper --statusz-url=<url> --check

    --check needs no privilege and touches no nft state — it just prints what the daemon would apply. See What an empty statusz response does.

  3. Is traffic actually landing in a class?

    sudo solo-provisioner network shape watch --device egress --iface enp0s1

    A non-zero rate against a class means classification is working.

  4. Is ingress shaping attached to the pod? Ingress lives on the pod's host-side veth and is attached per-pod by the daemon. Find the veth with ip link, then:

    sudo solo-provisioner network shape watch --device ingress --iface lxc1a2b3c

Rollback did not seem to run

--rollback-on-error is not shown in the TUI. Confirm it another way:

helm list -A

or read the workflow report YAML whose path is printed as report_path=….

Getting more detail

Expanded output

sudo solo-provisioner block node install --profile=local --verbose

Debug logs

sudo solo-provisioner block node install --profile=local --log-level=debug

or in the config file:

log:
  level: debug
  consoleLogging: true

Raw logs, no TUI

Useful in CI, or when the TUI is hiding something:

sudo solo-provisioner block node install --profile=local --non-interactive

Machine-readable output

sudo solo-provisioner block node install --profile=local -o json \
  | jq 'select(.type=="summary")'

Help

solo-provisioner --help
solo-provisioner block --help
solo-provisioner block node --help
solo-provisioner block node install --help

Still stuck? Open an issue at hashgraph/solo-weaver, and include the workflow report YAML named in the report_path=… line.