Skip to content

Add script-based SONiC cell runtime - #16

Merged
hardikdr merged 4 commits into
mainfrom
enh/fabriclet
Oct 19, 2026
Merged

hardikdr merged 4 commits into
mainfrom
enh/fabriclet

Conversation

@afritzler

@afritzler afritzler commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Proposed Changes

This adds a first working cellruntime.Runtime for SONiC switches (sonic/). A fabriclet runs on the device itself and provisions it once a Cell gets bound to its Node. For now the provisioning is done by the ZTP bash scripts we already have; a follow-up will talk to SONiC DB directly.

How it works:

  • On bind, ApplyCell takes the template for the switch role (sonic/templates/{inband-leaf,inband-spine}.sh),
    fills in the cell ID, region, IPv6 base and search domain, writes the result to a fresh temp file and runs it with bash. Exit 0 means configured, anything else is a terminal error and the Cell goes to Failed. I made failures terminal on purpose: a script that died halfway leaves partial config, and plain retries would just fail the same way again. Non-numeric cell IDs are rejected before anything runs, because bash arithmetic treats them as 0 and the script would exit 0 after writing garbage.
  • CellStatus just checks for /etc/sonic/fabric-cell.done, the flag the script touches at the end. Flag present → Active. Missing → the cell gets applied.
  • DeleteCell runs templates/reset.sh, a noop for now. That is where the device reset (ONIE re-install) goes later.
  • Interface admin-state handling is a noop, since that feature is leaving the runtime interface anyway. Oper state still comes from sysfs.

Running on a device

The fabriclet ships as a container image (ghcr.io/ironcore-dev/fabriclet-sonic:<tag>). On the switch:

docker pull ghcr.io/ironcore-dev/fabriclet-sonic:<tag>
docker run --rm -ti \
  --network=host --pid=host --privileged \
  -v /etc/sonic:/etc/sonic \
  -v /etc/sonic/fabriclet.kubeconfig:/etc/sonic/fabriclet.kubeconfig:ro \
  -e KUBECONFIG=/etc/sonic/fabriclet.kubeconfig \
  ghcr.io/ironcore-dev/fabriclet-sonic:<tag> \
  --name=swi1-ab-42 \
  --backend=script \
  --role=inband-leaf \
  --region=ab \
  --ipv6-base=fd00:1234:5678 \
  --search-domain=fabric.example.com

That's fine for a first try by hand. For anything real, use the systemd unit below so the fabriclet survives reboots.

Watch out for:

  • --name must match the Node name that cells reference.
  • The kubeconfig needs credentials (token or client cert) with rights on nodes, interfaces and cells. A switch is not a pod, so in-cluster config never applies; KUBECONFIG (or ~/.kube/config) it is.
  • No -I interfaces needed: the script configures the ports itself, and registration works with just the Node.
  • The script reboots the switch when it finishes. After the reboot the fabriclet finds /etc/sonic/fabric-cell.done and reports the cell Active, assuming the fabriclet actually comes back. 😊

Running as systemd unit on the device

/etc/systemd/system/fabriclet.service:

[Unit]
Description=Fabric fabriclet
After=docker.service
Requires=docker.service

[Service]
Restart=always
RestartSec=10
ExecStartPre=-/usr/bin/docker rm -f fabriclet
ExecStart=/usr/bin/docker run --rm --name fabriclet \
  --network=host --pid=host --privileged \
  -v /etc/sonic:/etc/sonic \
  -v /etc/sonic/fabriclet.kubeconfig:/etc/sonic/fabriclet.kubeconfig:ro \
  -v /run/dbus/system_bus_socket:/run/dbus/system_bus_socket \
  -e KUBECONFIG=/etc/sonic/fabriclet.kubeconfig \
  ghcr.io/ironcore-dev/fabriclet-sonic:<tag> \
  --name=swi1-ab-42 --role=inband-leaf --region=ab \
  --ipv6-base=fd00:1234:5678 --search-domain=fabric.example.com
ExecStop=/usr/bin/docker stop fabriclet

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now fabriclet

@github-actions github-actions Bot added the enhancement New feature or request label Sep 22, 2026
@hardikdr hardikdr added this to Roadmap Sep 23, 2026
@afritzler

Copy link
Copy Markdown
Member Author

Update on the current state of this PR. The first version rendered one of two bash templates picked by a --role flag, with ports and addresses hardcoded per device class. Now everything is rendered from the Cell. Here is how a full flow looks today.

What changed since the first version:

  • One switch.sh template. A fabriclet now only takes --name and its interface list.
  • The switch role is a Cell label (fabric.ironcore.dev/role: leaf|spine). The cell controller passes labels through without reading them; the sonic runtime owns the key and its values. Missing or unknown role fails the cell instead of rendering the wrong tier.
  • No hardcoded port ranges in the template anymore. Every configured port comes from a cell peer: peers with dhcpRelay are servers (VLAN, interface prefix, relay), peers without are fabric links (unnumbered BGP). Ports no peer names stay dark. This also replaces the old "configure all ports" spine behavior: a spine cell lists its leaf-facing ports as fabric peers and must not have server peers.
  • Interfaces are registered per port via -I and referenced from cells with interfaceRef. Each end of a link appears in its own device's cell; nothing cross-checks the two ends yet (that is the future topology oracle's job).

Example: one spine, one leaf

Each switch runs the same container, differing only in --name:

ExecStart=/usr/bin/docker run --rm --name fabriclet \
  --network=host --pid=host --privileged \
  -v /etc/sonic:/etc/sonic \
  -v /etc/sonic/fabriclet.kubeconfig:/etc/sonic/fabriclet.kubeconfig:ro \
  -e KUBECONFIG=/etc/sonic/fabriclet.kubeconfig \
  ghcr.io/ironcore-dev/fabriclet-sonic:<tag> \
  --name=swi1-ab-42 --interface=Ethernet0,Ethernet4,...,Ethernet124

That registers the Node and its 32 Interfaces. Two Cells describe the whole setup:

apiVersion: fabric.ironcore.dev/v1alpha1
kind: Cell
metadata:
  name: cell-swi1-ab-42
  labels:
    fabric.ironcore.dev/role: leaf
spec:
  nodeRef:
    name: swi1-ab-42
  id: "42"
  hostname: swi1-ab-42.fabric.example.com
  ips:
    - fd00:1234:5678:f2::
  prefixes:
    - fd00:1234:5678:f2::/64      # summary
    - fd00:1234:5678:f2:1::/80    # Ethernet0
    # ... one /80 per server port, in peer order ...
  peers:
    - interfaceRef:
        name: Ethernet0
      dhcpRelay: fd00:1234:5678:3201::1:547
    # ... Ethernet4 .. Ethernet104, same relay ...
    # North: uplink to swi2-ab-7, no dhcpRelay, no prefix
    - interfaceRef:
        name: Ethernet108
---
apiVersion: fabric.ironcore.dev/v1alpha1
kind: Cell
metadata:
  name: cell-swi2-ab-7
  labels:
    fabric.ironcore.dev/role: spine
spec:
  nodeRef:
    name: swi2-ab-7
  id: "7"
  hostname: swi2-ab-7.fabric.example.com
  ips:
    - fd00:1234:5678:7::
  prefixes:
    - fd00:1234:5678:7::/64
  peers:
    - interfaceRef:
        name: Ethernet0           # facing the leaf's Ethernet108

@afritzler

Copy link
Copy Markdown
Member Author

Follow-up on the deletion flow: DeleteCell actually resets the switch now.

How it works:

  • On cell deletion the fabriclet renders templates/reset.sh and runs it, but only if /etc/sonic/fabric-cell.done exists. Without the flag the device was never (or is no longer) provisioned, so DeleteCell reports NotFound and the controller expires the cell and removes its finalizer. This also fixes a bug in the previous version: DeleteCell returned success after running the noop script, so the controller kept polling forever and the finalizer never came off.
  • The reset script takes its input from the device, not from the cell. It reads the running CONFIG_DB (jq) and removes exactly what is there. Devices left half-provisioned by failed cells get cleaned up the same way, which a cell-driven inverse could not describe.

What the reset script undoes, in order: the loopback addresses; the server VLANs in dependency order (DHCP relay, interface IPs, members, then the VLAN); the port state (shutdown on every up front-panel port); config save; the FRR config, replaced by an empty one so BGP stops; the done flag; then a reboot.

Deliberately kept: hostname, management config, link-local, and the physical port parameters (mtu/fec/speed do no harm on a shut-down port).

So the deletion lifecycle end to end: delete the cell, the fabriclet resets and reboots the switch, on the next reconcile the flag is gone and the cell expires with its finalizer removed.

Comment thread sonic/templates/reset.sh
# TODO: reset the configuration of the device to a clean state.
echo "Resetting the device is not implemented yet"
exit 0
# Undoes what the cell script applied: server VLANs, loopback IPs, port

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maxmoehl had experimented with just resetting the initial config DB (json file that is loaded into redis on boot, and the rebooting the switch.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So @hardikdr and I just tested the Apply and Delete flow and both ways seem to be working. Of course we can improve the script based setup in the future. However the ultimate goal is to use the configdb based implementation @Giluerre is currently working on.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are careful they can work. But cleaning out config like this is very brittle and prone to leaving stuff behind. If this is just to bridge the gap for something "proper" I don't care too much, but the "wipe it" approach is even less script logic 😄

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for @Giluerre, I can only echo what Max wrote: cleaning out the config one by one for a reset is brittle and any weird state that you don't handle (but e.g. an operator has messed up) will not be reset. The clean slate with 'wipe it' is much more reliable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DeleteCell function of the cellruntime is async by design: If an operator couldn't clean everything, it should return an error. The reconciler will eventually trigger DeleteCell again until it succeeds. If not, the cell will stick around. The cleanup stability of the bash implementation of course isn't the best, but it serves as a great starting point for future optimizations.

@peanball peanball Sep 30, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem I mean is that if your deleter does not "catch" everything that needs to be deleted and says "it's all clean", but it actually isn't.

You could argue this is something to check later if we run into it, as this is an interim implementation to get something going. At the very least we should keep it in mind and address before we go anywhere near prod with an actual implementation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that's of course the one invariant you have to check thoroughly. If there's a bug we have to fix it.

@hardikdr hardikdr left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @afritzler, overall looks good, have a dropped a short question inline.

Comment thread sonic/script.go Resolved
hardikdr
hardikdr previously approved these changes Oct 16, 2026

@hardikdr hardikdr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@afritzler could you please rebase the PR when you have time. :)

Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
@hardikdr
hardikdr merged commit 91001aa into main Oct 19, 2026
10 checks passed
@hardikdr
hardikdr deleted the enh/fabriclet branch October 19, 2026 09:16
@github-project-automation github-project-automation Bot moved this from Needs Review/Discussion to Done in Fabric Automation Oct 19, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Roadmap Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants