Repository navigation
Add script-based SONiC cell runtime - #16
Conversation
5ae617a to
6acb25f
Compare
|
Update on the current state of this PR. The first version rendered one of two bash templates picked by a What changed since the first version:
Example: one spine, one leafEach switch runs the same container, differing only in ExecStart=/usr/bin/docker run --rm --name fabriclet \
--network=host --pid=host --privileged \
-v /etc/sonic:/etc/sonic \
-v /etc/sonic/fabriclet.kubeconfig:/etc/sonic/fabriclet.kubeconfig:ro \
-e KUBECONFIG=/etc/sonic/fabriclet.kubeconfig \
ghcr.io/ironcore-dev/fabriclet-sonic:<tag> \
--name=swi1-ab-42 --interface=Ethernet0,Ethernet4,...,Ethernet124That registers the Node and its 32 Interfaces. Two Cells describe the whole setup: apiVersion: fabric.ironcore.dev/v1alpha1
kind: Cell
metadata:
name: cell-swi1-ab-42
labels:
fabric.ironcore.dev/role: leaf
spec:
nodeRef:
name: swi1-ab-42
id: "42"
hostname: swi1-ab-42.fabric.example.com
ips:
- fd00:1234:5678:f2::
prefixes:
- fd00:1234:5678:f2::/64 # summary
- fd00:1234:5678:f2:1::/80 # Ethernet0
# ... one /80 per server port, in peer order ...
peers:
- interfaceRef:
name: Ethernet0
dhcpRelay: fd00:1234:5678:3201::1:547
# ... Ethernet4 .. Ethernet104, same relay ...
# North: uplink to swi2-ab-7, no dhcpRelay, no prefix
- interfaceRef:
name: Ethernet108
---
apiVersion: fabric.ironcore.dev/v1alpha1
kind: Cell
metadata:
name: cell-swi2-ab-7
labels:
fabric.ironcore.dev/role: spine
spec:
nodeRef:
name: swi2-ab-7
id: "7"
hostname: swi2-ab-7.fabric.example.com
ips:
- fd00:1234:5678:7::
prefixes:
- fd00:1234:5678:7::/64
peers:
- interfaceRef:
name: Ethernet0 # facing the leaf's Ethernet108 |
|
Follow-up on the deletion flow: How it works:
What the reset script undoes, in order: the loopback addresses; the server VLANs in dependency order (DHCP relay, interface IPs, members, then the VLAN); the port state (shutdown on every up front-panel port); Deliberately kept: hostname, management config, link-local, and the physical port parameters (mtu/fec/speed do no harm on a shut-down port). So the deletion lifecycle end to end: delete the cell, the fabriclet resets and reboots the switch, on the next reconcile the flag is gone and the cell expires with its finalizer removed. |
1865ccb to
02333db
Compare
02333db to
f84e493
Compare
| # TODO: reset the configuration of the device to a clean state. | ||
| echo "Resetting the device is not implemented yet" | ||
| exit 0 | ||
| # Undoes what the cell script applied: server VLANs, loopback IPs, port |
There was a problem hiding this comment.
@maxmoehl had experimented with just resetting the initial config DB (json file that is loaded into redis on boot, and the rebooting the switch.
There was a problem hiding this comment.
If you are careful they can work. But cleaning out config like this is very brittle and prone to leaving stuff behind. If this is just to bridge the gap for something "proper" I don't care too much, but the "wipe it" approach is even less script logic 😄
There was a problem hiding this comment.
So for @Giluerre, I can only echo what Max wrote: cleaning out the config one by one for a reset is brittle and any weird state that you don't handle (but e.g. an operator has messed up) will not be reset. The clean slate with 'wipe it' is much more reliable.
There was a problem hiding this comment.
The DeleteCell function of the cellruntime is async by design: If an operator couldn't clean everything, it should return an error. The reconciler will eventually trigger DeleteCell again until it succeeds. If not, the cell will stick around. The cleanup stability of the bash implementation of course isn't the best, but it serves as a great starting point for future optimizations.
There was a problem hiding this comment.
The problem I mean is that if your deleter does not "catch" everything that needs to be deleted and says "it's all clean", but it actually isn't.
You could argue this is something to check later if we run into it, as this is an interim implementation to get something going. At the very least we should keep it in mind and address before we go anywhere near prod with an actual implementation.
There was a problem hiding this comment.
I mean that's of course the one invariant you have to check thoroughly. If there's a bug we have to fix it.
f84e493 to
c65c4b0
Compare
There was a problem hiding this comment.
Thanks for the PR @afritzler, overall looks good, have a dropped a short question inline.
hardikdr
left a comment
There was a problem hiding this comment.
@afritzler could you please rebase the PR when you have time. :)
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
Signed-off-by: Andreas Fritzler <andreas.fritzler@sap.com>
c65c4b0 to
9fd0b26
Compare
Proposed Changes
This adds a first working
cellruntime.Runtimefor SONiC switches (sonic/). A fabriclet runs on the device itself and provisions it once aCellgets bound to itsNode. For now the provisioning is done by the ZTP bash scripts we already have; a follow-up will talk to SONiC DB directly.How it works:
ApplyCelltakes the template for the switch role (sonic/templates/{inband-leaf,inband-spine}.sh),fills in the cell ID, region, IPv6 base and search domain, writes the result to a fresh temp file and runs it with bash. Exit 0 means configured, anything else is a terminal error and the
Cellgoes toFailed. I made failures terminal on purpose: a script that died halfway leaves partial config, and plain retries would just fail the same way again. Non-numeric cell IDs are rejected before anything runs, because bash arithmetic treats them as 0 and the script would exit 0 after writing garbage.CellStatusjust checks for/etc/sonic/fabric-cell.done, the flag the script touches at the end. Flag present →Active. Missing → the cell gets applied.DeleteCellrunstemplates/reset.sh, a noop for now. That is where the device reset (ONIE re-install) goes later.Running on a device
The fabriclet ships as a container image (
ghcr.io/ironcore-dev/fabriclet-sonic:<tag>). On the switch:That's fine for a first try by hand. For anything real, use the systemd unit below so the fabriclet survives reboots.
Watch out for:
--namemust match theNodename that cells reference.nodes,interfacesandcells. A switch is not a pod, so in-cluster config never applies;KUBECONFIG(or~/.kube/config) it is.-Iinterfaces needed: the script configures the ports itself, and registration works with just theNode./etc/sonic/fabric-cell.doneand reports the cellActive, assuming the fabriclet actually comes back. 😊Running as systemd unit on the device
/etc/systemd/system/fabriclet.service:systemctl daemon-reload systemctl enable --now fabriclet