drivers/network: add ixgbe driver by terryzbai · Pull Request #682 · au-ts/sddf

terryzbai · 2026-03-24T05:46:32Z

This driver is for Intel X540/X550 NICs and works with IO-APIC ~~and MSI/MSI-X interrupts.~~ (remove MSI/MSI-X support for now to avoid hacky code)

Before a proper PCI driver (#622) gets implemented, this experimental driver needs the access to PCI configuration space and set things up by itself. Interrupt types need to be manually switched by (un)commenting code snippets in meta.py and ethernet.c.

midnightveil

Brief skim.

KurtWu10

I've only checked the provide/return functions of Rx/Tx. I need to read the intel manual further to understand its concurrency model. I don't see the need of using inline assembly for get_reg*/set_reg* either.

omeh-a · 2026-06-04T23:27:51Z

@terryzbai What's the current state on this PR? Is this just waiting on having a PCIe driver to be merged?

terryzbai · 2026-06-05T00:39:42Z

@terryzbai What's the current state on this PR? Is this just waiting on having a PCIe driver to be merged?

That's one of the reasons. Another reason is that I haven't got time to look at the event logics, which seem different from other ethernet drivers that we have. Btw, it would be great for development if we have a machine with less booting time.

omeh-a

Just a quick pass mostly looking at style / maintainability. I haven't had time to consult the ixgbe manual (and it's been way too long since I last touched it). There's a huge amount of dead code in ethernet.h that should probably be pruned. I also am not convinced that the changes to board.py and the echo server are prudent.

omeh-a · 2026-08-13T03:57:22Z

+    eth_regs->eims = 0xFF;
+}
+
+void print_mac_addr()


extract to header

omeh-a · 2026-08-13T04:19:09Z

 * This defines whether we actually try to benchmark, setup the PMU etc.
 */
-#if defined(CONFIG_ENABLE_BENCHMARKS) && (defined(CONFIG_ARCH_ARM) || defined(CONFIG_ARCH_RISCV))
+#if defined(CONFIG_ENABLE_BENCHMARKS) && (defined(CONFIG_ARCH_ARM) || defined(CONFIG_ARCH_RISCV) || defined(CONFIG_ARCH_X86_64))


isn't this allowing all of our architectures now (except 32 bit x86)? can probably reduce this check

I think what @omeh-a means is that you can just do

#if defined(CONFIG_ENABLE_BENCHMARKS)

since we now support all CPU architectures that the Microkit support.

Feel free to resolve the comment if I misunderstood.

yea, I did that but haven't uploaded it.

omeh-a · 2026-08-13T04:21:25Z

+        timer=0xFED00000,
+        ethernet=0xFE000000,
+    ),
+    Board(


IMO we really shouldn't be adding board.py listings for individual x86 machines that are in our server room. The sDDF is a project that's used by people outside the lab and I think we should avoid adding code that's dead for everybody outside of TS.

This should be fixed once the ACPI/PCI driver is merged. But for now, there seems no better solutions.

I think we should probably add some indirection to indicate what this is ... e.g. make a ts_x86_machines.py which we import from for this instead of board.py or something (and that doesn't necessarily need to be on main)... I just think it doesn't really make sense to upload this since it will be useless to all externals

...long in-person discussion...
Given that uploading configurations of TS internal x86 machines has no harm but gives the external users the examples to run on their own machines, so I will just leave some instructions in the board.py file instead.

I still think this really is not a good idea. Going to look extremely silly when we need to open a pull request to update these magic numbers one day... having internal machines explicitly doesn't give external users the option to run locally, it just gives them an example for how they can try match their local machine. It literally only benefits us. It's also worth observing that we have maintained the principle of not doing this for the entire history of the sDDF, under this very reasoning. I don't see why we should give it up now when the project is mature when it was too hacky for the past

We should instead have a guide for how users can collect their own magic info and we should give them a way to pass those numbers in instead. E.g. using argparse in the metaprogram

dreamliner787-9 · 2026-08-17T06:22:17Z

+    uint32_t lo, hi, unused;
+    __asm__ __volatile__("rdtscp" : "=a"(lo), "=d"(hi), "=c"(unused));
+    __asm__ __volatile__("lfence" ::: "memory");
+    cycle_count = ((uint64_t)hi << 32) | lo;


While this works for just measuring CPU utilisation, it is kind of sus because the TSC frequency is != to the CPU frequency. You should use the PMU counter for this kind of job. Given that the ARM and RISC-V already uses the proper cycle counter.

dreamliner787-9 · 2026-08-17T06:27:16Z

+        paddr_top=0x70000000,
+        serial=0x3F8,
+        timer=0xFED00000,
+        ethernet=0x90200000,


Please add detailed instruction on how these addresses are derived on from a working Linux boot.

hmm I just realised that the values read on Linux could be wrong, probably because Linux re-configures the BARs at its boot time. I got the BAR addresses via BAR registers on ECAM.

Does it? or does it rely on the platform firmware configured value? You could look at the PCI code in Linux and see what it does. Or write a small Microkit based program that walks the PCI bus and read out the value via the 0xCF8 and 0xCFC ports.

dreamliner787-9 · 2026-08-17T06:28:23Z

 #define UTILIZATION_PORT 1237

-#define TCP_ECHO_MAX_CONNS 4
+#define TCP_ECHO_MAX_CONNS 10


I'd prefer that changes like this are isolated in its own commit, so that you can justify why the change is necessary in the commit message. Rather than reviewers having to ask you later

dreamliner787-9 · 2026-08-17T12:01:10Z

From your PR body:

This driver is for Intel X540/X550 NICs and works with IO-APIC and MSI/MSI-X interrupts.

Yet I don't see the driver using MSI at all, does this need to be edited then? On the same note, is it hard to support MSI, that would be more performant and less brittle compared to INTx.

dreamliner787-9 · 2026-08-17T12:03:56Z

+    if (ch == timer_config.driver_id) {
+        if (device.init_stage == 0) {
+            init_1();
+        } else if (device.init_stage == 1) {
+            init_2();
+        } else if (device.init_stage == 2) {
+            init_3();
+        }


Might be a horrible idea, but now that #707 is merged, can we just spin wait / seL4_Yield() for the required amount of time instead of needing to use a timer driver?

Same applies for the imx mmc driver

We shouldn't spin wait if we can help it... A timeout is not that bad.

yea, and ixgbe driver needs 10s for stabilising the hardware.

10 seconds seems a long time. Really?

It would be best if you include a X550 manual reference for the timeout.

well, this isn't documented in the manual. The initial version of this driver is ported from atmosphereOS.
https://github.com/mars-research/atmosphere/blob/bd485f22f1d5e4d1623e133700dc233086059603/ixgbe_driver/src/device.rs#L258-L262

dreamliner787-9 · 2026-08-17T12:05:43Z

+        paddr_top=0x70000000,
+        serial=0x3F8,
+        timer=0xFED00000,
+        ethernet=0xFE000000,


How is this value derived? You need to document it, feel free to copy what I did in libvmm: https://github.com/au-ts/libvmm/tree/main/examples/virtio_pci#x86-64-hardware-requirements

For QEMU, I think it would be better to move your instructions to sDDF repo and link it in LibVMM example.

dreamliner787-9 · 2026-08-17T12:23:39Z

Slightly off-topic, I wonder if making this change to clang.mk will improve the echo server performance with this driver:

sddf/tools/make/toolchain/clang.mk

Line 30 in da05d66

CFLAGS_ARCH := -mtune=$(CPU)

-    CFLAGS_ARCH := -mtune=$(CPU)
+    CFLAGS_ARCH := -mtune=$(CPU) -march=x86-64-v2

With this change the compiler will emit more modern and faster instructions without requiring AVX vector instructions, which the Microkit doesn't enable on x86 currently.

dreamliner787-9 · 2026-08-18T00:42:23Z

+        # Legacy I/O APIC
+        eth_irq = SystemDescription.IrqIoapic(ioapic_id=0, pin=16, vector=8, trigger=IrqIoapic.Trigger.LEVEL,
+                                              polarity=IrqIoapic.Polarity.ACTIVELOW, id=16)


This is incorrect for sdfgen 0.33.0 no? You fixed the corresponding bug in sdfgen but have not triggered a release: au-ts/microkit_sdf_gen@608ef02

You will need to make a 0.34.0 release and pin it in your PR.

@Courtney3141 Can you please trigger a release for this?

dreamliner787-9 · 2026-08-18T00:53:39Z

Would this driver work on Microkit 2.3.0? I don't think so because sdfgen doesn't support generation of the IOMMU SDF elements. You will need to document how users can get it working on 2.3.0 (i.e. disable the IOMMU and rebuild the SDK from source), or support the 2.2.0 SDK.

dreamliner787-9 · 2026-08-18T00:59:40Z

+        ethernet_driver.add_map(Map(hw_tx_ring_buffer, vaddr=0x2404000, perms="rw"))
+
+        # Legacy I/O APIC
+        eth_irq = SystemDescription.IrqIoapic(ioapic_id=0, pin=16, vector=8, trigger=IrqIoapic.Trigger.LEVEL,


vector=8 is low priority. I reckon the ethernet IRQ should be a higher priority, make it one less than the HPET vector

It seems not matter too much in this example, as there are only HPET and ethernet driver having user IRQs. Note: virtio-net driver also has vector=1.

Does the priority order of IRQ always match the priority order of the corresponding driver PDs? If so, this might be worth being handled by sdf_gen/Acacia.

midnightveil · 2026-08-18T01:11:17Z

Would this driver work on Microkit 2.3.0? I don't think so because sdfgen doesn't support generation of the IOMMU SDF elements. You will need to document how users can get it working on 2.3.0 (i.e. disable the IOMMU and rebuild the SDK from source), or support the 2.2.0 SDK.

Note it's possible to pass disable_iommu to the kernel command line with recent Microkits, as a hotfix for this.

terryzbai · 2026-08-18T01:59:08Z

This driver is for Intel X540/X550 NICs and works with IO-APIC and MSI/MSI-X interrupts.

Yet I don't see the driver using MSI at all, does this need to be edited then? On the same note, is it hard to support MSI, that would be more performant and less brittle compared to INTx.

yes, I decided to remove MSI/MSI-x support for this PR because users would need to manually uncomment and comment code snippet, which is very ugly. Have updated the PR description.

cazb2 · 2026-08-18T02:29:08Z

Would this driver work on Microkit 2.3.0? I don't think so because sdfgen doesn't support generation of the IOMMU SDF elements. You will need to document how users can get it working on 2.3.0 (i.e. disable the IOMMU and rebuild the SDK from source), or support the 2.2.0 SDK.

Note it's possible to pass disable_iommu to the kernel command line with recent Microkits, as a hotfix for this.

Another method which is super easy is to change the IOMMU boolean from True to False in build_sdk.py

dreamliner787-9 · 2026-08-18T04:43:35Z

Oops sorry

To fix: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

To f i x this issue: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

This adds the initial ixgbe driver, which has been tested on X550/X540 NICs. The device reigsters and IRQ numbers are hard-coded before the PCIe driver is properly implemented. Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

This adds the physical TSC reading for CPU utilisation measurement. Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

dreamliner787-9 · 2026-08-18T23:56:33Z

+    SEL4_64B = $(MICROKIT_SDK)/board/$(MICROKIT_BOARD)/$(MICROKIT_CONFIG)/elf/sel4.elf
+    SEL4_32B := $(MICROKIT_SDK)/board/$(MICROKIT_BOARD)/$(MICROKIT_CONFIG)/elf/sel4_32.elf


Could be refactored, same with vtx.mk

dreamliner787-9 · 2026-08-19T00:09:09Z

+    if test_config.board.startswith("vb_105"):
+        # This x86 machine takes around 3 mintues to boot
+        timeout = 200
+


The boot timeout is separate from the test timeout, this shouldn't be here, see https://github.com/au-ts/systems-ci/blob/f1127dfb4c47d0c5bc3512d3678b3434c3835c34/ts_ci/backends/machine_queue.py#L19

So this block should be removed

To f i x this issue: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

midnightveil requested changes Mar 24, 2026

View reviewed changes

KurtWu10 reviewed Mar 31, 2026

View reviewed changes

dreamliner787-9 mentioned this pull request Apr 13, 2026

ixgbe: stable UDP but volatile TCP performance #702

Open

midnightveil added the network label Jun 4, 2026

terryzbai force-pushed the ixgbe_driver_cleanup branch from 6b14384 to 3e27cd9 Compare July 30, 2026 23:17

terryzbai requested a review from Courtney3141 July 30, 2026 23:39

terryzbai mentioned this pull request Aug 5, 2026

Echo Server: Simplify benchmarking setup #757

Draft

cazb2 reviewed Aug 11, 2026

View reviewed changes

Comment thread drivers/network/ixgbe/ethernet.c Outdated

omeh-a requested changes Aug 13, 2026

View reviewed changes

dreamliner787-9 requested changes Aug 17, 2026

View reviewed changes

dreamliner787-9 reviewed Aug 17, 2026

View reviewed changes

dreamliner787-9 requested changes Aug 18, 2026

View reviewed changes

Comment thread tools/make/board/x86_64_generic.mk

dreamliner787-9 requested changes Aug 18, 2026

View reviewed changes

Comment thread examples/echo_server/meta.py Outdated

dreamliner787-9 requested changes Aug 18, 2026

View reviewed changes

dreamliner787-9 mentioned this pull request Aug 18, 2026

Update sdfgen to 0.35.0 #770

Merged

dreamliner787-9 closed this in #770 Aug 18, 2026

dreamliner787-9 mentioned this pull request Aug 18, 2026

Update sdfgen to 0.35.0 au-ts/libvmm#395

Merged

midnightveil reopened this Aug 18, 2026

dreamliner787-9 added a commit to au-ts/libvmm that referenced this pull request Aug 18, 2026

Update sdfgen to 0.35.0

604f375

To fix: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

dreamliner787-9 closed this in au-ts/libvmm#395 Aug 18, 2026

midnightveil reopened this Aug 18, 2026

dreamliner787-9 mentioned this pull request Aug 18, 2026

Updated sdfgen to 0.35.0 au-ts/lionsos#340

Merged

dreamliner787-9 added a commit to au-ts/lionsos that referenced this pull request Aug 18, 2026

Updated sdfgen to 0.35.0

12dff43

To f i x this issue: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

terryzbai force-pushed the ixgbe_driver_cleanup branch from 889d272 to de3c4e0 Compare August 18, 2026 23:50

terryzbai added 4 commits August 19, 2026 09:51

drivers/network: add ixgbe driver

0d0c4a9

This adds the initial ixgbe driver, which has been tested on X550/X540 NICs. The device reigsters and IRQ numbers are hard-coded before the PCIe driver is properly implemented. Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

benchmark: add cycle counter reading for x86

9df960f

This adds the physical TSC reading for CPU utilisation measurement. Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

wip: fixes

5206b69

Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

wip: remove unreliable instructions

5bea2f7

Signed-off-by: Terry Bai <tianyi.bai@unsw.edu.au>

dreamliner787-9 force-pushed the ixgbe_driver_cleanup branch from de3c4e0 to 5bea2f7 Compare August 18, 2026 23:51

dreamliner787-9 reviewed Aug 18, 2026

View reviewed changes

dreamliner787-9 requested changes Aug 19, 2026

View reviewed changes

Courtney3141 pushed a commit to au-ts/lionsos that referenced this pull request Aug 19, 2026

Updated sdfgen to 0.35.0 (#340)

53c943e

To f i x this issue: au-ts/sddf#682 (comment) Signed-off-by: Bill Nguyen <bill.nguyen@unsw.edu.au>

		SEL4_64B = $(MICROKIT_SDK)/board/$(MICROKIT_BOARD)/$(MICROKIT_CONFIG)/elf/sel4.elf
		SEL4_32B := $(MICROKIT_SDK)/board/$(MICROKIT_BOARD)/$(MICROKIT_CONFIG)/elf/sel4_32.elf

Uh oh!

Conversation

terryzbai commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

midnightveil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KurtWu10 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

omeh-a commented Jun 4, 2026

Uh oh!

terryzbai commented Jun 5, 2026

Uh oh!

Uh oh!

omeh-a left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omeh-a Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terryzbai commented Mar 24, 2026 •

edited

Loading

KurtWu10 left a comment •

edited

Loading

omeh-a Aug 18, 2026 •

edited

Loading

dreamliner787-9 Aug 17, 2026 •

edited

Loading

dreamliner787-9 Aug 18, 2026 •

edited

Loading

dreamliner787-9 commented Aug 18, 2026 •

edited

Loading