Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
deploy_run_in_progress_config_map,
deploy_run_in_progress_namespace,
filter_hpp_tests,
filter_multiarch_tests,
get_artifactory_server_url,
get_base_matrix_name,
get_cnv_version_explorer_url,
Expand Down Expand Up @@ -643,6 +644,7 @@ def pytest_collection_modifyitems(session, config, items):
config.hook.pytest_deselected(items=discard)
items[:] = filter_deprecated_api_tests(items=items, config=config)
items[:] = filter_sno_only_tests(items=items, config=config)
items[:] = filter_multiarch_tests(items=items, config=config)
items[:] = filter_hpp_tests(items=items, config=config)
items[:] = mark_nmstate_dependent_tests(items=items)

Expand Down
90 changes: 85 additions & 5 deletions docs/MULTIARCH.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,93 @@
# Multi-Architecture (Heterogeneous) Clusters

Currently supported architectures for multi-arch runs: `amd64` and `arm64`.
A heterogeneous cluster has worker nodes of more than one CPU architecture — for example, amd64 and arm64 workers coexisting on the same cluster. The test framework detects this automatically.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

On clusters where nodes have different CPU architectures, you must pass `--cpu-arch` to select the architecture for the run. Use a single value (e.g. `--cpu-arch=amd64`) or, for tests marked with `multiarch`, a comma-separated list (e.g. `--cpu-arch=amd64,arm64`). Use the config file `tests/global_config_multiarch.py` and the `multiarch` marker for tests that run across multiple architectures. Do not pass `--cpu-arch` on homogeneous clusters.
**Supported architectures:** `amd64`, `arm64`

## Overview

The `--cpu-arch` option selects which architecture(s) to target. Behavior depends on cluster type and the value passed:

| Test suite | Cluster | `--cpu-arch` | What runs |
| ---------- | ------- | ------------ | --------- |
| **Standard** | Homogeneous (single arch) | Omit — auto-detected from node labels | Standard regression tests |
| **Multiarch regression** | Heterogeneous (multiarch) | Single value — e.g. `amd64` or `arm64` | Full suite, scoped to nodes of that arch |
| **Multiarch-dedicated** | Heterogeneous (multiarch) | Comma-separated — e.g. `amd64,arm64` | Only tests marked with `multiarch` |

**On a heterogeneous cluster:**

- `--cpu-arch` is **required**. Omitting it raises `UnsupportedCPUArchitectureError`.
- Pass `--tc-file=tests/global_config.py` for regression runs (preferred — it auto-imports the multiarch config). `tests/global_config_multiarch.py` also works.
- For multiarch-dedicated tests, `--tc-file=tests/global_config_multiarch.py` is **required**.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

**On a homogeneous cluster:**

- Run tests normally and do **not** pass `--cpu-arch`. Passing it raises `UnsupportedCPUArchitectureError`.

## Multiarch regression

Run the existing test suite against one architecture's nodes at a time.

```bash
# amd64 regression
uv run pytest --tc-file=tests/global_config.py --cpu-arch=amd64 ...

# arm64 regression
uv run pytest --tc-file=tests/global_config.py --cpu-arch=arm64 ...
```

## Multiarch-dedicated tests

Run tests that exercise behavior requiring multiple architectures simultaneously (e.g., golden image import across both archs, cross-arch scheduling).

```bash
uv run pytest --tc-file=tests/global_config_multiarch.py --cpu-arch=amd64 ...
uv run pytest --tc-file=tests/global_config_multiarch.py --cpu-arch=amd64,arm64 \
-m "iuo and multiarch" ...
```

## Limitations
### Run requirements

- The cluster must be heterogeneous.
- `--cpu-arch` must list multiple architectures (e.g. `amd64,arm64`)
- Every collected test must have the `multiarch` marker — use `-m multiarch`, or point pytest at a path that contains only multiarch tests (e.g. a `multiarch/` subdirectory). If any non-multiarch test is collected, pytest raises `UnsupportedCPUArchitectureError`.

### Writing dedicated tests
Comment thread
hmeir marked this conversation as resolved.

Multiarch-dedicated tests should be isolated from regular tests. Avoid modifying existing fixtures and functions — prefer creating dedicated ones for multiarch tests to reduce the risk of breaking regression suites.

Mark the entire file or specific classes with the `multiarch` marker:

```python
# Module-level (preferred — marks the whole file)
pytestmark = [pytest.mark.multiarch]

# Class-level
@pytest.mark.multiarch
class TestMultiarchFeature:
...
```

The `multiarch` marker is **required** on any test that runs in multiarch-dedicated mode. It also prevents the test from being collected on homogeneous clusters.

Place multiarch tests in a `multiarch/` subdirectory or use `_multiarch` in the filename (repo convention, not enforced by pytest):

```
Comment thread
coderabbitai[bot] marked this conversation as resolved.
tests/
install_upgrade_operators/
hco_enablement_golden_image_updates/
multiarch/
test_multiarch_golden_images_support.py
network/
connectivity/
test_pod_network_multiarch.py
```

### Framework constraints

Multiarch-dedicated runs (`--cpu-arch=amd64,arm64`) do not set a single target architecture. Tests must not rely on the helpers that regression runs provide:

`*_os_matrix` variables are not created for multi-arch runs (when `--cpu-arch` contains multiple architectures, e.g. `--cpu-arch=amd64,arm64`).
| Helper | Multiarch regression | Multiarch-dedicated |
| ------ | -------------------- | ------------------- |
| `py_config["cpu_arch"]` | Set to selected arch | **Not set** |
| OS matrix keys in `py_config` (e.g. `latest_rhel_os_dict`) | Generated for selected arch | **Not generated** |
| `schedulable_nodes` | Nodes of selected arch only | All schedulable nodes — filter by arch in the test |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
4 changes: 2 additions & 2 deletions tests/chaos/oadp/test_oadp.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
pytest.param(
{
"vm_name": "vm-node-reboot-12011",
"rhel_image": RHEL_LATEST["image_name"],
"rhel_image": RHEL_LATEST.get("image_name"),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
},
marks=pytest.mark.polarion("CNV-12011"),
),
Expand Down Expand Up @@ -49,7 +49,7 @@ def test_reboot_vm_node_during_backup(
pytest.param(
{
"vm_name": "vm-node-drain-12020",
"rhel_image": RHEL_LATEST["image_name"],
"rhel_image": RHEL_LATEST.get("image_name"),
},
marks=pytest.mark.polarion("CNV-12020"),
),
Expand Down
12 changes: 6 additions & 6 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -474,13 +474,12 @@ def nodes(admin_client):


@pytest.fixture(scope="session")
def schedulable_nodes(nodes):
def schedulable_nodes(nodes, nodes_cpu_architecture):
"""Get nodes marked as schedulable by kubevirt.

For multi-arch testing - filter nodes by the architecture being tested.
"""
schedulable_label = "kubevirt.io/schedulable"
cpu_arch = py_config.get("cpu_arch")
schedulable = [
node
for node in nodes
Expand All @@ -489,10 +488,12 @@ def schedulable_nodes(nodes):
and not node.instance.spec.unschedulable
and not kubernetes_taint_exists(node)
and node.kubelet_ready
and (not cpu_arch or node.labels.get(KUBERNETES_ARCH_LABEL) == cpu_arch)
and (not nodes_cpu_architecture or node.labels.get(KUBERNETES_ARCH_LABEL) == nodes_cpu_architecture)
]

LOGGER.info(f"Schedulable nodes: {[node.name for node in schedulable]}, node architecture: {cpu_arch or 'all'}")
LOGGER.info(
f"Schedulable nodes: {[node.name for node in schedulable]}, node architecture: {nodes_cpu_architecture or 'all'}"
)
yield schedulable


Expand Down Expand Up @@ -1496,7 +1497,6 @@ def cluster_info(
ocs_current_version,
kubevirt_resource_scope_session,
workers_type,
nodes_cpu_architecture,
):
title = "\nCluster info:\n"
virtctl_client_version, virtctl_server_version = None, None
Expand All @@ -1513,7 +1513,7 @@ def cluster_info(
f"\tOCS version: {ocs_current_version}\n"
f"\tCNI type: {get_cluster_cni_type(admin_client=admin_client)}\n"
f"\tWorkers type: {workers_type}\n"
f"\tCluster CPU Architecture: {py_config['cluster_arch']}\n"
f"\tCluster CPU Architecture: {', '.join(py_config['cluster_arch'])}\n"
f"\tIPv4 cluster: {ipv4_supported_cluster()}\n"
f"\tIPv6 cluster: {ipv6_supported_cluster()}\n"
f"\tVirtctl version: \n\t{virtctl_client_version}\n\t{virtctl_server_version}\n"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

import pytest

__test__ = False
pytestmark = [pytest.mark.multiarch]


class TestDisabledMultiarchGoldenImagesSupport:
Expand All @@ -29,6 +29,8 @@ class TestDisabledMultiarchGoldenImagesSupport:
- "enableMultiArchBootImageImport" feature gate disabled in HCO CR
"""

__test__ = False

@pytest.mark.polarion("CNV-15977")
def test_only_architecture_agnostic_golden_image_resources_exist(self):
"""
Expand Down
5 changes: 3 additions & 2 deletions tests/storage/cdi_upload/test_upload.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import tests.storage.utils as storage_utils
import utilities.storage
from tests.os_params import RHEL_LATEST
from utilities.constants import Images
from utilities.constants.components import CDI_UPLOADPROXY
from utilities.constants.timeouts import TIMEOUT_1MIN, TIMEOUT_3MIN, TIMEOUT_5MIN
Expand Down Expand Up @@ -322,8 +323,8 @@ def test_successful_concurrent_uploads(
[
pytest.param(
{
"image_path": py_config["latest_rhel_os_dict"]["image_path"],
"image_file": py_config["latest_rhel_os_dict"]["image_name"],
"image_path": RHEL_LATEST.get("image_path"),
"image_file": RHEL_LATEST.get("image_name"),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
},
{
"dv_name": "cnv-4511",
Expand Down
5 changes: 3 additions & 2 deletions tests/storage/cdi_upload/test_upload_virtctl.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from pytest_testconfig import config as py_config

from libs.net.cluster import is_ipv6_single_stack_cluster
from tests.os_params import RHEL_LATEST
from tests.storage.cdi_upload.utils import get_storage_profile_minimum_supported_pvc_size
from tests.storage.utils import assert_use_populator, create_windows_vm_validate_guest_agent_info
from utilities.constants import Images
Expand Down Expand Up @@ -495,8 +496,8 @@ def test_successful_vm_from_uploaded_dv_windows(
[
pytest.param(
{
"image_path": py_config["latest_rhel_os_dict"]["image_path"],
"image_file": py_config["latest_rhel_os_dict"]["image_name"],
"image_path": RHEL_LATEST.get("image_path"),
"image_file": RHEL_LATEST.get("image_name"),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
},
marks=(pytest.mark.polarion("CNV-4512")),
),
Expand Down
4 changes: 2 additions & 2 deletions tests/storage/golden_image/test_golden_image.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@


LOGGER = logging.getLogger(__name__)
LATEST_RHEL_IMAGE = RHEL_LATEST["image_path"]
RHEL_IMAGE_SIZE = RHEL_LATEST["dv_size"]
LATEST_RHEL_IMAGE = RHEL_LATEST.get("image_path")
RHEL_IMAGE_SIZE = RHEL_LATEST.get("dv_size")
Comment thread
coderabbitai[bot] marked this conversation as resolved.


DV_PARAM = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ def increased_high_performance_vm_core_count_by_one(high_performance_vm):
"os": RHEL_LATEST_OS,
"workload": Template.Workload.HIGHPERFORMANCE,
"flavor": Template.Flavor.SMALL,
"architecture": py_config["cpu_arch"],
"architecture": py_config.get("cpu_arch"),
},
},
],
Expand Down Expand Up @@ -151,7 +151,7 @@ def test_rhel_change_cpu_core_count(
"os": "win2k19",
"workload": Template.Workload.HIGHPERFORMANCE,
"flavor": Template.Flavor.MEDIUM,
"architecture": py_config["cpu_arch"],
"architecture": py_config.get("cpu_arch"),
},
},
],
Expand Down
17 changes: 16 additions & 1 deletion tox.ini
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tox]
envlist=pytest-check-x86, pytest-check-arm64, pytest-check-s390x, unused-code, utilities-unittests
envlist=pytest-check-x86, pytest-check-arm64, pytest-check-s390x, pytest-check-multiarch, unused-code, utilities-unittests
skipsdist=True

[testenv]
Expand Down Expand Up @@ -59,6 +59,21 @@ deps=
commands =
uv run pytest --tc-file=tests/global_config.py --collect-only -m s390x

[testenv:pytest-check-multiarch]
basepython = python3.14
recreate=True
setenv =
PYTHONPATH = {toxinidir}
LC_ALL = en_US.utf8
LANG = en_US.utf8
UV_PYTHON = python3.14
OPENSHIFT_VIRTUALIZATION_TEST_IMAGES_ARCH = amd64,arm64
deps=
uv
commands =
uv run pytest --tc-file=tests/global_config.py --cpu-arch=amd64 --collect-only
uv run pytest --tc-file=tests/global_config.py --cpu-arch=arm64 --collect-only
uv run pytest --tc-file=tests/global_config_multiarch.py --cpu-arch=amd64,arm64 -m multiarch --collect-only

# Polarion
# Should run on every commit.
Expand Down
2 changes: 1 addition & 1 deletion utilities/architecture.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def get_cluster_architecture() -> set[str]:

# Needed for CI
if arch := os.environ.get("OPENSHIFT_VIRTUALIZATION_TEST_IMAGES_ARCH"):
return {arch}
return set(arch.split(","))

# Skip cluster connection for pytest flags that exit immediately without collecting tests
_pytest_exit_flags = {"--help", "-h", "--version"}
Expand Down
22 changes: 22 additions & 0 deletions utilities/pytest_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -649,6 +649,28 @@ def update_cpu_arch_related_config(cpu_arch_option: str) -> None:
generate_instance_type_matrix_dicts(os_dict=py_config)


def filter_multiarch_tests(items: list[pytest.Item], config: pytest.Config) -> list[pytest.Item]:
"""Deselect multiarch-marked tests on homogeneous clusters.

On heterogeneous clusters (cluster_type=MULTIARCH), all tests pass through unchanged.
On homogeneous clusters, tests marked with 'multiarch' are deselected and reported
via pytest_deselected so they appear in the session summary.

Args:
items: Collected test items.
config: Pytest config object, used to report deselected items.

Returns:
Filtered list of test items with multiarch tests removed on homogeneous clusters.
"""
if py_config.get("cluster_type") == MULTIARCH:
return items
discard_tests, items_to_return = remove_tests_from_list(items=items, filter_str="multiarch")
if discard_tests:
config.hook.pytest_deselected(items=discard_tests)
return items_to_return

Comment thread
coderabbitai[bot] marked this conversation as resolved.

Comment thread
coderabbitai[bot] marked this conversation as resolved.
def assert_incremental_classes_fully_collected(items: list[pytest.Item]) -> None:
"""Verify that all tests defined in incremental classes were collected.

Expand Down
6 changes: 6 additions & 0 deletions utilities/unittests/test_architecture.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ def test_get_cluster_architecture_from_env_amd64(self):
result = get_cluster_architecture()
assert result == {"amd64"}

def test_get_cluster_architecture_from_env_multiarch(self):
"""Test getting architecture from environment variable - comma-separated multiarch"""
with patch.dict(in_dict=os.environ, values={"OPENSHIFT_VIRTUALIZATION_TEST_IMAGES_ARCH": "amd64,arm64"}):
result = get_cluster_architecture()
assert result == {"amd64", "arm64"}

@patch("utilities.architecture.cache_admin_client")
@patch("utilities.architecture.Node")
def test_get_cluster_architecture_from_nodes_amd64(self, mock_node_class, mock_cache_client):
Expand Down
Loading
Loading