use dmabuf option when the ibperf binaries have them configured#235
use dmabuf option when the ibperf binaries have them configured#235ahskabir wants to merge 4 commits into
Conversation
|
|
||
| try: | ||
| # detailed=True gives us exit_code per host | ||
| host_results = phdl.exec(cmd, detailed=True, print_console=False) |
There was a problem hiding this comment.
It assumes a homogeneous cluster — all nodes have the same binary at the same app_path. That's a safe assumption in the ibperf context (the binary path comes from a shared config and is the same for all nodes), and it's consistent with how the rest of the test already treats the cluster.
So we can just use shdl.exec instead of phdl.exec, shdl.exec will test the binary only in the head node(which is sufficent)
|
|
||
| dmabuf_supported_any = False | ||
|
|
||
| for host, result in host_results.items(): |
There was a problem hiding this comment.
when shdl is used this whole for loop to set "dmabuf_supported_any " becomes unnecessary.
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
Signed-off-by: Ahsan Kabir <Ahsan.Kabir@amd.com>
fa628ff to
6cef5d0
Compare
| return '/opt/rocm' | ||
|
|
||
|
|
||
| def check_perftest_dmabuf_support(shdl, binary_path, head_node): |
There was a problem hiding this comment.
We no need to pass head_node to this function, because shdl isa Pssh instance with only one node in the list(head_node) refer https://github.com/ROCm/cvs/pull/235/changes#diff-fc999b79285f07a1d04220513c8cc33f07b30514861fdf7baea8058c53a85bd8R170
| return dmabuf_available | ||
| try: | ||
| dmabuf_check_out = shdl.exec(cmd) | ||
| dmabuf_available = int(dmabuf_check_out[head_node].strip()) > 0 |
There was a problem hiding this comment.
dmabuf_check_out = shdl.exec(cmd)
for node in dmabuf_check_out .keys():
dmabuf_available = int(dmabuf_check_out [node].strip()) > 0
We can use above logic, without head_node argument, the for loop will iterate once as the dmabuf_check_out will have ouptput from one node(head_node)
| port_no=1516, | ||
| duration=60, | ||
| rocm_path='', | ||
| cluster_node_list=None, |
There was a problem hiding this comment.
We no need to pass cluster_node_list , because shdl is connection handle to head node only
| gid_index, | ||
| port_no=1516, | ||
| rocm_path='', | ||
| cluster_node_list=None, |
There was a problem hiding this comment.
We no need to pass cluster_node_list , because shdl is connection handle to head node only
| card_no = 'card' + str(gpu_no) | ||
| rdma_dev = gpu_nic_dict[node][card_no]['rdma_dev'] | ||
| cmd = f'numactl --physcpubind={gpu_numa_dict[node][card_no]["local_cpulist"]} --localalloc {app_path}/{bw_test} -d {rdma_dev} --use_rocm={gpu_no} -x {gid_index} --report_gbits -b -F -D {duration} -p {port_no} -s {msg_size} -q {qp_count} > /tmp/ib_perf_{inst_count}_logs & 2>&1' | ||
| if dmabuf_supported: |
There was a problem hiding this comment.
We can simplify like below to avoid code duplication in if else
`
cmd_parts = [
"numactl",
f'--physcpubind={gpu_numa_dict[node][card_no]["local_cpulist"]}',
"--localalloc",
f"{app_path}/{bw_test}",
"-d", rdma_dev,
f"--use_rocm={gpu_no}",
*(["--use_rocm_dmabuf"] if dmabuf_supported else []),
"-x", str(gid_index),
"--report_gbits", "-b", "-F",
"-D", str(duration),
"-p", str(port_no),
"-s", str(msg_size),
"-q", str(qp_count),
server_addr,
]
cmd = " ".join(cmd_parts) + f" > /tmp/ib_perf_{inst_count}_logs 2>&1 &"
`
2 participants
Motivation
IBPERF test makes use of the binaries for which path is provided in the configuration file. It also has an option to install the
ibperftool and then to run the test. But during the run-time when it prepares theibperfinvocation command(s) it doesn't use the--use_rocm_dmabufoption and the run could be less performant.Technical Details
A function is added in the ibperf library. This function check to see whether the ibperf binary was configured with --use_rocm_dmabuf. If yes, it will use the flag for invocation, otherwise not.
Test Plan
All ibperf test was run using ibperf binaries configured with --use_rocm_dmabuf and test passed.
All ibperf test was run using ibperf binaries not configured with --use_rocm_dmabuf and test passed.
Test Result
In both the scenarios, all tests passed.
Submission Checklist