Skip to content

test: CUDA memory pool fallback in python backend #8992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattwittwer merged 4 commits into from
Oct 7, 2026
Merged

test: CUDA memory pool fallback in python backend #8992

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
be2a7b4
test: Cover Python backend GPU outputs when the CUDA memory pool fall…
mattwittwer Oct 1, 2026
b16f15e
Merge branch 'main' into mwittwer/python_backend_pinned_fallback_test
mattwittwer Oct 5, 2026
25d9fa7
test: Run pinned fallback for both ensemble_io trials and harden serv…
mattwittwer Oct 6, 2026
aee4280
Merge branch 'mwittwer/python_backend_pinned_fallback_test' of github…
mattwittwer Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion qa/L0_backend_python/io/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright 2021-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright 2021-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
Expand Down Expand Up @@ -92,6 +92,53 @@ for trial in $TRIALS; do
done

# IOTest.test_empty_gpu_output
# IOTest.test_ensemble_io with GPU outputs falling back to pinned memory
export TRIAL=default
rm -rf ./models
for i in {1..3}; do
model_name=dlpack_io_identity_$i
mkdir -p models/$model_name/1/
cp ../../python_models/dlpack_io_identity/model.py ./models/$model_name/1/
cp ../../python_models/dlpack_io_identity/config.pbtxt ./models/$model_name/
(cd models/$model_name && \
sed -i "s/^name:.*/name: \"$model_name\"/" config.pbtxt)
done
mkdir -p models/ensemble_io/1/
cp ../../python_models/ensemble_io/config.pbtxt ./models/ensemble_io

# 1000 x FP32 tensors (4000 bytes) against a 1024-byte pool.
CUDA_MEMORY_POOL_SIZE_BYTES=1024
SAVED_SERVER_ARGS="${SERVER_ARGS}"
SERVER_ARGS="${SERVER_ARGS} --cuda-memory-pool-byte-size=0:${CUDA_MEMORY_POOL_SIZE_BYTES}"
SERVER_LOG="./io_server.cuda_pool_fallback.log"
run_server
if [ "$SERVER_PID" == "0" ]; then
echo -e "\n***\n*** Failed to start $SERVER\n***"
cat $SERVER_LOG
RET=1
fi

set +e
SUBTEST="test_ensemble_io"
python3 -m pytest --junitxml=${SUBTEST}.cuda_pool_fallback.report.xml ${UNITTEST_PY}::IOTest::${SUBTEST} > ${CLIENT_LOG}.${SUBTEST}.cuda_pool_fallback
if [ $? -ne 0 ]; then
echo -e "\n***\n*** IOTest.${SUBTEST} (CUDA pool fallback) FAILED. \n***"
cat $CLIENT_LOG.${SUBTEST}.cuda_pool_fallback
RET=1
fi
set -e

kill $SERVER_PID
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated
wait $SERVER_PID
Comment thread
mattwittwer marked this conversation as resolved.
Outdated

# The test is only meaningful if the fallback actually happened.
if ! grep -q "falling back to pinned system memory" $SERVER_LOG; then
echo -e "\n***\n*** Expected the CUDA memory pool to be exhausted and fall back to pinned memory. \n***"
RET=1
fi
SERVER_ARGS="${SAVED_SERVER_ARGS}"
SERVER_LOG="./io_server.log"

rm -rf models && mkdir models
mkdir -p models/dlpack_empty_output/1/
cp ../../python_models/dlpack_empty_output/model.py ./models/dlpack_empty_output/1/
Expand Down
Loading