Skip to content

Azure CLI connection timeout when running a long test using the ScenarioTest framework #26983

Description

@patrykkulik-microsoft

Describe the bug

My team is in the process of creating a new Azure CLI extension. As part of creating this extension, we wanted to add integration tests which would test our new commands. According to this document: https://github.com/Azure/azure-cli/blob/dev/doc/authoring_tests.md in the azure-cli repository, I should be able to author integration tests by taking advantage of the ScenarioTest class.

When I run azdev test aosm the publish command in our extension manages to deploy a few resources but fails after a few minutes with the error attached below.

Interestingly, if I reduce the number of deployments that need to be done by this command, the command manages to reach completion. This (and the error) suggest to me that the testing framework implements some sort of timeout which I cannot modify and which does not allow my command to complete running and succeed. Is this correct? Is there anything I can do to remove this timeout condition?

Related command

azdev test aosm

Errors

self = <azure.core.pipeline.transport._requests_basic.RequestsTransport object at 0x7f60b2b5b580>
request = <HttpRequest [POST], url: 'REDACTED'>
kwargs = {'stream': False}, response = None
error = ServiceResponseError("('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))")
connection_timeout = 300, timeout = (300, 300), read_timeout = 300

    def send(self, request, **kwargs): # type: ignore
        """Send request object according to configuration.

        :param request: The request object to be sent.
        :type request: ~azure.core.pipeline.transport.HttpRequest
        :return: An HTTPResponse object.
        :rtype: ~azure.core.pipeline.transport.HttpResponse

        :keyword requests.Session session: will override the driver session and use yours.
         Should NOT be done unless really required. Anything else is sent straight to requests.
        :keyword dict proxies: will define the proxy to use. Proxy is a dict (protocol, url)
        """
        self.open()
        response = None
        error = None    # type: Optional[AzureErrorUnion]

        try:
            connection_timeout = kwargs.pop('connection_timeout', self.connection_config.timeout)

            if isinstance(connection_timeout, tuple):
                if 'read_timeout' in kwargs:
                    raise ValueError('Cannot set tuple connection_timeout and read_timeout together')
                _LOGGER.warning("Tuple timeout setting is deprecated")
                timeout = connection_timeout
            else:
                read_timeout = kwargs.pop('read_timeout', self.connection_config.read_timeout)
                timeout = (connection_timeout, read_timeout)
            response = self.session.request(  # type: ignore
                request.method,
                request.url,
                headers=request.headers,
                data=request.data,
                files=request.files,
                verify=kwargs.pop('connection_verify', self.connection_config.verify),
                timeout=timeout,
                cert=kwargs.pop('connection_cert', self.connection_config.cert),
                allow_redirects=False,
                **kwargs)
            response.raw.enforce_content_length = True

        except (urllib3.exceptions.NewConnectionError, urllib3.exceptions.ConnectTimeoutError) as err:
            error = ServiceRequestError(err, error=err)
        except requests.exceptions.ReadTimeout as err:
            error = ServiceResponseError(err, error=err)
        except requests.exceptions.ConnectionError as err:
            if err.args and isinstance(err.args[0], urllib3.exceptions.ProtocolError):
                error = ServiceResponseError(err, error=err)
            else:
                error = ServiceRequestError(err, error=err)
        except requests.exceptions.ChunkedEncodingError as err:
            msg = err.__str__()
            if 'IncompleteRead' in msg:
                _LOGGER.warning("Incomplete download: %s", err)
                error = IncompleteReadError(err, error=err)
            else:
                _LOGGER.warning("Unable to stream download: %s", err)
                error = HttpResponseError(err, error=err)
        except requests.RequestException as err:
            error = ServiceRequestError(err, error=err)

        if error:
>           raise error
E           azure.core.exceptions.ServiceResponseError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

../../../../.virtualenvs/az-cli-env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py:361: ServiceResponseError

Issue script & Debug output

no different than the above error message

Expected behavior

I expect the test to pass.

Environment Summary

azure-cli 2.48.1 *

core 2.48.1 *
telemetry 1.0.8

Extensions:
connectedk8s 1.3.5
customlocation 0.1.3
k8s-extension 1.3.6
resource-graph 2.1.0
aosm 0.1.0 (dev) /home/developer/Projects/azure-cli-extensions/src/aosm

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Python location '/home/developer/.virtualenvs/az-cli-env/bin/python'
Extensions directory '/home/developer/.azure/cliextensions'
Development extension sources:
/home/developer/Projects/azure-cli-extensions

Python (Linux) 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

Activity

  1. added
    bugThis issue requires a change to an existing behavior in the product in order to be resolved.
    on Jul 24, 2023
  2. azure-client-tools-bot-prd commented on Jul 24, 2023

    @azure-client-tools-bot-prd

    Hi patrykkulik-microsoft,

    2.48.1 is not the latest Azure CLI(2.50.0).

    Please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

  3. ghost added on Jul 24, 2023
  4. ghost added
    Graph(doesn't work with label-triggered comments; use Graph.Microsoft instead) az ad
    on Jul 24, 2023
  5. yonzhan commented on Jul 24, 2023

    @yonzhan
    Collaborator

    Thank you for opening this issue, we will look into it.

  6. ghost added this to the Backlog milestone on Jul 24, 2023
  7. removed their assignment
    on Aug 14, 2023
  8. wangzelin007 commented on Aug 14, 2023

    @wangzelin007
    Member

    Hi patrykkulik-microsoft,
    I don't think we put a timeout limit on tests, like this one running for more than an hour.
    image
    How long will your test run?
    Can your test be split into multiple smaller tests?
    Is there a PR for me to debug?

  9. patrykkulik-microsoft commented on Aug 14, 2023

    @patrykkulik-microsoft
    Author

    ZelinWang (@wangzelin007) thank you for your reply.

    The limit that that I am hitting is not on the test scope but on the self.cmd scope. One command is running for about 10 minutes and then I hit the error which I pasted above.

    The test itself runs for about 20min but as I said the test is not the problem and the command cannot be split into smaller commands.

    We have a PR set to the azure-cli-extensions repo: Azure/azure-cli-extensions#6426

    The problematic test is src/aosm/azext_aosm/tests/latest/test_aosm_vnf_publish_and_delete.py,
    Note: in that PR, this test has been commented out and I have added a while loop to try the command twice which makes it pass so you will need to remove the while loop to see the error. I would be happy to connect with you to help you reproduce this error.

  10. evelyn-ys commented on Aug 15, 2023

    @evelyn-ys
    Contributor

    If one http request needs to take so long as >10 min, why don't you return a poller for continuously querying status? The timeout limit comes from azure-core which is adopted by all clients, not only cli

  11. patrykkulik-microsoft commented on Aug 15, 2023

    @patrykkulik-microsoft
    Author

    It is not a single http request that needs 10minutes. I am running a self.cmd() method of the ScenarioTest class. I am using this method to run a command from our CLI extension which publishes multiple resources in Azure. This commands runs multiple HTTP requests. None of these individually take longer than 10mins but all of them together do.

    I am hitting an error during the runtime of this command with multiple HTTP requests, which seems to happen because the CLI command as a whole takes too long rather than any individual request. In fact, if I run the command twice, the second time in runs, some resources have already been published by it in the first run before the error happened so the second run takes less time and passes fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Auto-AssignAuto assign by botAuto-ResolveAuto resolve by botGraph(doesn't work with label-triggered comments; use Graph.Microsoft instead) az adact-identity-squadbugThis issue requires a change to an existing behavior in the product in order to be resolved.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions