diff --git a/docs/bitrise-ci/testing/deploying-and-viewing-test-results.mdx b/docs/bitrise-ci/testing/deploying-and-viewing-test-results.mdx index b6f5fa44..8da784b3 100644 --- a/docs/bitrise-ci/testing/deploying-and-viewing-test-results.mdx +++ b/docs/bitrise-ci/testing/deploying-and-viewing-test-results.mdx @@ -1,7 +1,7 @@ --- title: "Deploying and viewing test results" description: "View your test results in one place after a Bitrise CI build. All tests will be available on the build page, even if you have multiple testing Steps in your Workflows." -sidebar_position: 7 +sidebar_position: 8 slug: /bitrise-ci/testing/deploying-and-viewing-test-results --- @@ -30,6 +30,7 @@ The supported testing Steps are the following: - **iOS Device Testing**: [Device testing for iOS](/bitrise-ci/testing/device-testing-with-firebase/device-testing-for-ios). - **Virtual Device Testing for Android**: [Device testing for Android](/bitrise-ci/testing/device-testing-with-firebase/device-testing-for-android). - **Flutter Test**: [Testing Flutter apps](/bitrise-ci/testing/testing-flutter-apps/running-the-dart-analyzer-on-bitrise). +- **Maestro Test**: [Running Maestro tests on Bitrise](/bitrise-ci/testing/running-maestro-tests-on-bitrise). If you use any of these Steps, make sure you have the **Deploy to Bitrise.io** Step in your Workflow. Bitrise will automatically deploy your test results to the build page. diff --git a/docs/bitrise-ci/testing/detecting-and-quarantining-flaky-tests.mdx b/docs/bitrise-ci/testing/detecting-and-quarantining-flaky-tests.mdx index 50d2fb84..f1bb264f 100644 --- a/docs/bitrise-ci/testing/detecting-and-quarantining-flaky-tests.mdx +++ b/docs/bitrise-ci/testing/detecting-and-quarantining-flaky-tests.mdx @@ -1,6 +1,6 @@ --- title: "Detecting and quarantining flaky tests" -sidebar_position: 8 +sidebar_position: 9 slug: /bitrise-ci/testing/detecting-and-quarantining-flaky-tests --- diff --git a/docs/bitrise-ci/testing/measuring-your-code-coverage-with-codecov.mdx b/docs/bitrise-ci/testing/measuring-your-code-coverage-with-codecov.mdx index e341115e..6be1c8bf 100644 --- a/docs/bitrise-ci/testing/measuring-your-code-coverage-with-codecov.mdx +++ b/docs/bitrise-ci/testing/measuring-your-code-coverage-with-codecov.mdx @@ -1,6 +1,6 @@ --- title: "Measuring your code coverage with Codecov" -sidebar_position: 9 +sidebar_position: 10 slug: /bitrise-ci/testing/measuring-your-code-coverage-with-codecov --- diff --git a/docs/bitrise-ci/testing/running-maestro-tests-on-bitrise.mdx b/docs/bitrise-ci/testing/running-maestro-tests-on-bitrise.mdx new file mode 100644 index 00000000..6675f835 --- /dev/null +++ b/docs/bitrise-ci/testing/running-maestro-tests-on-bitrise.mdx @@ -0,0 +1,247 @@ +--- +title: "Running Maestro tests on Bitrise" +description: "Run your Maestro flows on an Android emulator or iOS simulator in a Bitrise Workflow and view the results, with screenshots and recordings, on the Test Reports page." +sidebar_position: 6 +slug: /bitrise-ci/testing/running-maestro-tests-on-bitrise +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import GlossTerm from '@site/src/components/GlossTerm'; + +The [**Maestro Test**](https://github.com/bitrise-steplib/bitrise-step-maestro-test) Step runs your [Maestro](https://maestro.dev) flows on an Android emulator or iOS simulator on the Bitrise build machine, not on Maestro Cloud, and shows the results on the [Test reports](/bitrise-ci/testing/deploying-and-viewing-test-results) page. Each flow is a test case, with its screenshots, recordings, and logs attached. + +## Running Maestro flows in a Workflow + +A Maestro Workflow builds your app, starts a device, runs the flows, and uploads the results. The **Maestro Test** Step only runs the flows: it doesn't build your app or start a device, so add a build Step and a Step that starts the device before it, and the **Deploy to Bitrise.io** Step after it. The procedures below set this up for Android and iOS apps. + +### Android apps + + + + +1. Make sure that you have: + + - Maestro flows in your repository. By default, the Step runs the flows in the `.maestro` folder. To run other flow files or folders, list them in the **Flow path** input. To learn how to write flows, see the [Maestro documentation](https://docs.maestro.dev). + - An `.apk` of your app. Any framework works, including React Native and Flutter. The Step doesn't run Maestro's web flows. + - A Linux Android [stack](/bitrise-ci/configure-builds/configuring-build-settings/setting-the-stack-for-your-builds). Android emulators don't run on macOS stacks. + +1. Open the Workflow you want to use in the Workflow Editor. +1. Add the [**Android Build**](https://github.com/bitrise-steplib/bitrise-step-android-build) Step, and set its **Module** and **Variant** inputs. +1. Add the [**AVD Manager**](https://github.com/bitrise-steplib/steps-avd-manager) Step, and set the emulator in its **Android API Level**, **OS Tag**, and **ABI** inputs. The ABI must match the CPU architecture of the stack. + + :::important + + Only one emulator or simulator can run on the build machine: if there's none or more than one, the Step fails. + + ::: +1. Add the **Maestro Test** Step, and set its **App path** input to `$BITRISE_APK_PATH`. +1. Add the [**Deploy to Bitrise.io**](https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io) Step. + + + + +1. Make sure that you have: + + - Maestro flows in your repository. By default, the Step runs the flows in the `.maestro` folder. To run other flow files or folders, list them in the **Flow path** input. To learn how to write flows, see the [Maestro documentation](https://docs.maestro.dev). + - An `.apk` of your app. Any framework works, including React Native and Flutter. The Step doesn't run Maestro's web flows. + - A Linux Android [stack](/bitrise-ci/configure-builds/configuring-build-settings/setting-the-stack-for-your-builds). Android emulators don't run on macOS stacks. + +1. Open your app's Configuration YAML file. +1. Add the `android-build` Step to your Workflow, and set its `module` and `variant` inputs. +1. Add the `avd-manager` Step, and set the emulator in its `api_level`, `tag`, and `abi` inputs. The ABI must match the CPU architecture of the stack. + + :::important + + Only one emulator or simulator can run on the build machine: if there's none or more than one, the Step fails. + + ::: +1. Add the `maestro-test` Step, and set its `app_path` input to `$BITRISE_APK_PATH`. +1. Add the `deploy-to-bitrise-io` Step. + +```yaml +workflows: + maestro-android: + steps: + - git-clone: {} + - android-build: + inputs: + - module: app + - variant: debug + - avd-manager: + inputs: + - api_level: 34 + - tag: google_apis + - abi: x86_64 + - maestro-test: + inputs: + - app_path: $BITRISE_APK_PATH + - deploy-to-bitrise-io: {} +``` + + + + +An emulator can't install an Android App Bundle (`.aab`). If you build one, convert it to an APK with the **Export Universal APK** Step before the **Maestro Test** Step. + +### iOS apps + + + + +1. Make sure that you have: + + - Maestro flows in your repository. By default, the Step runs the flows in the `.maestro` folder. To run other flow files or folders, list them in the **Flow path** input. To learn how to write flows, see the [Maestro documentation](https://docs.maestro.dev). + - An `.app` of your app built for the iOS simulator. Any framework works, including React Native and Flutter. The Step doesn't run Maestro's web flows. + - A macOS [stack](/bitrise-ci/configure-builds/configuring-build-settings/setting-the-stack-for-your-builds). + +1. Open the Workflow you want to use in the Workflow Editor. +1. Add the [**Xcode Build for Simulator**](https://github.com/bitrise-steplib/steps-xcode-build-for-simulator) Step, and set its **Project path** and **Scheme** inputs. + + :::important + + Only one emulator or simulator can run on the build machine: if there's none or more than one, the Step fails. + + ::: +1. Add the [**Xcode Start Simulator**](https://github.com/bitrise-steplib/bitrise-step-xcode-start-simulator) Step, and set the simulator in its **Device destination specifier** input. +1. Set the **Simulator boot timeout (in seconds)** input, so that the Step waits for the simulator to boot. Otherwise, the **Maestro Test** Step might not find it. +1. Add the **Maestro Test** Step, and set its **App path** input to `$BITRISE_APP_DIR_PATH`. +1. Add the [**Deploy to Bitrise.io**](https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io) Step. + + + + +1. Make sure that you have: + + - Maestro flows in your repository. By default, the Step runs the flows in the `.maestro` folder. To run other flow files or folders, list them in the **Flow path** input. To learn how to write flows, see the [Maestro documentation](https://docs.maestro.dev). + - An `.app` of your app built for the iOS simulator. Any framework works, including React Native and Flutter. The Step doesn't run Maestro's web flows. + - A macOS [stack](/bitrise-ci/configure-builds/configuring-build-settings/setting-the-stack-for-your-builds). + +1. Open your app's Configuration YAML file. +1. Add the `xcode-build-for-simulator` Step to your Workflow, and set its `project_path` and `scheme` inputs. + + :::important + + Only one emulator or simulator can run on the build machine: if there's none or more than one, the Step fails. + + ::: +1. Add the `xcode-start-simulator` Step, and set the simulator in its `destination` input. +1. Set the `wait_for_boot_timeout` input, so that the Step waits for the simulator to boot. Otherwise, the `maestro-test` Step might not find it. +1. Add the `maestro-test` Step, and set its `app_path` input to `$BITRISE_APP_DIR_PATH`. +1. Add the `deploy-to-bitrise-io` Step. + +```yaml +workflows: + maestro-ios: + steps: + - git-clone: {} + - xcode-build-for-simulator: + inputs: + - project_path: MyApp.xcodeproj + - scheme: MyApp + - xcode-start-simulator: + inputs: + - destination: platform=iOS Simulator,name=Bitrise iOS default,OS=latest + - wait_for_boot_timeout: 120 + - maestro-test: + inputs: + - app_path: $BITRISE_APP_DIR_PATH + - deploy-to-bitrise-io: {} +``` + + + + +## Passing variables to flows + +Maestro passes every Environment Variable that starts with `MAESTRO_` to your flows. Store sensitive values as [Secrets](/bitrise-ci/configure-builds/secrets) with this prefix. For example, a Secret named `MAESTRO_PASSWORD` is `${MAESTRO_PASSWORD}` in a flow: + +```yaml +appId: com.example.myapp +--- +- launchApp +- tapOn: "Password" +- inputText: ${MAESTRO_PASSWORD} +``` + +## Selecting flows with tags + +Tag your flows in the flow file: + +```yaml +appId: com.example.myapp +tags: + - smoke +--- +- launchApp +``` + +Then use **Include tags** to run only the flows that have any of the listed tags, and **Exclude tags** to skip the flows that have any of them. Separate tags with commas or new lines. + +```yaml +- maestro-test: + inputs: + - app_path: $BITRISE_APK_PATH + - include_tags: smoke + - exclude_tags: flaky +``` + +## Running flows on several devices in parallel + +Maestro's own sharding needs several devices on one machine, which the Step doesn't support. Instead, run copies of the Workflow in parallel in a Pipeline, each on its own virtual machine: [Parallelism in Pipelines](/bitrise-ci/workflows-and-pipelines/build-pipelines/configuring-a-bitrise-pipeline#running-variations-of-the-same-workflow). + +1. Tag each flow with the copy that runs it: `shard0`, `shard1`, and so on. +1. Set **Include tags** to `shard$BITRISE_IO_PARALLEL_INDEX`. Every copy gets its own zero-based index in this Env Var. +1. Set a different **Test name** for each copy, so their results don't mix on the Test Reports page. + +```yaml +pipelines: + ui-tests: + workflows: + maestro-tests: + parallel: 2 + +workflows: + maestro-tests: + steps: + - git-clone: {} + - android-build: + inputs: + - module: app + - variant: debug + - avd-manager: + inputs: + - api_level: 34 + - tag: google_apis + - abi: x86_64 + - maestro-test: + inputs: + - app_path: $BITRISE_APK_PATH + - include_tags: shard$BITRISE_IO_PARALLEL_INDEX + - test_name: Maestro shard $BITRISE_IO_PARALLEL_INDEX + - deploy-to-bitrise-io: {} +``` + +## Viewing the results + +The results appear on the Test Reports page under the name in the **Test name** input, failed flows included. For the **Deploy to Bitrise.io** versions that show attachments, see [Deploying and viewing test results](/bitrise-ci/testing/deploying-and-viewing-test-results). + +:::tip[Give every flow a unique name] + +The Step attaches files to a flow by its name. If two flows share a name, their files aren't attached. + +::: + +The Step also exports the JUnit report in `$MAESTRO_JUNIT_PATH`, and Maestro's whole output folder as a zip in `$MAESTRO_TEST_OUTPUT_ZIP_PATH`. The zip is uploaded as a build artifact. + +## Troubleshooting + +The first line of the Step's error tells you what failed: + +| Error | Cause | +| --- | --- | +| `Failed to process Step inputs` | An input has an invalid value. The next lines name it. | +| `Failed to install the Maestro CLI` | The Maestro CLI couldn't be downloaded or verified. | +| `Failed to install the app` | No device is running, more than one is running, or the install failed. Make sure the Step that starts the device waits for the boot. | +| `Maestro flows failed` | At least one flow failed. See the Test Reports page. | +| `Maestro failed without a failing flow` | Maestro stopped before any flow failed, for example on a flow file it couldn't parse. See the Step's log. | diff --git a/docs/bitrise-ci/testing/test-sharding.mdx b/docs/bitrise-ci/testing/test-sharding.mdx index ba521f4a..85830012 100644 --- a/docs/bitrise-ci/testing/test-sharding.mdx +++ b/docs/bitrise-ci/testing/test-sharding.mdx @@ -1,7 +1,7 @@ --- title: "Test sharding" description: "Bitrise provides easy to use test sharding optimization features out of the box, depending on your project type and whether you want Bitrise to provide the the sharding optimization calculation." -sidebar_position: 6 +sidebar_position: 7 slug: /bitrise-ci/testing/test-sharding --- @@ -51,6 +51,8 @@ Or with yarn: yarn test --ci --silent --shard=$((BITRISE_IO_PARALLEL_INDEX + 1))/$BITRISE_IO_PARALLEL_TOTAL ``` +For Maestro flows, tag each flow with its shard and select the flows by `$BITRISE_IO_PARALLEL_INDEX`: [Running flows on several devices in parallel](/bitrise-ci/testing/running-maestro-tests-on-bitrise#running-flows-on-several-devices-in-parallel). + To read more about how to configure parallelism, check our guide: [Parallelism in Pipelines](/bitrise-ci/workflows-and-pipelines/build-pipelines/configuring-a-bitrise-pipeline#running-variations-of-the-same-workflow). diff --git a/docs/bitrise-ci/testing/testing-flutter-apps/running-unit-and-ui-tests-on-flutter-apps.mdx b/docs/bitrise-ci/testing/testing-flutter-apps/running-unit-and-ui-tests-on-flutter-apps.mdx index 0e2d8679..4c4b0287 100644 --- a/docs/bitrise-ci/testing/testing-flutter-apps/running-unit-and-ui-tests-on-flutter-apps.mdx +++ b/docs/bitrise-ci/testing/testing-flutter-apps/running-unit-and-ui-tests-on-flutter-apps.mdx @@ -15,6 +15,12 @@ The **Flutter Test** Step can also generate code coverage reports. ::: +:::tip[UI testing with Maestro] + +To test your Flutter app's UI from the outside, with flows that tap and type on an emulator or simulator, check out Maestro: [Running Maestro tests on Bitrise](/bitrise-ci/testing/running-maestro-tests-on-bitrise). + +::: + To run tests using **Flutter Test**: diff --git a/docs/bitrise-ci/testing/testing-react-native-apps/running-unit-and-ui-tests-for-react-native-apps.mdx b/docs/bitrise-ci/testing/testing-react-native-apps/running-unit-and-ui-tests-for-react-native-apps.mdx index 126043cc..d21bf930 100644 --- a/docs/bitrise-ci/testing/testing-react-native-apps/running-unit-and-ui-tests-for-react-native-apps.mdx +++ b/docs/bitrise-ci/testing/testing-react-native-apps/running-unit-and-ui-tests-for-react-native-apps.mdx @@ -14,7 +14,7 @@ You can run all kinds of tests for your React Native app on Bitrise: unit tests, :::tip[E2E testing] -For end-to-end testing, check out Detox: [Running Detox tests on Bitrise](/bitrise-ci/testing/testing-react-native-apps/running-detox-tests-on-bitrise). +For end-to-end testing, check out Detox or Maestro: [Running Detox tests on Bitrise](/bitrise-ci/testing/testing-react-native-apps/running-detox-tests-on-bitrise), [Running Maestro tests on Bitrise](/bitrise-ci/testing/running-maestro-tests-on-bitrise). :::