Skip to content

[WIP] test: add e2e testing infrastructure (iOS + Android) #2777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
testableapple wants to merge 13 commits into
base: master
Choose a base branch
from
Draft

[WIP] test: add e2e testing infrastructure (iOS + Android) #2777

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
2ed3dcf
Proof of concept
testableapple Jun 18, 2026
cde7d0a
Foundations & app-side plumbing
testableapple Jun 18, 2026
900b21b
Mock server client + control robots
testableapple Jun 18, 2026
e8d7845
Add sample pages
testableapple Jun 18, 2026
8f0bef7
DSL, robots & base
testableapple Jun 18, 2026
d24a40a
Git ignore
testableapple Jun 22, 2026
a980b12
Implement CI stuff
testableapple Jun 22, 2026
dc20586
Merge branch 'master' into ci/e2e-testing-infra
testableapple Jun 22, 2026
80df969
ci(e2e): gate e2e lanes with sources_matrix + is_check_required
testableapple Jun 22, 2026
ceb1088
ci(e2e): drop redundant comment from e2e workflow
testableapple Jun 22, 2026
f6ec10c
Potential fix for pull request finding 'CodeQL / Workflow does not co…
testableapple Jun 22, 2026
63ff234
Potential fix for pull request finding 'CodeQL / Workflow does not co…
testableapple Jun 22, 2026
778a29b
Do not skip tests
testableapple Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions .github/workflows/e2e_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
name: e2e

on:
workflow_dispatch:
pull_request:
types: [labeled, synchronize]

# Heavy mobile e2e — keep off the default PR path; run on demand or when a PR
# carries the `e2e` label. The nightly schedule lives in e2e_test_cron.yml.
concurrency:
group: e2e-${{ github.ref }}
cancel-in-progress: true

env:
flutter_version: "3.x"

Comment on lines +1 to +19

@coderabbitai coderabbitai Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add a workflow-level permissions block.

The workflow uses default permissions, which are overly broad. Since this workflow only needs to checkout code and upload artifacts, add a minimal permissions block.

🛡️ Proposed fix 
 concurrency:
   group: e2e-${{ github.ref }}
   cancel-in-progress: true
 
+permissions:
+  contents: read
+
 env:
   flutter_version: "3.x"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
name: e2e
on:
workflow_dispatch:
pull_request:
types: [labeled, synchronize]
# Heavy mobile e2e — keep off the default PR path; run on demand or when a PR
# carries the `e2e` label. The nightly schedule lives in e2e_test_cron.yml.
concurrency:
group: e2e-${{ github.ref }}
cancel-in-progress: true
env:
flutter_version: "3.x"
name: e2e
on:
workflow_dispatch:
pull_request:
types: [labeled, synchronize]
# Heavy mobile e2e — keep off the default PR path; run on demand or when a PR
# carries the `e2e` label. The nightly schedule lives in e2e_test_cron.yml.
concurrency:
group: e2e-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
flutter_version: "3.x"
🧰 Tools
🪛 zizmor (1.25.2)

[warning] 1-124: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e_test.yml around lines 1 - 16, Add a workflow-level
permissions block to the e2e workflow to restrict permissions to only what is
needed. The permissions block should be added after the 'on:' section (after the
pull_request trigger configuration) and before the 'concurrency:' section. Since
this workflow only needs to checkout code and upload artifacts, add a
permissions block that specifies contents: read for code checkout, and any other
minimal permissions required for the specific operations the workflow performs.
This replaces the default overly-broad permissions with an explicitly minimal
set.

Source: Linters/SAST tools

jobs:
android:
if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'e2e')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6

- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"

- uses: subosito/flutter-action@v2
with:
flutter-version: ${{ env.flutter_version }}
channel: stable
cache-key: flutter-:os:-:channel:-:version:-:arch:-:hash:-${{ hashFiles('**/pubspec.lock') }}

- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
bundler-cache: true
working-directory: sample_app/android

- name: Bootstrap
run: |
flutter pub global activate melos
melos bootstrap
dart pub global activate patrol_cli

- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm

- name: Run e2e (Android emulator)
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: 34
arch: x86_64
profile: pixel_6
script: |
cd sample_app/android
bundle exec fastlane run_e2e_test device:emulator-5554 mock_server_branch:main

- name: Upload Allure results
if: always()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/android && bundle exec fastlane allure_upload

- uses: actions/upload-artifact@v4
if: always()
with:
name: e2e-android-logs
path: |
sample_app/stream-chat-test-mock-server/logs
sample_app/build/app/reports

ios:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'e2e')
runs-on: macos-15
steps:
- uses: actions/checkout@v6

- uses: subosito/flutter-action@v2
with:
flutter-version: ${{ env.flutter_version }}
channel: stable
cache-key: flutter-:os:-:channel:-:version:-:arch:-:hash:-${{ hashFiles('**/pubspec.lock') }}

- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
bundler-cache: true
working-directory: sample_app/ios

- name: Bootstrap
run: |
flutter pub global activate melos
melos bootstrap
dart pub global activate patrol_cli

- name: Boot simulator
run: |
device_id=$(xcrun simctl list devices available --json \
| python3 -c "import sys,json;d=json.load(sys.stdin);print(next(x['udid'] for r in d['devices'].values() for x in r if x['name'].startswith('iPhone')))")
echo "device_id=$device_id" >> "$GITHUB_ENV"
xcrun simctl boot "$device_id"

- name: Run e2e (iOS simulator)
run: |
cd sample_app/ios
bundle exec fastlane run_e2e_test device:${{ env.device_id }} mock_server_branch:main

- name: Upload Allure results
if: always()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/ios && bundle exec fastlane allure_upload

- uses: actions/upload-artifact@v4
if: always()
with:
name: e2e-ios-logs
path: sample_app/stream-chat-test-mock-server/logs

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
168 changes: 168 additions & 0 deletions .github/workflows/e2e_test_cron.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
name: E2E Tests Nightly

on:
schedule:
- cron: "0 1 * * 1-5" # weeknights at 01:00 UTC
workflow_dispatch:
inputs:
mock_server_branch:
description: "Mock server branch"
type: string
required: true
default: main

concurrency:
group: e2e-nightly-${{ github.ref }}
cancel-in-progress: true

env:
flutter_version: "3.x"
mock_server_branch: ${{ github.event.inputs.mock_server_branch || 'main' }}
Comment on lines +1 to +23

@coderabbitai coderabbitai Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add a workflow-level permissions block.

Same as the on-demand workflow, this needs explicit minimal permissions.

🛡️ Proposed fix 
 concurrency:
   group: e2e-nightly-${{ github.ref }}
   cancel-in-progress: true
 
+permissions:
+  contents: read
+
 env:
   flutter_version: "3.x"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
name: E2E Tests Nightly
on:
schedule:
- cron: "0 1 * * 1-5" # weeknights at 01:00 UTC
workflow_dispatch:
inputs:
mock_server_branch:
description: "Mock server branch"
type: string
required: true
default: main
concurrency:
group: e2e-nightly-${{ github.ref }}
cancel-in-progress: true
env:
flutter_version: "3.x"
mock_server_branch: ${{ github.event.inputs.mock_server_branch || 'main' }}
name: E2E Tests Nightly
on:
schedule:
- cron: "0 1 * * 1-5" # weeknights at 01:00 UTC
workflow_dispatch:
inputs:
mock_server_branch:
description: "Mock server branch"
type: string
required: true
default: main
concurrency:
group: e2e-nightly-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
flutter_version: "3.x"
mock_server_branch: ${{ github.event.inputs.mock_server_branch || 'main' }}
🧰 Tools
🪛 zizmor (1.25.2)

[warning] 1-169: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e_test_cron.yml around lines 1 - 20, The E2E Tests
Nightly workflow in the e2e_test_cron.yml file is missing explicit minimal
permissions at the workflow level. Add a workflow-level permissions block to the
file (after the concurrency section or at the top level, similar to how other
workflow files are structured) and define explicit minimal permissions that
match what is configured in the corresponding on-demand workflow. This ensures
the workflow operates with the principle of least privilege.

Source: Linters/SAST tools


jobs:
android:
if: github.repository == 'GetStream/stream-chat-flutter'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
api-level: [34, 31, 28]
steps:
- uses: actions/checkout@v6

- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"

- uses: subosito/flutter-action@v2
with:
flutter-version: ${{ env.flutter_version }}
channel: stable
cache-key: flutter-:os:-:channel:-:version:-:arch:-:hash:-${{ hashFiles('**/pubspec.lock') }}

- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
bundler-cache: true
working-directory: sample_app/android

- name: Bootstrap
run: |
flutter pub global activate melos
melos bootstrap
dart pub global activate patrol_cli

- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm

- name: Create Allure launch
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/android && bundle exec fastlane allure_launch
Comment on lines +65 to +68

@coderabbitai coderabbitai Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Each matrix cell creates its own Allure launch instead of sharing one.

With the matrix strategy, each Android (3 cells) and iOS (2 cells) job independently runs fastlane allure_launch, creating 5 separate launches per nightly run. The PR objective states results should upload to a shared launch.

Consider extracting the launch creation into a separate setup job that runs once and outputs the launch ID for matrix jobs to consume:

🔧 Architectural suggestion 
jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      launch_id: ${{ steps.create.outputs.launch_id }}
    steps:
      - uses: actions/checkout@v6
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: "3.3"
          bundler-cache: true
          working-directory: sample_app/android
      - name: Create Allure launch
        id: create
        env:
          ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
        run: |
          cd sample_app/android
          launch_id=$(bundle exec fastlane allure_launch)
          echo "launch_id=$launch_id" >> "$GITHUB_OUTPUT"

  android:
    needs: setup
    # ... existing config ...
    steps:
      # Remove "Create Allure launch" step
      # In "Upload Allure results":
      - name: Upload Allure results
        env:
          ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
          ALLURE_LAUNCH_ID: ${{ needs.setup.outputs.launch_id }}
        run: cd sample_app/android && bundle exec fastlane allure_upload

Also applies to: 123-126

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e_test_cron.yml around lines 62 - 65, The current
workflow creates a separate Allure launch for each matrix cell (3 Android
variants and 2 iOS variants), but they should all upload to a single shared
launch. Create a new `setup` job that runs once before the matrix jobs to
generate a launch_id, then extract that launch_id as an output so the android
and ios jobs can consume it via `needs: setup`. Remove the "Create Allure
launch" step from both the android and ios job definitions, and update their
"Upload Allure results" steps to use the shared launch_id from the setup job
output instead of creating new launches in each matrix cell.


- name: Run e2e (Android emulator)
uses: reactivecircus/android-emulator-runner@v2
timeout-minutes: 90
with:
api-level: ${{ matrix.api-level }}
arch: x86_64
profile: pixel_6
emulator-options: -no-snapshot-save -no-window -no-audio -no-boot-anim -gpu swiftshader_indirect
script: cd sample_app/android && bundle exec fastlane run_e2e_test device:emulator-5554 mock_server_branch:${{ env.mock_server_branch }}

- name: Upload Allure results
if: success() || failure()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/android && bundle exec fastlane allure_upload

- name: Remove Allure launch (on cancel)
if: cancelled()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/android && bundle exec fastlane allure_launch_removal

- uses: actions/upload-artifact@v4
if: failure()
with:
name: e2e-nightly-android-${{ matrix.api-level }}
path: sample_app/stream-chat-test-mock-server/logs

ios:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
if: github.repository == 'GetStream/stream-chat-flutter'
runs-on: macos-15
strategy:
fail-fast: false
matrix:
device: ["iPhone 16", "iPhone 15"]
steps:
- uses: actions/checkout@v6

- uses: subosito/flutter-action@v2
with:
flutter-version: ${{ env.flutter_version }}
channel: stable
cache-key: flutter-:os:-:channel:-:version:-:arch:-:hash:-${{ hashFiles('**/pubspec.lock') }}

- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
bundler-cache: true
working-directory: sample_app/ios

- name: Bootstrap
run: |
flutter pub global activate melos
melos bootstrap
dart pub global activate patrol_cli

- name: Create Allure launch
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/ios && bundle exec fastlane allure_launch

- name: Boot simulator
run: |
device_id=$(xcrun simctl list devices available --json \
| python3 -c "import sys,json;d=json.load(sys.stdin);print(next(x['udid'] for r in d['devices'].values() for x in r if x['name']=='${{ matrix.device }}'))")
echo "device_id=$device_id" >> "$GITHUB_ENV"
xcrun simctl boot "$device_id"

- name: Run e2e (iOS simulator)
run: cd sample_app/ios && bundle exec fastlane run_e2e_test device:${{ env.device_id }} mock_server_branch:${{ env.mock_server_branch }}

- name: Upload Allure results
if: success() || failure()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/ios && bundle exec fastlane allure_upload

- name: Remove Allure launch (on cancel)
if: cancelled()
env:
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
run: cd sample_app/ios && bundle exec fastlane allure_launch_removal

- uses: actions/upload-artifact@v4
if: failure()
with:
name: e2e-nightly-ios-${{ matrix.device }}
path: sample_app/stream-chat-test-mock-server/logs

slack:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
name: Slack Report
runs-on: ubuntu-latest
needs: [android, ios]
if: failure() && github.event_name == 'schedule'
steps:
- uses: 8398a7/action-slack@v3
with:
status: failure
text: "Nightly e2e failed 🌙"
fields: repo,commit,author,workflow
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_NIGHTLY_CHECKS }}

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
8 changes: 8 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@
*.class
*.log
*.pyc
# E2E: mock-server checkout cloned by the `start_mock_server` Fastlane lane
/sample_app/stream-chat-test-mock-server/
# E2E: Patrol-generated test bundle (regenerated on every `patrol test`)
/sample_app/integration_test/test_bundle.dart
# E2E: Allure results assembled from test runs
/sample_app/allure-results/
# E2E: allurectl binary downloaded by the Fastlane allure lanes
/sample_app/allurectl
*.swp
.DS_Store
.atom/
Expand Down
10 changes: 10 additions & 0 deletions melos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ command:
freezed: ^3.0.0
json_serializable: ^6.13.2
mocktail: ^1.0.5
patrol: ^4.6.1
path: ^1.9.1
path_provider_platform_interface: ^2.1.2
plugin_platform_interface: ^2.1.8
Expand Down Expand Up @@ -203,12 +204,21 @@ scripts:
# `docs_*` are excluded — their goldens commit only the platform/macOS
# variant and would fail on Linux CI runners. They're regenerated by
# `update:goldens:docs` on a dedicated macOS workflow instead.
# Note: `flutter test` runs `test/` only, so `integration_test/` (the e2e
# suite, run via Patrol) is never picked up here.
run: melos exec -c 4 --fail-fast --ignore="docs_*" -- "flutter test --coverage"
description: Run Flutter tests for a specific package in this project.
packageFilters:
flutter: true
dirExists: test

e2e:run:
run: cd sample_app && patrol test

@xsahil03x xsahil03x Jun 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add a --depends-on="patrol" instead of navigating to sample_app and run the test command instead.

Check how we do it in the update:goldens command

@testableapple testableapple Jun 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will check it out, thanks a mill @xsahil03x

just a quick note, this PR is still in draft, i need to test and adjust a lot of things

@xsahil03x xsahil03x Jun 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes sure, I was just taking a quick look

description: >
Run the Patrol e2e suite in sample_app. Start the mock-server driver first
(`fastlane start_mock_server`), or use the `run_e2e_test` Fastlane lane to
orchestrate the mock server and tests in one step.

update:goldens:
run: melos exec -c 1 --depends-on="alchemist" -- "flutter test --tags golden --update-goldens"
description: Update golden files for all packages in this project.
Expand Down
Loading
Loading