Skip to content

[CI] Enhance AI-generated PR workflow with PR description reproducers and lint failure comments #3488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chuanqi129 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[CI] Enhance AI-generated PR workflow with PR description reproducers and lint failure comments #3488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 46 additions & 7 deletions .github/workflows/_linux_reproducer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ on:
type: string
default: '3.10'
description: Python version
pr_body_cmd:
type: string
default: ''
description: Reproducer command extracted from PR description (reproducer-cmd block)

permissions:
contents: read
Expand Down Expand Up @@ -66,13 +70,48 @@ jobs:
pytorch: ${{ inputs.pytorch }}
torch_xpu_ops: ${{ inputs.torch_xpu_ops }}
python: ${{ inputs.python }}
- name: Run reproducer tests
id: reproducer
- name: Get changed reproducer files
id: changed-repro
run: |
cd torch-xpu-ops
CHANGED_FILES=$(git diff --name-only --diff-filter=AM origin/${{ github.event.pull_request.base.ref }}...HEAD -- test/repro/test_*.py || true)
echo "files<<EOF" >> "$GITHUB_OUTPUT"
echo "$CHANGED_FILES" >> "$GITHUB_OUTPUT"
echo "EOF" >> "$GITHUB_OUTPUT"
Comment on lines +73 to +80
if [ -n "$CHANGED_FILES" ]; then
echo "Found changed reproducer files:"
echo "$CHANGED_FILES"
else
echo "No added/modified reproducer files in this PR."
fi
- name: Run reproducer files
id: reproducer-files
if: ${{ steps.changed-repro.outputs.files != '' }}
continue-on-error: true
run: |
set -o pipefail
cd pytorch
pytest ../torch-xpu-ops/test/repro/ -v 2>&1 | tee ${{ github.workspace }}/reproducer_output.txt
echo "=== Running changed reproducer files ==="
# Convert file paths to pytest-compatible paths under third_party/torch-xpu-ops/
REPRO_FILES=""
while IFS= read -r f; do
[ -z "$f" ] && continue
REPRO_FILES="$REPRO_FILES third_party/torch-xpu-ops/$f"
done <<< "${{ steps.changed-repro.outputs.files }}"
echo "Files: $REPRO_FILES"
pytest $REPRO_FILES -v 2>&1 | tee ${{ github.workspace }}/reproducer_output.txt
- name: Run reproducer command from PR description
id: reproducer-cmd
if: ${{ inputs.pr_body_cmd != '' }}
continue-on-error: true
env:
PR_BODY_CMD: ${{ inputs.pr_body_cmd }}
run: |
set -o pipefail
cd pytorch/third_party/torch-xpu-ops
echo "=== Running reproducer command from PR description ==="
echo "Command: $PR_BODY_CMD"
eval "$PR_BODY_CMD" 2>&1 | tee -a ${{ github.workspace }}/reproducer_output.txt
Comment on lines +110 to +114
- name: Upload reproducer log
if: ${{ ! cancelled() }}
uses: actions/upload-artifact@v4
Expand All @@ -81,10 +120,10 @@ jobs:
path: ${{ github.workspace }}/reproducer_output.txt
if-no-files-found: ignore
- name: Write summary
if: ${{ steps.reproducer.outcome == 'success' }}
run: echo "✅ Reproducer test passed. All tests in \`test/repro/\` executed successfully." >> $GITHUB_STEP_SUMMARY
if: ${{ steps.reproducer-files.outcome != 'failure' && steps.reproducer-cmd.outcome != 'failure' }}
run: echo "✅ Reproducer test passed. All reproducer tests executed successfully." >> $GITHUB_STEP_SUMMARY
Comment on lines 122 to +124
- name: Fail if reproducer failed
if: ${{ steps.reproducer.outcome == 'failure' }}
if: ${{ steps.reproducer-files.outcome == 'failure' || steps.reproducer-cmd.outcome == 'failure' }}
run: exit 1

summary:
Expand Down Expand Up @@ -145,7 +184,7 @@ jobs:
'1. Review the test failure output above (or the full log via the link)',
'2. Identify the root cause of the failure',
'3. Update the relevant source code to fix the issue',
'4. Ensure the reproducer in test/repro/ passes after your fix'
'4. Ensure the reproducer tests pass after your fix (files in test/repro/ and/or PR description command)'
].join('\n');
await github.rest.issues.createComment({
owner: context.repo.owner,
Expand Down
90 changes: 77 additions & 13 deletions .github/workflows/pull.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,56 +27,119 @@ jobs:
permissions:
contents: read
issues: write
outputs:
pr_body_cmd: ${{ steps.reproducer-check.outputs.pr_cmd }}
steps:
- name: Checkout torch-xpu-ops
uses: actions/checkout@v4
with:
path: torch-xpu-ops
- name: Run lint check
id: lint-python
continue-on-error: true
run: |
export ADDITIONAL_LINTRUNNER_ARGS="--skip CLANGTIDY,CLANGFORMAT,MERGE_CONFLICTLESS_CSV --all-files"
cd ./torch-xpu-ops
bash .github/scripts/lintrunner.sh
- name: Run lint check with Clang
id: lint-clang
continue-on-error: true
run: |
cd ./torch-xpu-ops
export ADDITIONAL_LINTRUNNER_ARGS="--take CLANGFORMAT --all-files"
bash .github/scripts/lintrunner.sh
- name: Comment on lint failure for AI-generated PR
if: >-
${{ (steps.lint-python.outcome == 'failure' || steps.lint-clang.outcome == 'failure')
&& contains(github.event.pull_request.labels.*.name, 'ai_generated') }}
env:
GH_TOKEN: ${{ secrets.MERGE_TOKEN }}
run: |
FAILED_STEPS=""
if [ "${{ steps.lint-python.outcome }}" = "failure" ]; then
FAILED_STEPS="Python lint (flake8/ruff)"
fi
if [ "${{ steps.lint-clang.outcome }}" = "failure" ]; then
if [ -n "$FAILED_STEPS" ]; then
FAILED_STEPS="${FAILED_STEPS} and Clang format"
else
FAILED_STEPS="Clang format"
fi
fi
JOB_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
printf -v COMMENT_BODY '%s\n' \
"@copilot The ${FAILED_STEPS} check failed. Please fix the lint errors in this PR." \
"" \
"Lint job log: ${JOB_URL}" \
"" \
"Instructions:" \
"1. Review the lint failure log via the link above" \
"2. Fix all lint errors in the relevant source files" \
"3. Ensure both Python lint and Clang format checks pass"
gh pr comment ${{ github.event.pull_request.number }} --body "${COMMENT_BODY}"
- name: Fail if lint failed
if: ${{ steps.lint-python.outcome == 'failure' || steps.lint-clang.outcome == 'failure' }}
run: |
echo "::error::Lint check failed"
exit 1
- name: Check reproducer exists for AI-generated PR
if: ${{ contains(github.event.pull_request.labels.*.name, 'ai_generated') }}
id: reproducer-check
env:
GH_TOKEN: ${{ secrets.MERGE_TOKEN }}
run: |
cd ./torch-xpu-ops
FAIL_MSG=""
HAS_REPRO_FILES=false
HAS_PR_CMD=false

# Check if reproducer files exist
if ! ls test/repro/test_*.py 1>/dev/null 2>&1; then
FAIL_MSG="No reproducer test found. AI-generated PRs must include at least one reproducer in \`test/repro/test_*.py\`."
else
# Validate pytest format: must contain test functions or test classes
VALID=false
if ls test/repro/test_*.py 1>/dev/null 2>&1; then
# Validate pytest format
for f in test/repro/test_*.py; do
if grep -qE '^[[:space:]]*(def test_|class Test)' "$f"; then
VALID=true
HAS_REPRO_FILES=true
else
FAIL_MSG="Reproducer \`$f\` is not in pytest format. It must contain pytest-style test functions (\`def test_...\`) or test classes (\`class Test...\`)."
break
fi
done
if [ "$VALID" = true ] && [ -z "$FAIL_MSG" ]; then
echo "All reproducer tests are valid pytest format:"
if [ "$HAS_REPRO_FILES" = true ] && [ -z "$FAIL_MSG" ]; then
echo "Found valid reproducer files:"
ls test/repro/test_*.py
fi
fi
Comment on lines 96 to 111

# Check if PR description contains a reproducer-cmd block
PR_BODY=$(gh pr view ${{ github.event.pull_request.number }} --json body -q '.body' 2>/dev/null || true)
PR_CMD=$(echo "$PR_BODY" | sed -n '/^```reproducer-cmd/,/^```/{/^```/d;p}')
if [ -n "$PR_CMD" ]; then
HAS_PR_CMD=true
echo "Found reproducer command in PR description:"
echo "$PR_CMD"
fi

# Export for later steps
echo "has_repro_files=$HAS_REPRO_FILES" >> "$GITHUB_OUTPUT"
echo "has_pr_cmd=$HAS_PR_CMD" >> "$GITHUB_OUTPUT"
# Use delimiter for multiline output
echo "pr_cmd<<ENDOFCMD" >> "$GITHUB_OUTPUT"
echo "$PR_CMD" >> "$GITHUB_OUTPUT"
echo "ENDOFCMD" >> "$GITHUB_OUTPUT"
Comment on lines +125 to +128

# Fail only if neither source is available
if [ "$HAS_REPRO_FILES" = false ] && [ "$HAS_PR_CMD" = false ] && [ -z "$FAIL_MSG" ]; then
FAIL_MSG="No reproducer found. AI-generated PRs must include either reproducer files in \`test/repro/test_*.py\` or a \`reproducer-cmd\` block in the PR description."
fi

if [ -n "$FAIL_MSG" ]; then
PR_AUTHOR="${{ github.event.pull_request.user.login }}"
printf -v COMMENT_BODY '%s\n' \
"@${PR_AUTHOR} ${FAIL_MSG}" \
"" \
"Please ensure your reproducer follows pytest conventions:" \
"- File name: \`test/repro/test_<description>.py\`" \
"- Contains \`def test_...()\` functions or \`class Test...\` classes" \
"- Runnable via \`pytest test/repro/\`"
"Please provide a reproducer using one of these methods:" \
"- Add reproducer files: \`test/repro/test_<description>.py\` (pytest format)" \
"- Add a reproducer command in PR description using a fenced code block with \`reproducer-cmd\` as the language identifier"
gh pr comment ${{ github.event.pull_request.number }} --body "${COMMENT_BODY}"
echo "::error::${FAIL_MSG}"
exit 1
Expand Down Expand Up @@ -185,7 +248,7 @@ jobs:
test_type: ${{ contains(github.event.pull_request.labels.*.name, 'disable_build') && 'wheel-cicd' || 'build-cicd' }}

reproducer-test:
needs: [linux-build]
needs: [linux-build, preci-lint-check]
if: ${{ contains(github.event.pull_request.labels.*.name, 'ai_generated') }}
secrets: inherit
permissions:
Expand All @@ -197,6 +260,7 @@ jobs:
runner: pvc_rolling
pytorch: ${{ contains(github.event.pull_request.labels.*.name, 'disable_build') && 'nightly_wheel' || github.base_ref }}
torch_xpu_ops: ${{ contains(github.event.pull_request.labels.*.name, 'disable_build') && 'pinned' || 'main' }}
pr_body_cmd: ${{ needs.preci-lint-check.outputs.pr_body_cmd }}

windows:
name: windows
Expand Down
Loading