Copy task should retry on FileNotFoundException (MSB3030) during parallel builds

[wtgodbe]

Problem

The Copy task checks File.Exists on source files before its retry loop. When the source doesn't exist, it immediately logs MSB3030 and returns false — no retry. The Retries/CopyRetryCount parameter only applies to IOException during the actual File.Copy call.

// src/Tasks/Copy.cs — outside the retry loop
{
    Log.LogErrorWithCodeFromResources("Copy.SourceFileNotFound", ...);
    return false;  // MSB3030 — no retry
}

This causes flaky build failures in large parallel builds (/m) where shared dependency projects are rebuilt by multiple consumer projects. When a dependency's CopyFilesToOutputDirectory overwrites a DLL/PDB, consumer projects running _CopyFilesMarkedCopyLocal can see the source file as transiently missing.

Impact

In dotnet/aspnetcore, this is tracked as Known Build Error #62807 with ~25 hits per month. The only current mitigation is full build retry via Build Analysis, which wastes 30+ minutes per occurrence.

Reproduction

Any large parallel MSBuild solution where:

1. Project A is a shared dependency (referenced by 100+ projects)
2. Build runs with /m (unlimited parallelism)
3. Project A's CopyFilesToOutputDirectory runs on one MSBuild node
4. A consumer's _CopyFilesMarkedCopyLocal runs on another node simultaneously
5. The source file in bin/ is briefly unavailable during the overwrite

Proposed Fix

Extend the retry loop in the Copy task to cover FileNotFoundException (or specifically the File.Exists pre-check), similar to how IOException is already retried during the copy operation. This would allow the Retries and RetryDelayMilliseconds parameters to protect against transiently-missing source files.

Workaround

We've added a BeforeTargets="_CopyFilesMarkedCopyLocal" gate in aspnetcore that waits for missing source files before the Copy task runs. This reduces the race window but doesn't eliminate the TOCTOU gap entirely.

[baronfel]

shared dependency projects are rebuilt by multiple consumer projects

why is this happening? This is the core race condition in the build, and if we paper over it here we won't fix the root cause.

[wtgodbe]

why is this happening? This is the core race condition in the build, and if we paper over it here we won't fix the root cause.

I think it would be fixed by switching to graph mode. Looking at the binlog, it appears that (in the bad case) the same project is getting built twice simultaneously with the same global properties:

it's MSBuild's classic scheduler deciding to build the same (project, globalprops) tuple twice in parallel. This happens when two consumer projects issue BuildProjectReferences simultaneously and the scheduler doesn't deduplicate fast enough.

[baronfel]

Thanks, that's really useful - is that issue logged anywhere?

[wtgodbe]

#9462 is the closest I could find

[wtgodbe]

I'm happy to share more details from my local investigation/binlog as well, if that'd be helpful

[baronfel]

That's not quite the same, that's about parallel dotnet msbuild top-level invocations, which today know nothing about each other. Your root issue is about intra-build scheduling. I'll try to see if we have something relevant already 👍

[baronfel]

@wtgodbe do you by any chance have that binlog for future reference?

[wtgodbe]

do you by any chance have that binlog for future reference?

Sent it to you offline

[OvesN]

@baronfel
This is related to #12927.
We investigated it, and we’re quite sure that MSBuild does not delete this file (at least we don’t see it in the binlog). However, another process seems to delete and rewrite the file, and sometimes the Copy task hits this narrow window when the file is missing.
The next step I’m planning is to add tracing to the pipeline so we can see which process rewrites the file.