Builds not reporting source-build prebuilts as errors

[mthalman]

The Roslyn CI pipeline includes a Source-Build (Managed) build leg which builds the repo in the context of source-build. When source-build prebuilts are detected, they are reported in that build leg. However, they are being reported as a warning and not an error. This seems to be unique to the roslyn repo since other repos are correctly reporting prebuilts as errors. It's not clear why this is happening. This is a problem since it lets changes get merged which cause prebuilts to be introduced. This will then fail later on in dependency flow (e.g. dotnet/installer#17319) which does detect the prebuilts and reports them as an error.

cc @MichaelSimons @mmitche

[genlu]

@333fred I think you are the expert here? :)

[mthalman]

Here are some binlogs to compare between roslyn, which doesn't reported prebuilts as errors, and sdk, which does: binlogs.zip. Search for the ValidateUsageAgainstBaseline task in the binlog viewer. I don't see any difference in the parameters being passed. I see that TreatWarningsAsErrors is set to true.

The weird thing is that FailOnPrebuiltBaselineError is always set to false. So I don't know how this actually works in all the other repos.

[mmitche]

What appears to be happening is that TreatWarningsAsErrors is set to false for the SB leg. This is the default in the roslyn repo https://github.com/dotnet/roslyn/blob/main/eng/build.sh#L76 unless --warnaserror is explicitly passed.

The other issue, as @mthalman has found, is that the baseline comparison task is not automatically failing on a failed baseline comparison. We were essentially relying on the typical repo setting of treating warnings as errors to get an error out.

I think the right thing to do here is to change the arcade infra to default to error on a baseline comparison failure, rather than changing roslyn's behavior. Roslyn only enables warnaserror for its code check leg, and primarily we just want to be as close to the default repo behavior for the source-build leg.

[mmitche]

dotnet/arcade#14022

[333fred]

@mthalman @mmitche is there anything for Roslyn to do here? It seems like the linked issue is closed?

[mmitche]

Nope, I think this can be closed.

[333fred]

Thanks!