SDK doesnt publish exes correctly when used as a project reference

[jaredpar]

Often a group of exes need to be built and deployed together. This is common in applications like Roslyn where we ship several exes as a single unit: csc, vbc and VBCSCompiler.

Rather than building each project separately and then copying the outputs together we create deployment projects. Essentially dummy exes projects that just reference all of the exes that we need via project reference elements

<ProjectReference Include="..\..\src\Compilers\CSharp\csc\csc.csproj" />
<ProjectReference Include="..\..\src\Compilers\VisualBasic\vbc\vbc.csproj" />
<ProjectReference Include="..\..\src\Compilers\Server\VBCSCompiler\VBCSCompiler.csproj" />

This approach works fine in desktop MSBuild as it will correctly deploy all of the runtime assets. The same approach does not work on SDK projects though as the publish step fails to include critical files like deps.json and runtimeconfig.json. That completely breaks this approach.

If this isn't meant to be supported by SDK that would be unfortunate but understandable. I would expect an error though positively asserting that this case is not supported. Today everything builds and publishes without error but the output is completely unusable.

Repo:

Clone https://github.com/jaredpar/roslyn and checkout the branch repro/group-exe. Then run the following commands:

> dotnet restore build/ToolsetPackages/BaseToolset.csproj
> dotnet restore build/ToolsetPackages/CoreToolset.csproj
> dotnet restore build/Toolset/CoreToolset.csproj
> dotnet publish -o ${HOME}/temp/test --framework netcoreapp2.0 build/Toolset/CoreToolset.csproj

The directory ${HOME}/temp/test should contain the deps.json / runtimeconfig.json for csc, vbc and VBCSCompiler but it does not. That means the output of publish is unusable.

CC @khyperia

[livarcocc]

So, if I understand this correct, you have a project that references three other self-contained netcoreapp projects as project references and you expect that publish this aggregator project will also publish all its references as an app?

If that's the case, you are right, we do not support that. P2P references are treated as libraries and not applications. In order to accomplish this I think you could change your aggregator project to have a list of projects that it needs to publish and for you to invoke publish in each of them separately.

[jaredpar]

. In order to accomplish this I think you could change your aggregator project to have a list of projects that it needs to publish and for you to invoke publish in each of them separately.

That doesn't quite get the behavior we want though. Part of the advantage to using the <ProjectReference> route is that MSBuild hepls ensure we have consistent dependencies across the projects. If any dependencies are wrong the standard warnings around conflicts will come into play.

This is important because the versions need to be consistent in order to ensure the exes will run correctly when all their dependencies are dumped into a single directory.

If that's the case, you are right, we do not support that. P2P references are treated as libraries and not applications

In that case why do the app.config files get deployed at all? Thats only relevant when running the application. I would assume, possibly wrong, that app.config, runtimeconfig.json and deps.json would all be deployed or none.

[dsplaisted]

I would also like some way to publish multiple apps to the same folder (and correctly handle the dependencies).

However, it's not as simple as copying the additional files when an app project is referenced. The deps.json and runtimeconfig.json will have been generated within the context of the app's dependencies. If the "publish" project ends up resolving different versions of those dependencies, then the versions deployed won't match what the app is expecting, which I think could lead to failures.

[jaredpar]

If the "publish" project ends up resolving different versions of those dependencies, then the versions deployed won't match what the app is expecting, which I think could lead to failures.

Exactly. We want these failures. It tells us that we screwed up our dependencies 😄

[dsplaisted]

Here's a potential workaround that includes the deps.json and runtimeconfig.json in the files that referenced projects will copy.

  <!-- Include MSBuild.deps.json and MSBuild.runtimeconfig.json in ContentWithTargetPath so they will be copied to the output folder of projects
       that reference this one. -->
  <Target Name="AddRuntimeDependenciesToContent" Condition=" '$(TargetFrameworkIdentifier)' == '.NETCoreApp'" BeforeTargets="GetCopyToOutputDirectoryItems">
    <ItemGroup>
      <ContentWithTargetPath Include="$(ProjectDepsFilePath)" CopyToOutputDirectory="PreserveNewest" TargetPath="$(ProjectDepsFileName)" />

      <ContentWithTargetPath Include="$(ProjectRuntimeConfigFilePath)" CopyToOutputDirectory="PreserveNewest" TargetPath="$(ProjectRuntimeConfigFileName)" />
    </ItemGroup>
  </Target>

[jaredpar]

@dsplaisted

How supported is that work around? This is a feature we sorely miss since switching to the new SDK / Exes (lots of duplication + missed errors). If this is fairly supported we'd consider switching to use it.

[dsplaisted]

@jaredpar I wouldn't call it supported. The problem is going to be if the different Exes have different versions of shared dependencies. If that's not the case, then I think everything should work well.

[jaredpar]

If it's not supported I'm rather hesitant to move to it. Depending on unsupported features is equally painful to both our teams when they eventually get broken

[ericstj]

@dsplaisted what part of this are you feeling like is unsupported? Ensuring the workaround stays working? Seems like we can just add a test case to make sure that is true. If the SDK does add support for this then you can make sure you won't break the workaround. I don't think there is anything fundamentally wrong with flowing the deps/runtimeconfig along with other things. I can see how this might be important for other project types like NuProj/WixProj/Azure-packages/etc.

It sounds like @jaredpar's not asking us to vouch for the deployment strategy of combining apps into a single directory as "supported" since he's expecting it to fail sometimes if they do the wrong thing and mentioned they'll need to test for that.

[ViktorHofer]

@dsplaisted can we please adjust the milestone? We have a tracking issue against this in dotnet/runtime: https://github.com/dotnet/runtime/blob/ec9d33daf4f1922b2f36f548752ac4539f370c91/eng/testing/runtimeConfiguration.targets#L19

[MartyIX]

This helped me https://github.com/dotnet/runtime/blob/ec9d33daf4f1922b2f36f548752ac4539f370c91/eng/testing/runtimeConfiguration.targets#L19 for my .NET Core 3.1 application:

  <ItemGroup Condition="'$(GenerateRuntimeConfigurationFiles)' == 'true'">
    <None Include="$(TestRuntimeConfigurationFile)"
          Condition="Exists('$(TestRuntimeConfigurationFile)')"
          Link="$(TargetName).exe.config"
          CopyToOutputDirectory="PreserveNewest"
          Visible="false" />
    <!--
      Include deps.json and runtimeconfig.json in ContentWithTargetPath so they will
      be copied to the output folder of projects that reference this one.
      Tracking issue: https://github.com/dotnet/sdk/issues/1675
    -->
    <ContentWithTargetPath Include="$(ProjectDepsFilePath)"
                           Condition="'$(TargetsNetCoreApp)' == 'true' and '$(GenerateDependencyFile)' == 'true'"
                           CopyToOutputDirectory="PreserveNewest"
                           TargetPath="$(ProjectDepsFileName)" />
    <ContentWithTargetPath Include="$(ProjectRuntimeConfigFilePath)"
                           Condition="'$(TargetsNetCoreApp)' == 'true'"
                           CopyToOutputDirectory="PreserveNewest"
                           TargetPath="$(ProjectRuntimeConfigFileName)" />
  </ItemGroup>

sbomer/linker@fda18bc

edit: useful link for my future reference: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.targets

[ViktorHofer]

That's me implementing (mostly copying) what @dsplaisted suggested above 😄

[MartyIX]

@ViktorHofer I've just spent an hour trying to fix the original issue, so I posted your version because it's not obvious that one needs to use the latest master version in git to get to a working solution.

Thank you for your work!

[ViktorHofer]

Sure, np.

because it's not obvious that one needs to use the latest master version in git to get to a working solution.

what do you mean by that?

[MartyIX]

@dsplaisted can we please adjust the milestone? We have a tracking issue against this in dotnet/runtime: https://github.com/dotnet/runtime/blob/ec9d33daf4f1922b2f36f548752ac4539f370c91/eng/testing/runtimeConfiguration.targets#L19

@ViktorHofer This refers to some commit from 2019. This one is newer: https://github.com/dotnet/runtime/blob/ec9d33daf4f1922b2f36f548752ac4539f370c91/eng/testing/runtimeConfiguration.targets#L19 and that code works for me.

[ViktorHofer]

Got it, thanks for clarifying.

[kirsan31]

The same approach does not work on SDK projects though as the publish step fails to include critical files like deps.json and runtimeconfig.json.

Just to note, that the problem appear not only on publish step, but on simple build too, see #11207.

[urosjovanovic]

Yes, it is also not copied on build action.

I have tried the workaround proposed above but had no luck. It throws a build error that it cannot copy *.deps.json and *.runtimeconfig.json because they do not exist.

I've added the target at the end of referenced .csproj (.net console app).

[TonyValenti]

We're running into challenges because of this too. It would be really nice if this could get resolved.

[MartyIX]

How can I use #1675 (comment) in .NET 5?

I believe that '$(TargetFrameworkIdentifier)' == '.NETCoreApp' needs to be modified but I'm not sure what the new identifier is.

[ltrzesniewski]

@MartyIX $(TargetFrameworkIdentifier) is still .NETCoreApp in .NET 5.

[ChrisUnityArto]

I'm glad to see this is due to be worked on.

[ViktorHofer]

Fixed by #14488

[kirsan31]

@ViktorHofer In What VS version it will be included?

[dsplaisted]

It should be in .NET SDK 5.0.200, which will ship with VS 16.9

[RandomEngy]

VS 16.9.0 is out and it works! Thanks for the fix.

[MartyIX]

Fixed by #14488

This works for me in .NET 5. However, I wonder whether this issue is back in .NET 6 RC 2 as I see the following issue: https://github.com/dotnet/efcore/issues/26350?

Would anybody know?

[jeremy-visionaid]

Looks like this is still a problem for Managed C++ dependencies, but the workarounds suggested here for adding it to ContentWithTargetPath manually work for those too. I'll open a new issue if I get the chance.