Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] Improve AOT builds #48718

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion eng/testing/tests.mobile.targets
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@
<CultureName>$([System.IO.Directory]::GetParent('%(Identity)').Name)</CultureName>
</WasmSatelliteAssemblies>

<WasmAssembliesToBundle Include="$(PublishDir)\*.dll"/>
<WasmAssembly Include="$(PublishDir)\*.dll"/>

<WasmFilesToIncludeInFileSystem Include="@(ContentWithTargetPath)" />
<WasmFilesToIncludeInFileSystem Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.BuildReference)' == 'true' and !$([System.String]::new('%(ReferenceCopyLocalPaths.Identity)').EndsWith('.resources.dll'))" />
Expand Down
2 changes: 1 addition & 1 deletion src/mono/sample/mbr/browser/WasmDelta.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

<Target Name="PrepareDeltasForWasmApp" DependsOnTargets="Build;CompileDiff;ComputeDeltaFileOutputNames">
<ItemGroup>
<WasmAssembliesToBundle Include="$(TargetDir)publish\*.dll" />
<WasmAssembly Include="$(TargetDir)publish\*.dll" />
<WasmFilesToIncludeInFileSystem Include="@(_DeltaFileForPublish)">
<TargetPath>\%(_DeltaFileForPublish.Filename)%(_DeltaFileForPublish.Extension)</TargetPath>
</WasmFilesToIncludeInFileSystem>
Expand Down
2 changes: 1 addition & 1 deletion src/mono/sample/wasm/Directory.Build.targets
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<Project>
<Target Name="PrepareForWasmBuild" BeforeTargets="WasmBuildApp">
<ItemGroup>
<WasmAssembliesToBundle Include="$(TargetDir)publish\*.dll" />
<WasmAssembly Include="$(TargetDir)publish\*.dll" />
</ItemGroup>
</Target>
<Import Project="$(MonoProjectRoot)\wasm\build\WasmApp.InTree.targets" />
Expand Down
8 changes: 4 additions & 4 deletions src/mono/wasm/build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,24 @@ various properties before the target `WasmBuildApp` gets executed.
to do that.

- By default, the `WasmLoadAssembliesAndReferences` task is not run, and
the specified `@(WasmAssembliesToBundle)` are directly passed to
the specified `@(WasmAssembly)` are directly passed to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WasmAOTAssembly? WasmAssembly feels very non-specific.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are not AOT specific

`WasmAppBuilder`.
- If the project needs assembly dependencies to be resolved, then
set `$(WasmResolveAssembliesBeforeBuild) == true`.
- Should you need to run the AOT toolset, ensure `$(RunAOTCompilation) == true`
and set `$(WasmAOTDir)` to the directory that you want to AOT. Make sure that both
`@(WasmAssembliesToBundle)` and `$(WasmAOTDir)` are absolute paths.
`@(WasmAssembly)` and `$(WasmAOTDir)` are absolute paths.

- Assemblies to be bundled with the app are set via
`@(WasmAssembliesToBundle)` (which optionally will have dependencies
`@(WasmAssembly)` (which optionally will have dependencies
resolved)

The various task inputs correspond to properties as:

```
AssemblySearchPaths : @(WasmAssemblySearchPaths)

Assemblies : @(WasmAssembliesToBundle)
Assemblies : @(WasmAssembly)

AppDir : $(WasmAppDir)
MainAssembly : $(WasmMainAssemblyPath)
Expand Down
80 changes: 44 additions & 36 deletions src/mono/wasm/build/WasmApp.targets
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<!--
Required public items/properties:
- $(WasmMainJSPath)
- @(WasmAssembliesToBundle) - list of assemblies to package as the wasm app
- @(WasmAssembly) - list of assemblies to package as the wasm app
- $(EMSDK_PATH) - points to the emscripten sdk location.

Public properties (optional):
Expand Down Expand Up @@ -67,7 +67,7 @@
<Target Name="WasmBuildApp" AfterTargets="$(WasmBuildAppAfterThisTarget)" />

<Target Name="_WasmAotCompileApp" Condition="'$(RunAOTCompilation)' == 'true'">
<Error Condition="'@(_WasmAssemblies)' == ''" Text="Item _WasmAssemblies is empty" />
<Error Condition="'@(_WasmAssemblyInternal)' == ''" Text="Item _WasmAssemblyInternal is empty" />
<Error Condition="'$(EMSDK_PATH)' == ''" Text="%24(EMSDK_PATH) should be set to emscripten sdk" />

<ItemGroup>
Expand All @@ -77,12 +77,12 @@
<MonoAOTCompilerDefaultAotArguments Include="deterministic" />
</ItemGroup>
<ItemGroup>
<_AotInputAssemblies Include="@(_WasmAssemblies->Distinct())">
<_AotInputAssemblies Include="@(_WasmAssemblyInternal->Distinct())">
<AotArguments>@(MonoAOTCompilerDefaultAotArguments, ';')</AotArguments>
<ProcessArguments>@(MonoAOTCompilerDefaultProcessArguments, ';')</ProcessArguments>
</_AotInputAssemblies>

<_WasmAssemblies Remove="@(_WasmAssemblies)" />
<_WasmAssemblyInternal Remove="@(_WasmAssemblyInternal)" />

<_WasmAOTSearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)\native" />
<_WasmAOTSearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)\lib\net6.0" />
Expand All @@ -107,7 +107,7 @@
UseLLVM="true"
DisableParallelAot="true"
LLVMPath="$(EMSDK_PATH)\upstream\bin">
<Output TaskParameter="CompiledAssemblies" ItemName="_WasmAssemblies" />
<Output TaskParameter="CompiledAssemblies" ItemName="_WasmAssemblyInternal" />
<Output TaskParameter="FileWrites" ItemName="FileWrites" />
</MonoAOTCompiler>
</Target>
Expand All @@ -119,31 +119,31 @@
</PropertyGroup>

<ItemGroup>
<_WasmStrippedAssemblies Include="@(_WasmAssemblies->'$(_WasmStrippedAssembliesPath)\%(FileName)%(Extension)')" OriginalPath="%(_WasmAssemblies.Identity)" />
<_WasmStrippedAssemblies Include="@(_WasmAssemblyInternal->'$(_WasmStrippedAssembliesPath)\%(FileName)%(Extension)')" OriginalPath="%(_WasmAssemblyInternal.Identity)" />
</ItemGroup>

<!-- Run mono-cil-strip on the assemblies -->
<!-- TODO: replace this with a linker task -->
<MakeDir Directories="$(_WasmStrippedAssembliesPath)" />
<Exec Command="mono-cil-strip -q %(_WasmStrippedAssemblies.OriginalPath) %(_WasmStrippedAssemblies.Identity)" />
<Exec Command='mono-cil-strip -q "%(_WasmStrippedAssemblies.OriginalPath)" "%(_WasmStrippedAssemblies.Identity)"' />

<ItemGroup>
<_WasmAssemblies Remove="@(_WasmAssemblies)" />
<_WasmAssemblies Include="@(_WasmStrippedAssemblies)" />
<_WasmAssemblyInternal Remove="@(_WasmAssemblyInternal)" />
<_WasmAssemblyInternal Include="@(_WasmStrippedAssemblies)" />
</ItemGroup>
</Target>

<Target Name="_WasmResolveReferences" Condition="'$(WasmResolveAssembliesBeforeBuild)' == 'true'">
<WasmLoadAssembliesAndReferences
Assemblies="@(_WasmAssemblies)"
Assemblies="@(_WasmAssemblyInternal)"
AssemblySearchPaths="@(WasmAssemblySearchPaths)"
SkipMissingAssemblies="$(WasmSkipMissingAssemblies)">
<Output TaskParameter="ReferencedAssemblies" ItemName="_TmpWasmAssemblies" />
<Output TaskParameter="ReferencedAssemblies" ItemName="_TmpWasmAssembly" />
</WasmLoadAssembliesAndReferences>

<ItemGroup>
<_WasmAssemblies Remove="@(_WasmAssemblies)" />
<_WasmAssemblies Include="@(_TmpWasmAssemblies)" />
<_WasmAssemblyInternal Remove="@(_WasmAssemblyInternal)" />
<_WasmAssemblyInternal Include="@(_TmpWasmAssembly)" />
</ItemGroup>
</Target>

Expand All @@ -168,7 +168,7 @@
<MicrosoftNetCoreAppRuntimePackRidDir Condition="!HasTrailingSlash('$(MicrosoftNetCoreAppRuntimePackRidDir)')">$(MicrosoftNetCoreAppRuntimePackRidDir)\</MicrosoftNetCoreAppRuntimePackRidDir>
</PropertyGroup>
<ItemGroup>
<_WasmAssemblies Include="@(WasmAssembliesToBundle)" />
<_WasmAssemblyInternal Include="@(WasmAssembly)" />
</ItemGroup>
</Target>

Expand Down Expand Up @@ -196,7 +196,7 @@
<WasmAppBuilder
AppDir="$(WasmAppDir)"
MainJS="$(WasmMainJSPath)"
Assemblies="@(_WasmAssemblies)"
Assemblies="@(_WasmAssemblyInternal)"
InvariantGlobalization="$(InvariantGlobalization)"
SatelliteAssemblies="@(WasmSatelliteAssemblies)"
FilesToIncludeInFileSystem="@(WasmFilesToIncludeInFileSystem)"
Expand Down Expand Up @@ -237,17 +237,19 @@
<!--This pinvoke-table.h will be used instead of the one in the runtime pack because of -I$(_WasmIntermediateOutputPath) -->
<PInvokeTableGenerator
Modules="@(_WasmPInvokeModules)"
Assemblies="@(_WasmAssemblies)"
Assemblies="@(_WasmAssemblyInternal)"
OutputPath="$(_WasmIntermediateOutputPath)pinvoke-table.h" />

<!-- ICall table generation -->
<Exec Condition="'$(WasmLinkIcalls)' == 'true'" Command="$(MonoAotCrossCompilerPath) --print-icall-table > $(_WasmIntermediateOutputPath)/runtime-icall-table.h" />
<Exec Condition="'$(WasmLinkIcalls)' == 'true'" Command='"$(MonoAotCrossCompilerPath)" --print-icall-table > "$(_WasmIntermediateOutputPath)/runtime-icall-table.h"' />
<IcallTableGenerator Condition="'$(WasmLinkIcalls)' == 'true'"
RuntimeIcallTableFile="$(_WasmIntermediateOutputPath)/runtime-icall-table.h"
Assemblies="@(_WasmAssemblies)"
Assemblies="@(_WasmAssemblyInternal)"
OutputPath="$(_WasmIntermediateOutputPath)/icall-table.h" />
<PropertyGroup>
<EmccFlags Condition="'$(WasmLinkIcalls)' == 'true'">$(EmccFlags) -DLINK_ICALLS=1</EmccFlags>
<_WasmIncludeDir>$(MicrosoftNetCoreAppRuntimePackRidDir)native/include</_WasmIncludeDir>
<_WasmSrcDir>$(MicrosoftNetCoreAppRuntimePackRidDir)native/src</_WasmSrcDir>
</PropertyGroup>

<ItemGroup>
Expand All @@ -260,39 +262,45 @@
<_WasmRuntimePackNativeLibs Include="libmono-profiler-aot.a"/>
<_WasmRuntimePackNativeLibs Include="libicuuc.a" Condition="'$(InvariantGlobalization)' != 'true'" />
<_WasmRuntimePackNativeLibs Include="libicui18n.a" Condition="'$(InvariantGlobalization)' != 'true'" />

<_WasmObjectsToBuild Include="$(_WasmIntermediateOutputPath)driver.o"/>
<_WasmObjectsToBuild Include="$(_WasmIntermediateOutputPath)pinvoke.o"/>
<_WasmObjectsToBuild Include="$(_WasmIntermediateOutputPath)corebindings.o"/>

<_WasmObjects Include="@(_WasmRuntimePackNativeLibs->'$(MicrosoftNetCoreAppRuntimePackRidDir)\native\%(FileName)%(Extension)')" />
<_WasmObjects Include="@(_WasmObjectsToBuild)" />

<_WasmObjects Include="$(_WasmIntermediateOutputPath)driver.o"/>
<_WasmObjects Include="$(_WasmIntermediateOutputPath)pinvoke.o"/>
<_WasmObjects Include="$(_WasmIntermediateOutputPath)corebindings.o"/>
<_DotnetJSSrcFile Include="$(_WasmSrcDir)/library_mono.js" />
<_DotnetJSSrcFile Include="$(_WasmSrcDir)/binding_support.js" />
<_DotnetJSSrcFile Include="$(_WasmSrcDir)/dotnet_support.js" />
<_DotnetJSSrcFile Include="$(_WasmSrcDir)/pal_random.js" />

<_BitcodeFile Include="%(_WasmAssemblies.LlvmBitcodeFile)" />
<_BitcodeFile Include="%(_WasmAssemblyInternal.LlvmBitcodeFile)" />
</ItemGroup>

<Error Condition="'$(RunAOTCompilation)' == 'true' and @(_BitcodeFile->Count()) != @(_WasmAssemblies->Count())"
Text="Bug: Number of assemblies doesn't match the number of generated bitcode files. BitcodeFiles: @(_BitcodeFile->Count()) vs Assemblies: @(_WasmAssemblies->Count())" />
<Error Condition="'$(RunAOTCompilation)' == 'true' and @(_BitcodeFile->Count()) != @(_WasmAssemblyInternal->Count())"
Text="Bug: Number of assemblies doesn't match the number of generated bitcode files. BitcodeFiles: @(_BitcodeFile->Count()) vs Assemblies: @(_WasmAssemblyInternal->Count())" />

<PropertyGroup>
<_WasmIncludeDir>$(MicrosoftNetCoreAppRuntimePackRidDir)native/include</_WasmIncludeDir>
<_WasmSrcDir>$(MicrosoftNetCoreAppRuntimePackRidDir)native/src</_WasmSrcDir>
<EmccCFlags>$(EmccFlags) -DCORE_BINDINGS -DGEN_PINVOKE=1 -I$(_WasmIntermediateOutputPath) -I$(_WasmIncludeDir)/mono-2.0 -I$(_WasmIncludeDir)/wasm</EmccCFlags>
<EmccCFlags>$(EmccFlags) -g -DCORE_BINDINGS -DGEN_PINVOKE=1 "-I$(_WasmIncludeDir)/mono-2.0" "-I$(_WasmIncludeDir)/wasm"</EmccCFlags>

<!-- on windows, make sure we don't get "-Ic:\foo\bar\" -->
<EmccCFlags Condition="$(_WasmIntermediateOutputPath.EndsWith('\')) ">$(EmccCFlags) "-I$(_WasmIntermediateOutputPath)""</EmccCFlags>
<EmccCFlags Condition="!$(_WasmIntermediateOutputPath.EndsWith('\'))">$(EmccCFlags) "-I$(_WasmIntermediateOutputPath)"</EmccCFlags>

<EmccLDFlags>$(EmccFlags) -s TOTAL_MEMORY=536870912</EmccLDFlags>
<_WasmOptCommand>$([MSBuild]::NormalizePath('$(EMSDK_PATH)', 'upstream', 'bin', 'wasm-opt$(_ExeExt)'))</_WasmOptCommand>
</PropertyGroup>

<RunWithEmSdkEnv Command="emcc $(EmccCFlags) $(_WasmSrcDir)/driver.c -c -o $(_WasmIntermediateOutputPath)driver.o" EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command="emcc $(EmccCFlags) $(_WasmSrcDir)/corebindings.c -c -o $(_WasmIntermediateOutputPath)corebindings.o" EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command="emcc $(EmccCFlags) $(_WasmSrcDir)/pinvoke.c -c -o $(_WasmIntermediateOutputPath)pinvoke.o" EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command="emcc $(EmccLDFlags) --js-library $(_WasmSrcDir)/library_mono.js --js-library $(_WasmSrcDir)/binding_support.js --js-library $(_WasmSrcDir)/dotnet_support.js --js-library $(_WasmSrcDir)/pal_random.js @(_WasmAssemblies->'%(LlvmBitcodeFile)', ' ') @(_WasmObjects, ' ') -o $(_WasmIntermediateOutputPath)dotnet.js" EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command="$(_WasmOptCommand) --strip-dwarf $(_WasmIntermediateOutputPath)dotnet.wasm -o $(_WasmIntermediateOutputPath)dotnet.wasm" Condition="'$(WasmNativeStrip)' == 'true'" IgnoreStandardErrorWarningFormat="true" EmSdkPath="$(EMSDK_PATH)" />
<Message Text="Compiling native assets with emcc. This may take a while ..." Importance="High" />
<RunWithEmSdkEnv Command='emcc $(EmccCFlags) "$(_WasmSrcDir)/%(_WasmObjectsToBuild.Filename).c" -c -o "%(_WasmObjectsToBuild.Identity)"' EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command="emcc $(EmccLDFlags) @(_DotnetJSSrcFile->'--js-library &quot;%(Identity)&quot;', ' ') @(_WasmAssemblyInternal->'&quot;%(LlvmBitcodeFile)&quot;', ' ') @(_WasmObjects->'&quot;%(Identity)&quot;', ' ') -o &quot;$(_WasmIntermediateOutputPath)dotnet.js&quot;" EmSdkPath="$(EMSDK_PATH)" />
<RunWithEmSdkEnv Command='"$(_WasmOptCommand)" --strip-dwarf "$(_WasmIntermediateOutputPath)dotnet.wasm" -o "$(_WasmIntermediateOutputPath)dotnet.wasm"' Condition="'$(WasmNativeStrip)' == 'true'" IgnoreStandardErrorWarningFormat="true" EmSdkPath="$(EMSDK_PATH)" />

<ItemGroup>
<WasmNativeAsset Include="$(_WasmIntermediateOutputPath)\dotnet.wasm" />
<WasmNativeAsset Include="$(_WasmIntermediateOutputPath)\dotnet.js" />
</ItemGroup>

<!-- If a bundle wouldn't be generated, then copy the files to app dir -->
<Copy SourceFiles="@(WasmNativeAsset)" DestinationFolder="$(WasmAppDir)" Condition="'$(WasmGenerateAppBundle)' != 'true'" />
</Target>

<Target Name="_GenerateDriverGenC" Condition="'$(RunAOTCompilation)' != 'true' and '$(WasmProfilers)' != ''">
Expand Down Expand Up @@ -356,7 +364,7 @@ EMSCRIPTEN_KEEPALIVE void mono_wasm_load_profiler_aot (const char *desc) { mono_

<Target Name="_AfterWasmBuildApp">
<ItemGroup>
<WasmAssembliesFinal Include="@(_WasmAssemblies)" />
<WasmAssemblyFinal Include="@(_WasmAssemblyInternal)" />
</ItemGroup>
</Target>
</Project>
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</PropertyGroup>

<ItemGroup>
<WasmAssembliesToBundle Include="$(OutDir)\$(TargetFileName)" />
<WasmAssembly Include="$(OutDir)\$(TargetFileName)" />
<WasmAssemblySearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)native"/>
<WasmAssemblySearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)lib\$(NetCoreAppCurrent)"/>

Expand Down
4 changes: 2 additions & 2 deletions src/tasks/AotCompilerTask/MonoAOTCompiler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ private bool PrecompileLibrary(ITaskItem assemblyItem, string? monoPaths)
processArgs.AddRange(p.Split(";", StringSplitOptions.RemoveEmptyEntries));
}

Utils.LogInfo($"[AOT] {assembly}");
Log.LogMessage(MessageImportance.Low, $"[AOT] {assembly}");

processArgs.Add("--debug");

Expand Down Expand Up @@ -361,7 +361,7 @@ private bool PrecompileLibrary(ITaskItem assemblyItem, string? monoPaths)
try
{
// run the AOT compiler
Utils.RunProcess(CompilerBinaryPath, string.Join(" ", processArgs), envVariables, directory, silent: false, outputMessageImportance: MessageImportance.Low);
Utils.RunProcess(CompilerBinaryPath, string.Join(" ", processArgs), envVariables, directory, silent: false, outputMessageImportance: MessageImportance.Low, logMessageImportance: MessageImportance.Low);
}
catch (Exception ex)
{
Expand Down
5 changes: 3 additions & 2 deletions src/tasks/Common/Utils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@ public static string RunProcess(
string? workingDir = null,
bool ignoreErrors = false,
bool silent = true,
MessageImportance outputMessageImportance=MessageImportance.High)
MessageImportance outputMessageImportance=MessageImportance.High,
MessageImportance logMessageImportance=MessageImportance.High)
{
LogInfo($"Running: {path} {args}");
LogInfo($"Running: {path} {args}", logMessageImportance);
var outputBuilder = new StringBuilder();
var errorBuilder = new StringBuilder();
var processStartInfo = new ProcessStartInfo
Expand Down
4 changes: 2 additions & 2 deletions src/tests/Common/wasm-test-runner/WasmTestRunner.proj
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
</PropertyGroup>

<ItemGroup>
<WasmAssembliesToBundle Include="$(TestBinDir)\*.dll" />
<WasmAssembliesToBundle Include="$(CORE_ROOT)\System.Private.Runtime.InteropServices.JavaScript.dll" />
<WasmAssembly Include="$(TestBinDir)\*.dll" />
<WasmAssembly Include="$(CORE_ROOT)\System.Private.Runtime.InteropServices.JavaScript.dll" />
<WasmAssemblySearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)\native"/>
<WasmAssemblySearchPaths Include="$(MicrosoftNetCoreAppRuntimePackRidDir)"/>
<WasmAssemblySearchPaths Include="$(CORE_ROOT)/TargetingPack" />
Expand Down
2 changes: 1 addition & 1 deletion src/tests/FunctionalTests/wasm/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<Target Name="PrepareForWasmBuild">
<ItemGroup>
<WasmAssembliesToBundle Include="$(TargetDir)publish\*.dll" />
<WasmAssembly Include="$(TargetDir)publish\*.dll" />
</ItemGroup>
</Target>
</Project>