[Single-Exe] Run from bundle #26504

swaroop-sridhar · 2019-09-04T08:46:27Z

Changes

corebundle Host:

Remove code that deals with extraction to temporary dirs.
Process the bundle manifest and export a callback to look for files in the bundle.
The TPA doesn't contain bundled assemblies any more.

CoreCLR Initialization:

Setup BundleInfo structure, and pass it to appropriate initializers
Expect to find System.Private.Corelib in the bundle

Bundle Info:

Holds information about the bundle
Exports a wrapper around the bundle runner's probe function for path adjustments
Performs Unicode/UTF8 translations as necessary (because host uses UTF8 on Unix)

AppDomain:

Add methods obtain BundleInfo

Binder:

Look for assemblies in the Bundle before searching the TPA

PEImage:

FlatImageLayout and MappedImageLayout can take files embedded within the bundle.
Memory mapping routines can take files embedded within the bundle.

A few other fixes necessary to build/run.

Testing

Tested templates: console, webapi

Performance

Console Hello World

No noticable change in execution time.

Run from (normal) publish directory: 104ms
Run from single-file: 99ms

Startup of WebApiTemplate:

Measured using https://github.com/rynowak/link-a-thon on a local (real) machine.

Both cases had runs with similar timing; but the average startup was consistently slightly faster for single-exe runs.

Run from (normal) publish directory:
801.8271
782.9489
790.8015
798.7449
797.3141
783.0968
790.0275
786.7621
802.2383
778.9866
Average startup time (ms): 791.27478

Run from single-file:
742.1464
692.1949
795.1612
798.4845
809.4075
794.2644
790.6254
634.1305
686.2718
806.5665
Average startup time (ms): 754.92531

Steps to build single-exe `app`

The following steps will work for most apps, minor adjustments may be necessary.

dotnet publish -r linux-x64 -c release -o pub
cp pub/*.dll source/
cp coreclr/bin/Product/Linux.x64.Release/corebundle source/<app>
cp coreclr/bin/Product/Linux.x64.Release/System.Private.Corelib.dll source/
dotnet coreclr/bin/Product/Linux.x64.Release/bundle.dll --source source --host <app>
Run app

vitek-karas

Do you have a sample how to create a bundle like this which can be tested?

src/coreclr/hosts/unixcorebundle/CMakeLists.txt

vitek-karas · 2019-09-04T09:53:43Z

src/coreclr/hosts/unixcorebundle/corebundle.cpp

@@ -15,6 +15,13 @@
 #include <runner.h>
 #include <utils.h>

+static bundle::runner_t bundle_runner;
+
+bool probe_bundle(const pal::char_t* path, int64_t *size, int64_t *offset)


This should declare explicitly the calling convention - as it's passed to a different module.

src/md/CMakeLists.txt

vitek-karas · 2019-09-04T10:14:08Z

src/dlls/mscoree/unixinterface.cpp

@@ -168,6 +169,7 @@ int coreclr_initialize(
            int propertyCount,
            const char** propertyKeys,
            const char** propertyValues,
+            bool bundleProbe(const char*, int64_t*, int64_t*),


I understand that this is a prototype so this is probably the fastest way to do this, but I don't like it.
Among other things, this means that builds from this branch will not work with core-setup hosts - so tests are going to break...

Also - we should at least start to think about how to wire this without breaking existing interfaces. This applies both to the C-ABI here, as well as the COM-ABI in corhost.

For the COM-ABI we could probably wire this through IHostControl and add a new host interface for GetHostManager. Alternatively we could rewrite the unixinterface code here to not use the COM APIs through interfaces but instead get the underlying C++ class directly and set something that way. I don't see a reason why we should support the bundle via the COM APIs.

For the C-ABI we should either pass this through the runtime properties - for now - somehow (we can pass an address as a string or something similarly ugly). Eventually we should design something real - there's been some discussion of being able to provide hosting services to the runtime - maybe we should resurect that. One way could be to add coreclr_initialize2 (we already have coreclr_shutdown2) or similar "new" API, which we will make sure is extensible enough so that we can pass around more things in the future.

We decided to not to try-design the Host-Runtime interface as part of the prototype. However, I do want to have the discussion about how the interface will look in the final scheme.

I do want to clean up the APIs at least to the point of test-pass. I'll start with the change to not pass around BundleInfo as @jkotas suggested.

I'm fine using some kind of a prototype way - that's not the main concern here (although we should start the discussion on the real solution soon - it might take a while). My main concern is that this breaks the existing hosting.
With your recent changes the COM-ABI is "fixed", but the C-ABI is still broken - meaning CLR built from this branch will not work with hostpolicy. I think we should avoid doing that - as it makes lot of things unnecessarily complex - at the very least it will break tests which rely on real host (probably not that many in coreclr repo, but still).

vitek-karas · 2019-09-04T10:18:11Z

src/inc/corbundle.h

+#ifndef _COR_BUNDLE_H_
+#define _COR_BUNDLE_H_
+
+class BundleInfo


Depending on how we solve the runtime extensibility problem... but I don't think we should be going with C++ classes in public interfaces. For one this basically breaks the COM-ABI interface since that one is supposed to be usable by any language - but this requires usage of C++.

I don't have a problem with having this class a "helper" for the binder and as a way to wrap the parameters passed in during init to the runtime, but it should not be exposed externally.

Agreed, C++ classes are not suitable for cross binary api surface ever.

vitek-karas · 2019-09-04T10:25:58Z

src/binder/assemblybinder.cpp

+            // Bundled assemblies are treated similar to TPAs.
+            if (g_BinderVariables->fIsBundle)
+            {
+                SString candidates[] = { W(".dll"), W(".ni.dll") };


Do we really need the .ni.dll support - for .NET Core I don't think we use that suffix anymore...
I'm fine including it for consistency - but since we're making "larger" changes, maybe we can clean things up as well.

I believe the .ni.dll stuff is still needed for managed c++ R2R at this time.

vitek-karas · 2019-09-04T10:29:50Z

src/binder/assemblybinder.cpp

+            {
+                SString candidates[] = { W(".dll"), W(".ni.dll") };
+
+                // Loop through the binding paths looking for a matching assembly


Nit: This is not looping through paths, just candidate extensions - the comment is just confusing.

vitek-karas · 2019-09-04T10:32:24Z

src/vm/peimagelayout.cpp

@@ -599,12 +606,21 @@ FlatImageLayout::FlatImageLayout(HANDLE hFile, PEImage* pOwner)
    CONTRACTL_END;
    m_Layout=LAYOUT_FLAT;
    m_pOwner=pOwner;
+
+    HANDLE hFile = pOwner->GetFileHandle();


Just curious: This refactoring to not pass around the file handle and always get it from owner - is this something we can do as a refactoring on the master as well? Or is it something which breaks certain pieces and thus is part of the "prototype, make it simple"?

The hFile argument was was always passed in as GetFileHandle(). So, we can make the change in master as well.

vitek-karas · 2019-09-04T11:24:29Z

src/binder/inc/coreclrbindercommon.h

@@ -22,7 +22,7 @@ class CLRPrivBinderCoreCLR;
 class CCoreCLRBinderHelper
 {
 public:
-    static HRESULT Init();
+    static HRESULT Init(bool isBundle);


Why do we need to store this in the binder if it's already on the AppDomain?
I don't think we should have this stored in two places if avoidable.

Not sure which place is better - right now it would make sense to only have it on the binder - I would also not store it as a global variable, but as an instance variable - basically extension of the TPA list itself.
But going forward maybe the AppDomain is a better place as that would allow us to expose this to the managed world in a much easier way.

Binder cannot access AppDomain members since the type is not constructed yet. The fIsBundle flag is similar to other variables such as AppBaseURL; DynamicDirectory; DynamicBaseDirectory; AppName; AppConfigFile; AppDomainId. But fIsBundle may not be necessary if we make BundleInfo a global.

(If we have single-file plugins in future, we may need to come back and think about this interface).

vitek-karas · 2019-09-04T12:17:41Z

src/binder/assemblybinder.cpp

+            // Is assembly in the bundle?
+            // Single-file bundle contents take precedence over TPA.
+            // Bundled assemblies are treated similar to TPAs.
+            if (g_BinderVariables->fIsBundle)


I'm wondering why do we even need this. The TPA should contain the bundle assemblies anyway - and the code below to handle the TPA will end up calling the GetAssembly with the TPA file path. Currently the BundleInfo will actually deal with full file paths as well (it matches the prefix), but even if it would not, it would make sense to have the TPA modified to contain the "bundle paths".

Since you implemented the bundle hookup in the PEImage all the callers to the PEImage can be pretty much oblivious to whether the file comes from bundle or not.

The bundle meta-data is not reflected in the TPA. For self-contained single-file apps, typically the TPA is just the app.dll.

I don't think we should add the bundled contents to the TPA -- it is unnecessary to search for the bundle contents in two places. In a full implementation, it is better to retire the TPA and use a common host-callback instead.

Thanks I think I finally understand the change 😄
Can you please extend the comment to mention:

TPA is intentionally incomplete for bundles

Bundled assemblies are not listed in TPA - and so we first probe the bundle

The code will call GetAssembly with just a file name, no path

janvorli · 2019-09-04T11:47:29Z

src/pal/src/loader/module.cpp

 {
-    ENTRY("PAL_LOADLoadPEFile (hFile=%p)\n", hFile);
+    ENTRY("PAL_LOADLoadPEFile (hFile=%p, offset=%x)\n", hFile, offset);


The %x is not a correct formatting character for size_t. It should be %zx

janvorli · 2019-09-04T11:50:26Z

src/pal/src/map/map.cpp

@@ -1090,6 +1090,8 @@ CorUnix::InternalMapViewOfFile(
    CFileMappingImmutableData *pImmutableData = NULL;
    CFileMappingProcessLocalData *pProcessLocalData = NULL;
    IDataLock *pProcessLocalDataLock = NULL;
+    INT64 offset = (INT64)dwFileOffsetHigh | (INT64)dwFileOffsetLow;


It seems you are missing shifting the high offset by 32 bits.

janvorli · 2019-09-04T11:52:01Z

src/pal/src/map/map.cpp

@@ -2215,7 +2218,7 @@ void * MAPMapPEFile(HANDLE hFile)
    char* envVar;
 #endif

-    ENTRY("MAPMapPEFile (hFile=%p)\n", hFile);
+    ENTRY("MAPMapPEFile (hFile=%p offset=%x)\n", hFile, offset);


Again, the formatting character for the offset should be %zx

janvorli · 2019-09-04T11:54:20Z

src/vm/appdomain.cpp

@@ -1820,7 +1820,7 @@ void SystemDomain::Init()
        sizeof(MethodDesc),
        sizeof(FieldDesc),
        sizeof(Module)
-    ));
+        ));


The original formatting was intentional - aligning with the LOG.

janvorli · 2019-09-04T13:04:05Z

src/coreclr/hosts/CMakeLists.txt

@@ -1,4 +1,5 @@
 include_directories(inc)
+include_directories(../../inc)


I was wondering why we need this added include and so I've tried to compile without this change - and it was ok both on Linux and Windows.

jkotas · 2019-09-04T15:43:55Z

src/inc/MSCOREE.IDL

@@ -221,7 +224,7 @@ typedef enum
 interface ICLRRuntimeHost : IUnknown
 {
    // Starts the runtime. This is equivalent to CoInitializeCor().


This is legacy hosting interface that number of folks out there still depends on. It is even mentioned in the docs: https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting

Avoid changing it

jkotas · 2019-09-04T15:45:15Z

src/vm/ceemain.h


 class EEDbgInterfaceImpl;

 // Ensure the EE is started up.
-HRESULT EnsureEEStarted(COINITIEE flags);
+HRESULT EnsureEEStarted(COINITIEE flags, const BundleInfo* bundleInfo);


Do you really need to pass the BundleInfo* everywhere? Can we just store it in a static and get everybody who needs to fetch it from the static?

:+1 at the very least we should pass it along at the same place where we pass runtime properties - which is CreateAppDomainWithManager, but ideally this would not go through the COM APIs at all. As mentioned in other comments, I think this should be stored only in one place - the AppDomain is probably as good a place as any right now.

swaroop-sridhar · 2019-09-04T19:36:12Z

Do you have a sample how to create a bundle like this which can be tested?

I've added the steps to the PR description

vitek-karas · 2019-09-05T08:52:36Z

src/vm/corbundle.cpp

+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+//*****************************************************************************
+// CorHost.cpp


We should rename the file to bundleinfo (And the header as well).
Also the comment here is wrong (copy/paste).

src/vm/corbundle.cpp

vitek-karas · 2019-09-05T08:55:15Z

src/vm/corbundle.cpp

+    if (wcsncmp(m_base_path, path, m_base_len) == 0)
+    {
+        path += m_base_len + 1;
+    }


Do we really want to call the probe even if the path doesn't start with the base path?
Is this to support relative paths? In that case all kinds of weird things can happen - I think we should first make sure the path is absolute/normalized before passing it through here.
On Unix this is OK, but on Windows we will also need to normalize the casing - or perform case insensitive compare.

vitek-karas · 2019-09-05T08:59:38Z

src/dlls/mscoree/unixinterface.cpp

@@ -168,6 +169,7 @@ int coreclr_initialize(
            int propertyCount,
            const char** propertyKeys,
            const char** propertyValues,
+            bool bundleProbe(const char*, int64_t*, int64_t*),


I'm fine using some kind of a prototype way - that's not the main concern here (although we should start the discussion on the real solution soon - it might take a while). My main concern is that this breaks the existing hosting.
With your recent changes the COM-ABI is "fixed", but the C-ABI is still broken - meaning CLR built from this branch will not work with hostpolicy. I think we should avoid doing that - as it makes lot of things unnecessarily complex - at the very least it will break tests which rely on real host (probably not that many in coreclr repo, but still).

vitek-karas · 2019-09-05T09:13:01Z

src/binder/assemblybinder.cpp

+            // Is assembly in the bundle?
+            // Single-file bundle contents take precedence over TPA.
+            // Bundled assemblies are treated similar to TPAs.
+            if (g_BinderVariables->fIsBundle)


Thanks I think I finally understand the change 😄
Can you please extend the comment to mention:

TPA is intentionally incomplete for bundles

Bundled assemblies are not listed in TPA - and so we first probe the bundle

The code will call GetAssembly with just a file name, no path

vitek-karas · 2019-09-05T09:23:03Z

src/binder/assemblybinder.cpp

+                for (DWORD i = 0; i < 2; i++)
+                {
+                    SString bundledName(simpleName);
+                    bundledName.Append(candidates[i]);


Nit: please rename the bundleName to probably just assemblyFileName or something similar - it is NOT the name of the bundle itself - confusing.

vitek-karas · 2019-09-05T09:31:47Z

src/binder/assemblybinder.cpp

+                    hr = GetAssembly(bundledName,
+                        TRUE,  // fIsInGAC
+                        FALSE, // fExplicitBindToNativeImage
+                        &pTPAAssembly);


This is sort of wrong - what we want here is to check the bundle probe - if it finds it, great - load that. But if the probe doesn't find it, we want to skip this step and instead continue below with the TPA search.
Right now I think this will actually behave differently:

If the probe finds it - it will work fine

If the probe doesn't find it, I think we will eventually call CreateFile(assemblyFileName, ...) - which I think will search for the file in current directory (didn't try this though) - we should avoid that behavior. Specifically if the assembly is still in TPA, we should use the path from TPA in this case, even if we can find the assembly also in current directory (which may not be the app's directory at all - so we might end up loading random code basically).

The short version is that this change modifies GetAssembly contract - before the path was assumed to be absolute (and it always was) - now it can be relative, but we don't limit that case to just probing the bundle.

swaroop-sridhar · 2019-09-11T10:36:41Z

@vitek-karas I've addressed your feedback in the new commit.

Fix coreclr_initialize interface
- Pass BundleProbe as a property, instead of an additional argument to coreclr_intiialize
Move Corbundle.h to Bundle.h
Introduce BundleLoc to pass around Bundle location information coupled together
Perform the Bundle-Probe close to TPA-probe (and a few other places like BindToSystem)
so that if an assembly is not found in the bundle, it is not accidentally loaded from the disk.
Check the bundle contents while loading assemblies through Assembly.LoadFrom(path).

vitek-karas · 2019-09-11T11:07:53Z

src/dlls/mscoree/unixinterface.cpp

@@ -128,7 +130,7 @@ static void ConvertConfigPropertiesToUnicode(
    for (int propertyIndex = 0; propertyIndex < propertyCount; ++propertyIndex)
    {
        propertyKeysW[propertyIndex] = StringToUnicode(propertyKeys[propertyIndex]);
-        propertyValuesW[propertyIndex] = StringToUnicode(propertyValues[propertyIndex]);
+        propertyValuesW[propertyIndex] = propertyValues[propertyIndex] != nullptr ? StringToUnicode(propertyValues[propertyIndex]) : nullptr;


Does this mean we try to actually convert the bundle probe pointer to UTF16 string? Isn't that like super dangerous...

vitek-karas · 2019-09-11T11:08:22Z

src/inc/bundle.h

+
+class Bundle;
+
+struct BundleLoc


Nit: I would prefer BundleFileLocation or something along those lines. 'loc' to me refers to localization - and there's no need to use abbreviation here. Also "bundle location" is ambiguous - normally I would read that as "the location of the bundle".

vitek-karas · 2019-09-11T11:09:31Z

src/binder/inc/assembly.hpp

+                            PEImage   **ppPEImage,
+                            PEImage   **ppNativeImage,
+                            BOOL        fExplicitBindToNativeImage,
+                            BundleLoc   bundleLoc);


Can we pass this as const & - const to make it explicit and & since it's two 64bit values...

I originally used the type const BundleLoc &. But this method is declared as extern C API, and I think the behavior of C++ reference parameters is not well-defined on such functions.

I could use const BundleLoc *, but it'll need nullptr checks in a few places. Also, passing by value is convenient in a few other methods due to the use of default parameters. So, I left it passed by value here too.

I wonder if the STDAPI is still needed...
@jkotas - Two questions:

Is there a reason the binder APIs are declared as if exported? (although they really are not)

What is the guidance on using a bit more modern C++ - like byref params, const and so on... in CLR ?

This is left-over from fusion. Fusion was compiled as a separate thing, with its own infrastructure. Fusion did not include any VM headers, and VM did not include any fusion implementation headers. The hand shake between VM and fusion was done via extern "C" methods and COM interfaces.

What is the guidance on using a bit more modern C++

As long as all compilers that we support work with it, it is ok.

swaroop-sridhar · 2019-09-11T19:53:00Z

@vitek-karas I've made the corrections you suggested in d2a0846

swaroop-sridhar · 2019-09-11T19:56:08Z

@vitek-karas: I suggest we checkin the change to single-exe branch, so that @sbomer doesn't have to work with two branches for perf-measurement. I'm happy to address more issues in subsequent checkins.

vitek-karas · 2019-09-11T20:00:38Z

If it works - then sure - let's merge it...
The one thing I'm really nervous about is the usage of property value pointer as the probe pointer - not in itself, but the conversion to UTF16 which I pointed out above. That one I think we absolutely need to address, even for the prototype... (and I know that the final product will use different mechanism).

swaroop-sridhar · 2019-09-11T20:44:44Z

If it works - then sure - let's merge it...
The one thing I'm really nervous about is the usage of property value pointer as the probe pointer - not in itself, but the conversion to UTF16 which I pointed out above. That one I think we absolutely need to address, even for the prototype... (and I know that the final product will use different mechanism).

@vitek-karas definitely agree about the Probe-pointer issue, I've already fixed that (d2a0846). I've also renamed BundleLoc. I was only talking about optimizing parameters and such -- in a later checkin. I know of a few more things that can be improved.

This commit has the following changes: corebundle Host: - Remove code that deals with extraction to temporary dirs. - Process the bundle manifest and export a callback to look for files in the bundle. - The TPA doesn't contain bundled assemblies any more. CoreCLR Initialization: - Setup BundleInfo structure, and pass it to appropriate initializers - Expect to find System.Private.Corelib in the bundle Bundle Info: - Holds information about the bundle - Exports a wrapper around the bundle runner's probe function for path adjustments - Performs Unicode/UTF8 translations as necessary (because host uses UTF8 on Unix) AppDomain: - Add methods obtain BundleInfo Binder: - Look for assemblies in the Bundle before searching the TPA PEImage: - FlatImageLayout and MappedImageLayout can take files embedded within the bundle. - Memory mapping routines can take files embedded within the bundle. A few other fixes necessary to build/run. Testing Tested templates: console, webapi Performance Console Hello World No noticable change in execution time. Run from (normal) publish directory: 0.104 Run from single-file: 0.099s Startup of WebApiTemplate: Measured using https://github.com/rynowak/link-a-thon on a local (real) machine. Both cases had runs with similar timing; but the average startup was consistently slightly faster for single-exe runs. Run from (normal) publish directory: 801.8271 782.9489 790.8015 798.7449 797.3141 783.0968 790.0275 786.7621 802.2383 778.9866 Average startup time (ms): 791.27478 Run from single-file: 742.1464 692.1949 795.1612 798.4845 809.4075 794.2644 790.6254 634.1305 686.2718 806.5665

* Fix coreclr_initialize interface * Pass BundleProbe as a property, instead of an additional argument to coreclr_intiialize * Move Corbundle.h to Bundle.h * Introduce BundleLoc to pass around Bundle location information coupled together * Perform the Bundle-Probe close to TPA-probe (and a few other places like BindToSystem) so that if an assembly is not found in the bundle, it is not accidentally loaded from the disk. * Check the bundle contents while loading assemblies through Assembly.LoadFrom(path).

Also fix an error during initialization of Bundle() structure.

Fix a bug that caused errno check after open() to be re-written

janvorli · 2019-09-16T08:59:06Z

src/binder/assemblybinder.cpp

        ReleaseHolder<Assembly> pSystemAssembly;
+        StackSString sCoreLib;


A nit - it seems this move was not necessary. It is better to keep the variable definition close to its first usage.

janvorli · 2019-09-16T09:03:51Z

src/coreclr/hosts/unixcorebundle/bundle/runner.cpp

+{
+    for (file_entry_t& entry : m_manifest.files)
+    {
+        if (strcmp(entry.relative_path().c_str(), relative_path) == 0)


You can use just entry == relative_path instead of strcmp here.

janvorli · 2019-09-16T09:09:47Z

src/dlls/mscoree/unixinterface.cpp

@@ -160,6 +175,7 @@ extern "C" int coreclr_create_delegate(void*, unsigned int, const char*, const c
 // Returns:
 //  HRESULT indicating status of the operation. S_OK if the assembly was successfully executed
 //
+


A nit -can you please remove this extra empty line?

janvorli · 2019-09-16T22:05:55Z

src/dlls/mscoree/unixinterface.cpp

@@ -127,6 +130,18 @@ static void ConvertConfigPropertiesToUnicode(

    for (int propertyIndex = 0; propertyIndex < propertyCount; ++propertyIndex)
    {
+        if (strcmp(propertyKeys[propertyIndex], "BUNDLE_PROBE") == 0)


I don't like how this makes the function do something it is not supposed to do just to save an iteration over the few properties. This function should be what its name says - just convert property keys and values to unicode.
I would much more prefer passing the bundle probe address formatted as hex string and parse it from the string in the coreclr_initialize. This is not perf sensitive code in any way, so we should avoid hacks like passing a non-string in an array of strings.

I certainly don't like this way of passing in the bundle-probe (or the implementation of this function). This is pretty much a hack for now -- that communicates the bundle-probe address without changing the host layer interfaces.

In the product, I think we should design (an additional) host-interface, to pass in a structure of well-typed members, instead of a list of strings. We should design the host-interface to accommodate this and other upcoming features. Until then, my opinion is to let the hack remain, instead of polishing something that'll go away.

Ok, if this is just a temporary way, then let's keep it for now. Can you please create an issue for fixing that so that we don't forget about the need to fix it?

we should design (an additional) host-interface

For new host-interface designs, we should also take into account preloading scenario described in #25725

cc @alpencolt @kbaladurin

I created an issue to track this: https://github.com/dotnet/coreclr/issues/26752

swaroop-sridhar · 2019-09-17T04:56:17Z

@janvorli I didn't notice your latest comments (was only looking at test-pass, since there wasn't much activity the last few days). I'll address your feedback in a subsequent change. Thanks.

@janvorli

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in dotnet#26504

@janvorli

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in dotnet#26504

@janvorli

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in #26504

This change implements: * Runtime changes necessary to load assemblies directly from the bundle: * Design notes about [Load from Bundle](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#peimage-loader) * Most of these changes are directly from dotnet/coreclr#26504 and dotnet/coreclr#26904 * Hostpolicy change to not add bundled assemblies to TPA list: * Design notes about [Dependency Resolution](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#dependency-resolution) * TBD (separately) items: Fix for hammer servicing dotnet#36031 Fixes dotnet#32822

* Single-File: Run from Bundle This change implements: * Runtime changes necessary to load assemblies directly from the bundle: * Design notes about [Load from Bundle](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#peimage-loader) * Most of these changes are directly from dotnet/coreclr#26504 and dotnet/coreclr#26904 * Hostpolicy change to not add bundled assemblies to TPA list: * Design notes about [Dependency Resolution](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#dependency-resolution) * TBD (separately) items: Fix for hammer servicing #36031 Fixes #32822 * Address some code review feedback * Address futher review feedback * Fix Assembly.ni.dll and Assembly.dll path usage. * Remove unused full-path computation for corelib when loading from bundle. * Add tracing when loading from bundle. * Add bundle-probing for satellite assemblies. * Adapt the BundleRename test to the .net5 scenario where assemblies are loaded from bundle.

* Move Mobile test runners to libraries/common and use P2P references (#36411) * Move Mobile test runners to libraries/common and use P2P references * PR Feedback Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> * Fix execution of large version bubble composite images (#36373) Large-bubble composite images are special in having more entries in the manifest metadata than in the component assembly table: When the build starts, all component assemblies get hard-injected into the manifest metadata and subsequently we lazily add those additional reference assemblies (within the same version bubble) as we need for encoding signatures. Thanks Tomas * Remove Microsoft.NETCore.Platforms.Future project (#36407) It's no longer used. * [master] Update dependencies from mono/linker dotnet/llvm-project dotnet/xharness (#36336) * Update dependencies from https://github.com/mono/linker build 20200512.1 - Microsoft.NET.ILLink.Tasks: 5.0.0-preview.3.20261.2 -> 5.0.0-preview.3.20262.1 * Update dependencies from https://github.com/dotnet/llvm-project build 20200512.1 - runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 - runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 - runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 - runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 - runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 - runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: 6.0.1-alpha.1.20261.3 -> 9.0.1-alpha.1.20262.1 * Update dependencies from https://github.com/dotnet/xharness build 20200513.4 - Microsoft.DotNet.XHarness.Tests.Runners: 1.0.0-prerelease.20261.4 -> 1.0.0-prerelease.20263.4 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Rework rejit test to ensure subprocess environment is suitable (#36420) This test is optimization sensitive, but it invokes a subprocess that will inherit environment variables like `COMPlus_JITMinOpts` that can impact optimization of code jitted in the subprocess and cause the test to fail. So, update the parent process code to override `COMPlus_JITMinOpts` and `COMPlus_JitStress` for the child process. Closes #35742. * Implement GetEntrypointExecutableAbsPath on SunOS (#36430) * Implement `GetEntrypointExecutableAbsolutePath`. * Fix a warning from newer gawk (v5.0.1 from 2019): > ```sh > awk: /runtime/src/coreclr/src/nativeresources/processrc.awk:54: > warning: regexp escape sequence `\"' is not a known regexp operator > ``` * Fix mono runtime build warnings when building iOS config (#36435) ``` /Users/alexander/dev/runtime/src/mono/mono/mini/aot-runtime.c:5647:13: warning: unused variable 'image' [-Wunused-variable] /Users/alexander/dev/runtime/src/mono/mono/mini/simd-intrinsics-netcore.c:11:1: warning: no previous prototype for function 'mono_simd_intrinsics_init' [-Wmissing-prototypes] /Users/alexander/dev/runtime/src/mono/mono/utils/mono-state.c:1230:1: warning: no previous prototype for function 'mono_crash_save_failfast_msg' [-Wmissing-prototypes] /Users/alexander/dev/runtime/src/mono/mono/utils/mono-state.c:1236:1: warning: no previous prototype for function 'mono_crash_get_failfast_msg' [-Wmissing-prototypes] ``` * Update windows prerequisites to be more specific about SDK (#36438) * Update windows prerequisites to be more specific about SDK * Global installation of nightly SDK is required to normally browse the solution files in VS, as there is no way to supply SDKs when the solution files are opened through VS. * Update windows prerequisites * allow newer versions of VS workloads * [debugger] Removing some asserts (#36234) Removing some asserts and returning err_invalid_argument with an error message when it's possible. Fixes https://github.com/mono/mono/issues/19651 Co-authored-by: thaystg <thaystg@users.noreply.github.com> * Work around -no_weak_imports issue with recent Xcode (#36436) See https://github.com/mono/mono/issues/19393. We can use the `-Werror=partial-availability` as a good alternative until the Xcode bug is fixed. * Remove duplicate configuration properties (#36442) * Update message for generic type. (#36434) * Consolidate subset projects into ProjectToBuild (#36441) Consolidating subset projects into a single ProjectToBuild item type to allow specifying projects to build from different subsets after the subset was already built. * [master] Update dependencies from dotnet/arcade mono/linker dotnet/xharness (#36445) * Update dependencies from https://github.com/dotnet/arcade build 20200511.9 Microsoft.DotNet.XUnitExtensions , Microsoft.DotNet.VersionTools.Tasks , Microsoft.DotNet.ApiCompat , Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Build.Tasks.Packaging , Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk , Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk , Microsoft.DotNet.CodeAnalysis , Microsoft.DotNet.XUnitConsoleRunner , Microsoft.DotNet.GenAPI , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.RemoteExecutor , Microsoft.DotNet.GenFacades From Version 5.0.0-beta.20258.8 -> To Version 5.0.0-beta.20261.9 * Update dependencies from https://github.com/mono/linker build 20200514.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20262.1 -> To Version 5.0.0-preview.3.20264.1 * Update dependencies from https://github.com/dotnet/xharness build 20200514.1 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20263.4 -> To Version 1.0.0-prerelease.20264.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Support additional friendly names on ECC OIDs Expand the name/OID table to support a m:m relationship (secp256r1 and nistP256 are both 1.2.840.10045.3.1.7; 1.3.14.7.2.3.1 and 1.2.840.113549.1.1.4 are both md5RSA) and add in the alternative names for the secp{size}r1 curves (size in 256, 384, 521). * Add an atexit handler to bypass calls into ERR_ string routines. This works around Ubuntu's apparent lack of NO_ATEXIT support in their build of OpenSSL. * [mono] Shrink Android APK size (#36437) * Shrink Android apk size * bump xharness cli, use cmake config * Update Linux docker instructions (#36370) * Update linux-instructions.md to use root build.sh * Make depproj intermediate output paths unique again (#36451) Fixes https://github.com/dotnet/runtime/issues/36255 Depprojs depend on IntermediateOutputPath being set in a props file early enough as restore is happening per configuration. Even though it isn't recommended that the TargetFramework property is read before the project is loaded, we currently encode the TargetFramework in the IntermediateOutputPath for depproj files. The long-term fix is to get rid of per configuration restores by getting rid of our depproj files. * [Mono] Don't set thread name of main thread on Linux (#36116) * [mono] Record MonoNativeThreadId of the main thread We would like to know the MonoNativeThreadId (pthread_t on Linux) of the main thread of the application. We can identify the main thread (on linux) because it is the one for which `gettid () == getpid ()`. (`gettid()` returns a `pid_t` which is not the same thing as a `pthread_t`, hence this roundabout way of detecting it.) A complication arises in embedding scenarios: the main thread is not necessarily the one that calls `mono_jit_init` or otherwise interacts with the runtime. Therefore we do the `gettid() == getpid ()` test at `MonoThreadInfo` creation time when we call `register_thread`. If the main thread never interacts with Mono, the main thread is not known to us. * [mono] Don't set name of main thread on Linux Setting the name of the main thread also changes the name of the process. Fixes https://github.com/dotnet/runtime/issues/35908 The corresponding fix for CoreCLR is https://github.com/dotnet/runtime/pull/34064 * Re-enable test from https://github.com/dotnet/runtime/pull/34064 * Startup optimization w.r.t. manifest access in composite mode (#36446) During my work on fixing runtime crashes in composite build with large version bubble enabled I noticed room for startup perf improvement and a very slight working set optimization: For component assemblies of a composite image, we can basically share the cache of those manifest assembly references that have already been resolved (GetNativeMetadataAssemblyRefFromCache) within the native image because that is the logical owner of the manifest metadata. In the "asymptotic" case of composite images with many components, the pre-existing behavior was basically a quadratic O(n^2) algorithm in the number of component assemblies. This change reduces it to linear in the sense that all assembly references from the composite image get resolved only once. Thanks Tomas * Replace System.Native call in Interop.Errors.cs * Optimize ToScalar() and GetElement() to use arm64 intrinsic (#36156) * ARM64 intrisic for ToScalar() and GetElement() * Fixed GetElement to just operate on constants * Fix bug in rationalize for Vector64<long> * fix NotSupported issue for GetElement and ToScalar * Reuse the baseType/retType in impSpecialIntrinsic and impBaseIntrinsic * Update comment * fix breaks * add comments * ran jit-format * Refactored to move common logic inside isSupportedBaseType * review comments * reuse simdSize * formatting * one missing formatting * Next round of Struct Improvements (#36146) - Allow accessing a SIMD12 as 16 bytes if it's the single field of a parent struct of 16 bytes. - On x64/ux don't copy an argument just because it is promoted; the copy would force it to memory anyway. - Use block init for a promoted struct that's been marked lvDoNotEnregister. - Allow field-by-field copy of structs with the same fields * Issue 36212: Remove space in parameter name (#36439) * Issue 36212: Remove space in parameter name * Update src/coreclr/src/vm/comutilnative.cpp Co-authored-by: Stephen Toub <stoub@microsoft.com> Co-authored-by: Stephen Toub <stoub@microsoft.com> * Remove unnessary guard against switched property names * R2RTest - Partial composite images (#36471) Allow compiling composite R2R images which reference assemblies not in the composite. For example, this would allow compiling a set of application assemblies with references to ASP.NET / Framework. Currently R2RTest treats all references as unrooted inputs for the composite image. * Add issue templates (#36431) * more * commented * yml * Add an issue template for API proposals * Update .github/ISSUE_TEMPLATE/api-proposal.md Co-authored-by: Stephen Toub <stoub@microsoft.com> * Update .github/ISSUE_TEMPLATE/api-proposal.md Co-authored-by: Stephen Toub <stoub@microsoft.com> * Update .github/ISSUE_TEMPLATE/api-proposal.md Co-authored-by: Stephen Toub <stoub@microsoft.com> * comment out template instructions * apply consistent naming * feedback * add blank * H2 and H3 Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: Stephen Toub <stoub@microsoft.com> * Implement unwrapping a ComWrappers CCW when dumping a stowed exception. (#36360) * Move mobile test runners to libs.pretest instead of P2P (#36473) * Add options to ignore default values during serialization (#36322) * Add options to ignore default values during serialization * Address review feedback * Fix typo in test * Adding the support of reusing machine-wide credentials in the case where the machine is already domain-joined to the LDAP Server. (#36405) * Adding the support of reusing machine-wide credentials in the case where the machine is already domain-joined to the LDAP Server. Co-authored-by: Alexander Chermyanin <flamencist@mail.ru> * Addressing feedback and adding ldap4net to TPN Co-authored-by: Alexander Chermyanin <flamencist@mail.ru> * Add dotnet cli to runtime test Helix jobs (#35426) * Currently managed tools such as crossgen2 and XUnit are run against the built runtime which is slow on Debug builds, and can obscure errors when the XUnit test harness fails due to an introduced runtime bug. * Add `xunit.console.runtimeconfig.dev.json` which allows `xunit.console` to use the repo-local dotnet.cmd on dev boxes and the same version installed on the path in Helix. * When running crossgen2 during test execution, support both local dev and Helix scenario when deciding which dotnet to run. On Helix, simply use `dotnet` which assumes a compatible dotnet runtime is in the path. Locally, tests are run with `runtest.cmd|sh` which sets `__TestDotNetCmd` to the repo-local dotnet script. This preserves the characteristic that no machine-wide 5.0 dotnet runtime must be installed for the runtime tests. * Update batch scripting to also use `dotnet.cmd|sh` when running Crossgen2 replacing corerun.exe. * crossgen2's `runtimes` folder is not getting copied to `CORE_ROOT` which causes the runtime host to abort the launch on the Unix CI VMs since crossgen.deps.json refers to files in that subfolder. Adjust the `CORE_ROOT` pruning in `Directory.Build.targets` to include subfolders for the two tools that need it. * Improve XUnit test boilerplate. Printing `Exception.Message` doesn't include stack trace. Use `ToString()` instead. * Import notargets sdk in `helixpublicwitharcade.proj`. It doesn't use the official sdk so `BundledNETCoreAppPackageVersion` wasn't set. Import the `Microsoft.Build.NoTargets` sdk so we can find the bundled runtime package version. * Fix DAC layout in checked builds (#35542) * Improve Uri.Equals by removing unsafe code (#36444) * Improve Uri.Equals * Remove not useful and duplicated comments * Refactor `HasMultiRegRetVal` and `impFixupCallStructReturn`. (#36465) * Fix target definitions. They were used in asserts only, no changes. * Fix failures after a recent HW changes. * Add a const getter for `ReturnTypeDesc` from a call. Used to make some new methods const as well. * Refactor `HasMultiRegRetVal` and `impFixupCallStructReturn`. Delete an unnecessary nested condition and make checks more straightforward. * Delete an extra `.` in some dumps. * Add an additional check that `ReturnTypeDesc` is initialized. * Remove old `const_cast` around `GetReturnTypeDesc`. * Replace non-const `GetReturnTypeDesc` with other methods. * Fix uninitialized `gtSpillFlags, gtOtherRegs, gtReturnTypeDesc` in `fgMorphIntoHelperCall`. * Implement get_own_executable_path for SunOS (#36514) * [mono] Improve Android OpenSSL temp hack (#36463) * [docs] How to run tests on iOS and Android (#36297) * Update XHarness for latest fixes (#36484) This includes a couple of fixes in xharness. * order issue template files (#36524) * Update 02_api_proposal.md * Update 02_api_proposal.md * Consolidate NetCoreAppCurrent properties (#35953) * Rename 'Blank' issue template to 'Blank issue' (#36533) * Update and rename 04_blank.md to 04_blank_issue.md * Use consistent casing * Ignore .github folder changes in CI (#36530) * Clean up text in config.yml issue template (#36529) * Convert Extract(0) to ToScalar() (#36474) * Convert Extract(0) to ToScalar() * Update the shim * Move mobile AppBuilder & AOTCompiler projects into tools-local (#36478) Moving the projects will make sure their artifacts are always available to the different CI legs. * Add comment to NetworkStream's ctor about socket.Blocking check (#36539) * [master] Update dependencies from mono/linker Microsoft/vstest dotnet/xharness (#36525) * Update dependencies from https://github.com/mono/linker build 20200515.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20264.1 -> To Version 5.0.0-preview.3.20265.1 * Update dependencies from https://github.com/microsoft/vstest build 20200515-01 Microsoft.NET.Test.Sdk From Version 16.7.0-preview-20200429-01 -> To Version 16.7.0-preview-20200515-01 * Update dependencies from https://github.com/dotnet/xharness build 20200515.1 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20264.9 -> To Version 1.0.0-prerelease.20265.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> * Disable MemoryCacheTest.Trim in arm64 and enable Runtime.Caching tests on Unix (#36494) * Enable basic generation of ngen pdb from composite image (#36311) * Enable basic generation of ngen pdb from composite image - Port non-line number handling portion of ngen pdb from crossgen to r2rdump - This pdb generation logic may be used for both composite and non-composite R2R images - Only pdb generation is supported. Perfmap generation for unix is not supported - pdb generation is only supported on Windows x86 and amd64 platforms. Diasymreader is not supported on other systems - Pdb generation does not generation symbols with the same names as crossgen. Instead it uses the name generator from r2rdump. For current needs this should be sufficient - Update composite file format so that pdb generation process will work. Major difference is that a CorHeader is always produced for composite images now. This CorHeader is not used by the runtime, but is required for DiaSymReader to generate an ngen pdb. * Next round of multireg preliminary changes (#36155) This is a zero-diff set of mostly refactoring changes in preparation for supporting multireg locals: - Move `genRegCopy()` and `genStructReturn()` to codegencommon.cpp, making a new method for `genSIMDSplitReturn` which is target-specific. - Factor out a new `genUnspillLocal` method from `genUnspillRegIfNeeded()`. - Similarly factor out `genSpillLocal()` - Rename `genMultiRegCallStoreToLocal()` and more generally support multireg local stores. - Fix a bug in the order and shift amount for last-use bits on `GenTreeLclVar` - Some additional cleanup and preparatory changes * Fix argument exception warnings on runtime (#35717) * Fix argument exception warnings on runtime * Apply feedback * Addressing feedback and test fix * Fix test failures * Fix tests for NetFx run * Fix suppressed warning for socket.SendAsyn(e) and fix corresponding tests * Applied feedback * More HTTP/2 performance (and a few functional) improvements (#36246) * Use span instead of array for StatusHeaderName * Fix potential leak into CancellationToken We need to dispose of the linked token source we create. Also cleaned up some unnecessarily complicated code nearby. * Fix HttpConnectionBase.LogExceptions My previous changes here were flawed for the sync-completing case, and also accidentally introduced a closure. * Clean up protocol state if/else cascades into switches * Consolidate a bunch of exception throws into helpers * Fix cancellation handling of WaitFor100ContinueAsync * Change AsyncMutex's linked list to be circular * Remove linked token sources Rather than creating temporary linked token sources with the request body source and the supplied cancellation token, we can instead just register with the supplied token to cancel the request body source. This is valid because canceling any part of sending a request cancels any further sending of that request, not just that one constituent operation. * Avoid registering for linked cancellation until absolutely necessary We can avoid registering with the cancellation token until after we know that our send is completing asynchronously. * Remove closure/delegate allocation from WaitForDataAsync `this` was being closed over accidentally. I can't wait for static lambdas. * Avoid a temporary list for storing trailers Since it only exists to be defensive but we don't expect response.TrailingHeaders to be accessed until after the whole response has been received, we can store the headers into an HttpResponseHeaders instance and swap that instance in at the end. Best and common case, we avoid the list. Worst and uncommon case, we pay the overhead of the extra HttpResponseHeaders instead of the List. * Delete dead AcquireWriteLockAsync method * Reduce header frame overhead Minor optimizations to improve the asm * Remove unnecessary throws with GetShutdownException * Avoid extra lock in SendHeadersAsync * Move Http2Stream construction out of lock Makes a significant impact on reducing lock contention. * Streamline RemoveStream Including moving credit adjustment out of the lock * Move response message allocation to ctor Remove it from within the lock * Reorder interfaces on Http2Stream IHttpTrace doesn't need to be prioritized. * Address PR feedback * Delete DebugThreadTracking from networking code (#36549) The System.Net.* libs in dotnet/runtime inherited this from .NET Framework. To my knowledge it's not once helped flag any issues in dotnet/runtime, it's only built into debug builds, it's become very inconsistent as the code base has evolved, and it's just cluttering stuff up. So, goodbye. * Build an apphost with hostfxr and hostpolicy linked in (#36230) * hostfxr: Build most of hostfxr as a static library This is part of the work to create an apphost that bundles both hostfxr and hostpolicy. The main distinction between the static and shared versions of hostfxr is that the static version contains a hostpolicy resolver that references the hostpolicy symbols directly rather than loading them from a DLL. * hostpolicy: Build as a static library This change is part of the work to enable an apphost that bundles both hostfxr and hostpolicy. There's no distinction between hostpolicy that's built as a shared library and as a static library: the shared library is built by linking an empty object file with the static library. * corehost: Allow linking of hostfxr and hostpolicy with apphost Provide a hostfxr_iface class, that abstracts how the hostfxr functions called by the early stage in the hosting layer is resolved. * dotnet: Teach the muxer binary about hostfxr_iface * apphost: Teach apphost about hostfxr_iface This provides two implementations of hostfxr_iface: one for the static apphost, which bundles hostfxr and hostpolicy, and another for the conventional apphost, which loads them dynamically on startup. * Add exports for hostfxr and policy * Exports for unix * EXPORTS_LINKER_OPTION * use generateversionscript.awk from ENG * Move fxr files out of static * Fixes for Linux * Fix for win-x86 * move HEADERS next to SOURCES similarly to other files. * PR feedback (simplifying hostpolicy_resolver::try_get_dir for static host) * Publish static_apphost to Microsoft.NETCore.App.Host * bind to entry points without probing, when in a static host. * Add a test case * renamed hostfxr_iface --> hostfxr_resolver_t * renamed shared --> standalone * rename static_apphost --> singlefilehost * Signing exclusions for singlefilehost * switched StaticHost tst to a different asset (mostly a copy of StandaloneApp) * get_method_module_path Co-authored-by: Leandro Pereira <leandro.pereira@microsoft.com> Co-authored-by: Swaroop Sridhar <swaroop.sridhar@microsoft.com> * Produce DropFromSingleFile annotations in RuntimeList.xml (#36578) * SingleFileHostInclude * style fixes * PR feedback * Fix IL projects build inside visual studio (#36570) * SslStream.AuthenticateAs sync overloads with SslOptions made public (#36221) * [mono] Use "dotnet publish" for Android sample with ILLink (#36593) * Port changes to shared files Nullable.cs, Enum.cs (#36597) * updating area owners (#36467) * updating area owners * adding cross-gen contrib as owner * Fix StackTraceTests to work with JIT optimization (#36596) Disable optimization/inlining on methods that are expected to remain on the stack. * [mono] Enable some System.Reflection.Emit tests (#35872) * [interp] Don't share interp_in signatures for different valuetypes (#36520) We share interp_in wrappers for different types of signatures if the corresponding params are equivalent. This was added in https://github.com/mono/mono/commit/5cbe93884798684efbb81abd79e0e2a170544b75. This was reusing some sharing mechanism used by gsharedvt. Those wrappers are shared with regard to managed->managed transitions so it takes additional freedoms, converting all valuetypes to ValueTuples instances. These can end up being marshalled differently from the initial struct so we can't use this valuetype sharing infrastructure in the interp_in_wrappers which can operate on native structs. Fixes test_0_marshal_struct_delegate from pinvoke3.cs Co-authored-by: BrzVlad <BrzVlad@users.noreply.github.com> * [master] Update dependencies from mono/linker Microsoft/vstest dotnet/xharness (#36598) * Update dependencies from https://github.com/mono/linker build 20200515.2 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20265.1 -> To Version 5.0.0-preview.3.20265.2 * Update dependencies from https://github.com/microsoft/vstest build 20200515-03 Microsoft.NET.Test.Sdk From Version 16.7.0-preview-20200515-01 -> To Version 16.7.0-preview-20200515-03 * Update dependencies from https://github.com/dotnet/xharness build 20200515.8 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20265.1 -> To Version 1.0.0-prerelease.20265.8 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Removed left-over NUnit references from comments (#36606) * Sync crossgen2 shared files (#36610) * CMake: Point download URLs directly to https:// (#36615) * Naricc/ci interpreter arm64 (#36258) Change mono interpreter runs to be a scenario instead of a seperate leg. Also enable arm64 interpreter runs, and add test exclusions. * Single-File: Run from Bundle This change implements: * Runtime changes necessary to load assemblies directly from the bundle: * Design notes about [Load from Bundle](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#peimage-loader) * Most of these changes are directly from https://github.com/dotnet/coreclr/pull/26504 and https://github.com/dotnet/coreclr/pull/26904 * Hostpolicy change to not add bundled assemblies to TPA list: * Design notes about [Dependency Resolution](https://github.com/dotnet/designs/blob/master/accepted/2020/single-file/design.md#dependency-resolution) * TBD (separately) items: Fix for hammer servicing #36031 Fixes #32822 * Address some code review feedback * Add iOS x86 build (#36602) Needed for the iOS Simulator for 32bit iOS devices. * [mono] Fix AssemblyLoadContext.GetRuntimeAssembly() for AssemblyBuilders (#36367) * Fix mono file name on iOS/Browser (#36646) We initially intended to just use libmono.so/dylib as the name to simplify and follow the libcoreclr.dylib pattern and we did that by just copying to a different name after the build. However that didn't work on Android since the name gets embedded inside the binary and Android checks that these match, so we'd either have to change the (auto)make to use the correct library name (and possibly creates complex conditionals in the Makefile for netcore) or go back to using `libmonosgen-2.0` on iOS. We decided to do the latter. * [mono] Linking statically ICU shim on mono (#35790) Linking statically ICU shim on mono for windows, linux, macOs and android. * Update dependencies from https://github.com/mono/linker build 20200518.2 (#36647) Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20265.2 -> To Version 5.0.0-preview.3.20268.2 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Sync one more shared crossgen2 file (#36638) * Delete OrderBy(...).First{OrDefault}(...) optimization (#36643) The optimization removes the O(n log n) cost of the OrderBy. But it can result in executing the predicate passed to First{OrDefault} more than in .NET Framework; it would always execute it n times, whereas previously it would execute it <= n times. Developers have expressed concern about the change, in particular when using a relatively expensive predicate on a relatively short list, or when unadvisedly relying on side-effecting predicates. * Fix failing Sockets tests after argument exception changes (#36645) * Removed redundant visibility check (#36648) * Remove unnecessary initialization from Utf8JsonWriter ctors (#36651) * Remove PreserveDependency on non-existing type (#36657) The DefaultArrayConverter type was been refactored away causing the linker to warn about it. * Produce Mono+LLVM runtime packs on desktop platforms (#35841) * Add LLVM Mono runtime build * Switch from 'llvm' boolean to 'runtimeVariant' freeform string in yaml This makes it easier to add oddball variant builds, without a big pile of booleans for every possible variant * Add an LLVM suffix to installer nupkgs * Add runtimeVariant to CoreCLR artifact names * Add installer run for LLVM JIT Mono * Actually specify LLVM or not to installer build * Unique name for LLVM installer run * Ensure log uploads are disambiguated * Fix dependency in full matrix * Add LLVMAOT variant, which bundles llc/opt for current arch * Make sure we don't use Mono.LLVM package names on CoreCLR or Mobile * Fix perf runs to deal with runtimeVariant * Try to reconcile perf test artifact names * Make bundling llc/opt the default when LLVM enabled on Mono * Fix R2RTest parsing of the issues.targets (#36650) The format of the file has changed a bit some time ago, but the R2RTest wasn't updated accordingly, so test exclusion stopped working. This change fixes it. * Remove BuildOS (#36667) * Prefix protected fields with underscore for the internal XmlRawWriter (#35759) * Prefix protected fields with underscore for the internal XmlRawWriter and its descendant classes * WASM app builder changes. (#36422) * Add an ExtraAssemblies parameter to the WasmAppBuilder task. * Pass more assemblies to the pinvoke table generator. * Improve the wasm sample. * Move WasmAppBuilder to tools-local. * [mono] Enable System.Runtime.Tests on Android (#36655) * Rewrite NegotiateStream.XxAsync operations with async/await (#36583) * Rewrite NegotiateStream.Read/Write* operations with async/await Gets rid of a bunch of IAsyncResult cruft and makes the XxAsync APIs cancelable. * Combine NegoState into NegotiateStream * Rewrite AuthenticateAs* with async/await * Add more NegotiateStream tests Including for cancellation and a product fix to enable cancellation. * Update ref with overrides * Remove custom IAsyncResults from System.Net.Security * Fix UnitTests project * Libunwind1.5rc2 (#36027) * Add libunwind 1.5rc2 source Rename unused autoconfig dir aux -> aux_ consistent with original checkin * Delete files added by 1.3-rc1 * Update libunwind version in CMake * Add libunwind-version.txt * Add changes for SunOS from @am11 * Revert change to oop * Remove obsolete add_definition * Fix musl build * Fix comment * Be consistent and use HOST * Fix error in unw_sigcontext libunwind/libunwind#179 Introduced by libunwind/libunwind#71 __reseverved needs to be big enough to store a unw_fpsimd_context_t Which includes 32 128-bit registers, stored as 64 64-bit half registers. Fix off by 2 issue * Arm64 support !UNWIND_CONTEXT_IS_UCONTEXT_T Co-authored-by: Adeel <adeelbm@outlook.com> * Checking strings against length seem to have better perf (#36443) * Chckecing strings against length seem to have better perf * Apply suggestions from code review Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> * Apply suggestions from code review Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> * Update src/libraries/System.Drawing.Common/src/System/Drawing/BitmapSelector.cs Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> * Apply suggestions from code review * Update src/libraries/System.Private.CoreLib/src/System/Reflection/AssemblyNameFormatter.cs * fix build error * Update SmtpClient.cs * Update AssemblyNameFormatter.cs * Update LoggingEventSource.cs * Fix build error * Apply suggestions from code review * Update src/libraries/System.ServiceModel.Syndication/src/System/ServiceModel/Syndication/SyndicationContent.cs * Fix build error * Apply suggestions from code review Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> * Fix RuntimeInformation.IsOSPlatform for Browser/WASM (#36665) The native code was still using the previous `WEBASSEMBLY` name instead of `BROWSER` as decided in https://github.com/dotnet/runtime/issues/33328. * [master] Update dependencies from mono/linker Microsoft/vstest (#36692) * Update dependencies from https://github.com/mono/linker build 20200518.5 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20268.2 -> To Version 5.0.0-preview.3.20268.5 * Update dependencies from https://github.com/microsoft/vstest build 20200518-01 Microsoft.NET.Test.Sdk From Version 16.7.0-preview-20200515-03 -> To Version 16.7.0-preview-20200518-01 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Remove duplicated code from SR.cs (#36277) * Add RequiresUnreferencedCodeAttribute (#36674) This attribute is used by the linker to know which methods are unsafe to use when an application is trimmed. Fix #33862 * Revert unintentional alignment change for Vector256<T> (#36673) Addresses https://github.com/dotnet/runtime/pull/35864#issuecomment-629803015. In an earlier change I also added the `type.Instantiation[0].IsPrimitive` condition to the `IsVectorType` predicate. Revert that part as well and allow only primitive numeric types for HFA/HVA purpose. The same list of 10 types is recognized in `MethodTable::GetVectorSize` and `Compiler::getBaseTypeAndSizeOfSIMDType`. * Improve sort function in crossgen2 (#36676) - Increase memory locality substantially in composite images - Sort non-generic methods with module together - Sort generic methods near other generic methods with similar instantiations Json benchmark showed improvement from 237,827 RPS to 270,306 RPS, and reduced working set by about 10MB * Fixed Incorrectly named function arguments in TryGetPlatformSocketOption. (#36694) Fixes #36686. * Try using socket syscalls that accepts a single buffer to improve performance (#36371) * Try using socket syscalls that accepts a single buffer to improve performance * Remove ref from Receive calls * Prefix Pal methods invoking methods with Sys * Also use single-buffer syscalls for sync Socket Receive methods * Improve comment * PR feedback * Assert SocketAddress null instead of checking * fix handling of Ssl2 and enable disabled tests (#36098) * attempt to fix ssl2 * adjust length calculation * enable tests * fix ReadAsyncInternal * use PlatformDetection.SupportsSsl2 * feedback from review Co-authored-by: Tomas Weinfurt <furt@Shining.local> * Removed extra slash. (#36722) Remove extra / from path on runtimetests. * Contain non-candidate regOptional lclVars (#36601) * Adding a ci leg for Source build (#36141) * successfullsource build * adding a new source build leg. * remove yy * add default vlaue * addressing feedback * use boolean value * addind comment and other feedback * adding colon and removing unintentional change * adding default value of isSourceBUild * Remove TheadPool initialization volatile (#36697) * Remove TheadPool initialization volatile * Better ThreadPoolGlobals setup for Mono * Feedback * Move back to lambda * Remove duplicated tests from System.Text.Json.Tests (#36483) * Add build information to libraries helix jobs (#36713) * Add build information to libraries helix jobs * Remove BUILD_URI * Disable EventPipeProvider Context for runtime providers on EventPipe rundown * JIT: fix no return call accounting (#36719) The jit tracks the number of no return calls to determine if it should run throw helper merging and to decide if no return calls should tail called. The accounting is currently done when the calls are initially imported, so if code is duplicated (by say finally cloning the count may end up being an under-estimate. While not a correctness issue, it is better for the count to be accurate (or an over-estimate). So, update the count when cloning a no-return call. Closes #36584. * Remove Fedora29 from test matrix and add Fedora32 (#36716) * Code sharing between generic methods in member accessor (#36710) * Code sharing between generic methods in member accessor * Removed redudant value type check * Use latest compiler toolset from Arcade (#36741) * [mono] Fix iOS sample and use `dotnet publish` (#36745) Use `dotnet publish` with linker just like for Android. * [interp] Small cleanups (#36706) Co-authored-by: BrzVlad <BrzVlad@users.noreply.github.com> * [interp] Fix interp entry for methods with lots of arguments in llvmonly+interp mode. (#36678) Fixes https://github.com/mono/mono/issues/19801.  Co-authored-by: vargaz <vargaz@users.noreply.github.com> * Use a Brewfile for installing brew packages (#36747) This means we won't be upgrading existing packages on the system that we don't need for the build. Marks install-native-dependencies.sh as executable (+x) so we don't need to start it with `sh` in the build .yml Fixes https://github.com/dotnet/runtime/issues/36727 * Adding OSX support for System.DirectoryServices.Protocols (#36669) * Adding OSX support for System.DirectoryServices.Protocols * Addressing PR Feedback * Fixing issue in netfx builds where local member is assigned but never used * Optimize call indirect for R2R, Arm and Arm64 scenarios (#35675) * Use a different approach to optimize the indirect calls for R2R During lowering, don't create a controlExpr for indirect call. Instead use temp register for such calls and during codegen, load the indirect from x11 into that temp register before calling the address in that temp register. * Add FullAOT mode for Simulator to AppleAppBuilder (#36759) * Add managed array type support for EventPipe (#36242) Add support for emitting an event with an arbitrary number of arguments over EventPipe. * Remove overwriting RuntimeIdentifier in runtime.depproj for wasm/ios/android (#36757) It shouldn't be needed anymore since we have real assets for these targets now. * Cache Uri.IdnHost and Uri.PathAndQuery (#36460) * Cache Uri.IdnHost and Uri.PathAndQuery * Continue caching DnsSafeHost * Move PathAndQuery cache from MoreInfo to UriInfo * Move DnsSafeHost and IdnHost logic to IdnHost getter * Add more Host tests * [interp] Add separate opcode for pop vt (#36760) Before this change, pop-ing a vt from the stack was done in 2 opcodes, a MINT_POP (decrementing the stack) and the weird MINT_VTRESULT (decrementing the vtstack). However, optimizations could have removed both the initial loading of the value type on the stack as well as the MINT_POP, leaving MINT_VTRESULT underflowing the vtstack. Fix this and cleanup the code by pop-ing a value type from the stack in a single instruction. Co-authored-by: BrzVlad <BrzVlad@users.noreply.github.com> * [master] Update dependencies from mono/linker dotnet/llvm-project dotnet/xharness (#36764) * Update dependencies from https://github.com/dotnet/llvm-project build 20200518.2 runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk From Version 9.0.1-alpha.1.20262.1 -> To Version 9.0.1-alpha.1.20268.2 * Update dependencies from https://github.com/mono/linker build 20200519.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20268.5 -> To Version 5.0.0-preview.3.20269.1 * Update dependencies from https://github.com/dotnet/xharness build 20200520.1 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20265.8 -> To Version 1.0.0-prerelease.20270.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Support !JitDoOldStructRetyping on other platforms. (#35943) * Add more test cases. * Initialize `ReturnTypeDesc` when we keep struct types. * Add a few const modifiers. * Additional checks in `LowerRet` * Support `return double(cnst int)`. * Optimize `LowerRetStruct`: no need for bitcast when read from memory. * Prepare `LowerNode` for store local and local field to multireg. * Compile the new methods with FEATURE_MULTIREG_RET. * Improve `LowerRetStructLclVar`. Don't use bitcast if the source is in memory or has the same type. * Extract `LowerStoreLocCommon`. * Support 3, 5. 6, 7 bytes structs in `LowerCallStruct`. Move call handling to the users. * Disable `JitDoOldStructRetyping` for x86 and x64. Windows x64 was supported in a previous PR, this adds x86 (Windows and Linux) and x64 Unix. * Fix suggestions. * Disable by default for the merge. * [mono] Implement AsAny marshalling for simple arrays (#35686) * [mono] Implement AsAny marshalling for simple arrays Simple here means of rank 1, and with a blittable value type as element type. * Also implement char arrays * Do not allow in-params to be marshalled this way, as the callee would have write-ability that is not allowed by in-params * JIT: widen type of constant or special static fields (#36769) Since we're pushing the value of these fields we need to widen the value to a stack type. Addresses issues seen in #36592. * Optimize ToVector128, ToVector128Unsafe and Vector128.GetLower() (#36732) * Optimize ToVector128 , ToVector128Unsafe and Vector128.GetLower() * Fix evaluate paths bug when multiple include paths are specified (#36780) * Fix signing due to subset projects rename (#36791) * Adding a case requiring RuntimeJit (#36772) Handle CORINFO_HELP_THROW_NOT_IMPLEMENTED * Fix non-ascii absolute file path handling in Uri (#36429) * Fix non-ascii absolute file path Uri handling * Fix Unix tests * Revert unrelated style change * Add asserts to FilePathHandlesNonAscii to test AbsoluteUri roundtrips properties * Update public GC.AddMemoryPressure() and GC.RemoveMemoryPressure() (#36767) * Update public GC.AddMemoryPressure() and GC.RemoveMemoryPressure() to use newer algorithm. * Remove old AddMemoryPressure/RemoveMemoryPressure impls * Update test. * Removing duplicate tests from coreclr directory. * Fix two issues with hosting on SunOS (#36527) Fix two issues with hosting on SunOS: * Compile corehost without use-cxa-atexit with gcc * Disable RapidJson's 48-bit optimization on SunOS * Compile without [`use-cxa-atexit`](https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Dialect-Options.html) in case of gcc * libc tries to invoke `pthread_cond_destroy()` in `dlclose()`ed fxr library in the consumer code. PR disables this feature. * Backtrace available at http://sprunge.us/bx4dlk. This happens _after_ the main() exits. * Disable RapidJson's 48-bit pointer optimization on SunOS. * This optimization relies on zeros in higher 16-bits, whereas SunOS has 1s. More details at https://github.com/Tencent/rapidjson/issues/1596. * The impact here was that `runtimeOptions` key available in `hwapp.runtimeconfig.json` was not located by RapidJson's `FindMember()` API and we were getting `false` from: https://github.com/dotnet/runtime/blob/78b303df8fbb242985d049a277d0d199cafd51b5/src/installer/corehost/cli/runtime_config.cpp#L416-L422 Contributes to: #34944. * Add R2R testing on Windows arm/arm64 (#36793) * Add JIT EventCounters (#36489) * Add JIT counters * Some renames * fix build * change return type of GetMethodsJittedCount to int * remove ifdef * Fix * CR feedback * More CR feedback * More CR Feedback * more code review feedback * Fix build error and add displayunit to IL bytes jitted counter * Fix missing symbol issues in SunOS build (#36455) * Fix missing symbol issues in SunOS build * Fix SIGSEGV due to invalid free() * Fixing tests on android. (#36788) The linker was removing the icu shim functions even exporting them. The unique solution that we found until now is to force the linking using the -u flag. This is a temporary fix until we don't implement QCalls. Fixes https://github.com/dotnet/runtime/issues/36685 * Add autoconf, automake and libtool as dependencies (#36475) * Add autoconf, automake and libtool as dependencies * Update instructions * Handle multi-reg copies/reloads (#36802) Fix #36619 * Adding check to bypass decoding pdbs that have no sequence points. (#36771) Unity has a pdb with zero sequence points somehow. If a user added a reference to the library and then attempted to set a managed breakpoint mono would hit an assert and crash when it would attempt to decode this pdb. Call stack would look like: ``` > mono-2.0-boehm.dll!mono_ppdb_get_seq_points(_MonoDebugMethodInfo * minfo, char * * source_file, _GPtrArray * * source_file_list, int * * source_files, MonoSymSeqPoint * * seq_points, int * n_seq_points) Line 508 C mono-2.0-boehm.dll!mono_debug_get_seq_points(_MonoDebugMethodInfo * minfo, char * * source_file, _GPtrArray * * source_file_list, int * * source_files, MonoSymSeqPoint * * seq_points, int * n_seq_points) Line 1093 C mono-2.0-boehm.dll!get_source_files_for_type(_MonoClass * klass) Line 6920 C mono-2.0-boehm.dll!get_types_for_source_file(void * key, void * value, void * user_data) Line 7003 C mono-2.0-boehm.dll!monoeg_g_hash_table_foreach(_GHashTable * hash, void(*)(void *, void *, void *) func, void * user_data) Line 364 C mono-2.0-boehm.dll!mono_de_foreach_domain(void(*)(void *, void *, void *) func, void * user_data) Line 95 C mono-2.0-boehm.dll!vm_commands(int command, int id, unsigned char * p, unsigned char * end, Buffer * buf) Line 7341 C mono-2.0-boehm.dll!debugger_thread(void * arg) Line 10323 C mono-2.0-boehm.dll!start_wrapper_internal(StartInfo * start_info, unsigned __int64 * stack_ptr) Line 1241 C mono-2.0-boehm.dll!start_wrapper(void * data) Line 1315 C kernel32.dll!00007ffc12017bd4() Unknown ntdll.dll!00007ffc121ece51() Unknown ``` Adding a check to ignore pdbs like this prevents the crash and debugging can continue normally. Related unity issue: https://issuetracker.unity3d.com/issues/macos-editor-crashes-on-mono-log-write-logfile-when-attaching-a-debugger-and-then-setting-a-breakpoint Co-authored-by: UnityAlex <UnityAlex@users.noreply.github.com> * Allow building Browser/wasm config on Windows (#36816) Seems to work fine for the `Mono.CoreLib` subset, which is all we care about in the IL Linker-land. * Enable passing coreclr tests (#36702) * Updating the HWIntrinsics to support dropping unnecessary casts (#36512) * Updating the HWIntrinsics to support dropping unnecessary casts * Applying formatting patch * Only drop the cast operation if the source is greater than or equal to the expected size * Minor documentation improvements (#36834) * [Arm64] ASIMD Shift instructions (#36552) * sqrshl * sqrshrn * sqrshrn2 * sqrshrun * sqrshrun2 * sqshl * sqshlu * sqshrn * sqshrn2 * sqshrun * sqshrun2 * srshl * sshl * uqrshl * uqrshrn * uqrshrn2 * uqshl * uqshrn * uqshrn2 * urshl * ushl * Stop emitting weird intermediate folder artifacts/tests/artifacts (#36833) The logic in dir.common.props for relativizing test directories doesn't work well for the generated XUnit wrapper csproj files as these are generated under artifacts\tests. Relativizing this directory against src\coreclr\tests\src ended up with a weird sequence that ended up duplicating the tests/artifacts folder level. I have fixed this by explicitly passing the root directory for relativization for the XUnit wrapper projects. Thanks Tomas Fixes: #1655 * Adding A log presence check before trying to delete (#36779) * checking if log is present before deleteing * Update src/libraries/System.Diagnostics.EventLog/tests/EventLogTests/EventLogSourceCreationTests.cs Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com> Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com> * Move app local ICU string length validation to native shim (#36735) * Move app local string length validation to native shim * Fix build * JIT: don't call back into morph from fgRemoveConditionalJump (#36792) This can get called before morph and run into issues where morph specific data is not yet initialized. The remorphing wasn't going to do much other than extract side effects and rethread the statement ordering. So just do these latter bits. Closes #36468. * [wasm][debugger] Allow invoking property getters for objects (#35934) Co-authored-by: radical <radical@users.noreply.github.com> * Remove Microsoft.DotNet.PlatformAbstractions (#36707) * Remove Microsoft.DotNet.PlatformAbstractions This library overlapped with other System APIs and is now obsolete. For API replacement: ApplicationEnvironment.ApplicationBasePath => AppContext.BaseDirectory HashCodeCombiner => System.HashCode RuntimeEnvironment.GetRuntimeIdentifier() => RuntimeInformation.RuntimeIdentifier RuntimeEnvironment.OperatingSystemPlatform => RuntimeInformation.IsOSPlatform(OSPlatform) RuntimeEnvironment.RuntimeArchitecture => RuntimeInformation.ProcessArchitecture RuntimeEnvironment.OperatingSystem => RuntimeInformation.OSDescription RuntimeEnvironment.OperatingSystemVersion => RuntimeInformation.OSDescription / Environment.OSVersion.Version Fix #3470 * Porting more of the SIMD intrinsics to be implemented as HWIntrinsics (#36579) * Porting Ceiling and Floor to use SimdAsHWIntrinsic * Porting SquareRoot to use SimdAsHWIntrinsic * Porting ConditionalSelect to use SimdAsHWIntrinsic * Porting get_AllBitsSet, get_Count, and get_Zero to use SimdAsHWIntrinsic * Porting op_Explicit to use SimdAsHWIntrinsic * Changing Vector2/3/4 and Vector<T>.Equals to forward to operator == * Removing SIMDIntrinsicAbs * Removing SIMDIntrinsicMax and SIMDIntrinsicMin * Removing SIMDIntrinsicCeil and SIMDIntrinsicFloor * Porting op_Equality and op_Inequality to use SimdAsHWIntrinsic * Removing SIMDIntrinsicSqrt * Removing SIMDIntrinsicSelect * Removing SIMDIntrinsicBitwiseAndNot and SIMDIntrinsicBitwiseXor * Removing SIMDIntrinsicGreaterThanOrEqual and SIMDIntrinsicLessThanOrEqual * Removing SIMDIntrinsicGreaterThan and SIMDIntrinsicLessThan * Removing SIMDIntrinsicInstEquals * Removing SIMDIntrinsicOpEquality and SIMDIntrinsicOpInEquality * Porting this.Equals to use SimdAsHWIntrinsic * Don't handle IEquatable`1.Equals via SimdAsHWIntrinsic * Applying formatting patch * Account for op2 being able to precede op1 in the LIR order * Ensure SimdAsHWIntrinsic with 0 args are properly handled * Fixup the arm64 LowerHWIntrinsicCmpOp implementation to use LowerNodeCC * Fixing an assert in the NotSupported HWIntrinsic tests * Use socketpair to implement Process.Start redirection (#34861) * Use socketpair to implement Process.Start redirection Today on Unix, we create an anonymous pipe via pipe/pipe2 to be used for stdin/stdout/stderr on processes created by Process.Start. We then wrap the resulting file descriptors with FileStreams to hand out via Process.StandardInput/Output/Error. This has a few issues, however. Any async operations on the resulting stream (or wrapping stream reader) will actually be async-over-sync, and that in turn means that a) any async read will end up blocking a thread pool thread until it's satisified, and b) the operation isn't cancelable. The implications of (b) are obvious, and the problem with (a) is that code which launches a bunch of processes and uses BeginOutput/ErrorReadLine or the like will end up blocking a bunch of thread pool threads. This change replaces the pipe/pipe2 calls with socketpair calls, and instead of wrapping the resulting file descriptors with FileStream, wraps them in Sockets and NetworkStreams. This gives us the full capabilities of the networking stack, which fully supports asynchronous and cancelable reads and writes. * Try to fix macOS failures with socketpair * Extract shared functionality to a TryRemoveCastIfPresent method (#36837) * Added CreateDelegate<T> overloads to MethodInfo (#36680) * Added generic overloads for MethodInfo.CreateDelegate<>() * Added full support of CreateDelegate<T> and the necessary tests. * Refactored to apply the new CreateDelegate overloads as necessary throughout runtime. * Reverted Linq.Expressions changes and fully qualified Delegate to System.Delegate * [mono] Fix MonoRunner.java - assets were not copied correctly (#36824) * Move where wasm binaries are generated (#36781) This change shifts producing wasm binaries to the native dir so that they'll come over to the installer on CI. * Strip the ILLinkTrim.xml file from System.Security assemblies (#36864) Contributes to #35199 * Consider armel arch (#36872) * Fix a summary typo in ConsoleLoggerOptions.cs (#36870) * Remove x86 OSX code (#36874) This removes x86 OSX code that was never used for .NET Core. * Ensure Image.Save can handle non readable / seekable Streams (#36805) * Ensure Image.Save can handle non readable / seekable Streams * Address feedback * Fix Image.Save on Unix for write only non-seekable stream * Remove test which uses bogus handle value * Use linux-musl runtime on Alpine Helix workers (#36827) * Use linux-musl runtime on Alpine Helix workers * Add OS Subgroup to send-to-helix-step * Pass in osSubgroup Also, capitalize it in line with the rest of the repo * Use __TargetOS which understands linux_musl for rid generation I don't fully understand why we use `TargetOS` in `helixpublishwitharcade.proj` but set `__TargetOS` in the calling `send-to-helix-step.yml`. * Fix wasm build when runtime configuration and configuration are different (#36855) * Fix wasm build when runtime configuration and configuration are different * PR Feedback * Fix CoreCLR initialization on Illumos-based distros (#36808) * [Arm64] Polynomial Multiply Long Intrinsics (#36853) Implements PolynomialMultiplyWideningLower and PolynomialMultiplyWideningUpper * Add NotNullIfNotNull, MemberNotNull attributes to doc (#36856) * [Arm64] Follow-up on renaming intrinsics to Narrowing for consistency with Widening (#36857) * Rename AddHighNarrow to AddHighNarrowing * Rename AddRoundedHighNarrow to AddRoundedHighNarrowing * Rename ExtractAndNarrowHigh to ExtractNarrowingUpper * Rename ExtractAndNarrowLow to ExtractNarrowingLower * Rename SubtractHighNarrow to SubtractHighNarrowing * Rename SubtractRoundedHighNarrow to SubtractRoundedHighNarrowing * Renamed pInitRegZeroed to pInitRegModified, initRegZeroed to initRegM… (#36321) * Add MONO_LOADER_LIBRARY_NAME define (#36835) * Fixing make dist after implementation of static ICU Shim. (#36795) Fixing make dist after implementation of static ICU Shim. Co-authored-by: thaystg <thaystg@users.noreply.github.com> * Improve obsoletion message for IgnoreNullValues (#36886) * Disable System.Numerics.Tests.Vector3Tests.Vector3InequalityTest on arm64 (#36906) * Pipelines performance tweaks (#36815) * Pipelines performance tweaks * Packaging * Don't use Unsafe for array cast * Remove Unsafe use * Remove System.Globalization.Native shim from shared framework (#36904) * Remove System.Globalization.Native shim from shared framework * Add System.Globalization.Native sources to mono paths in pipeline * Remove dead Exception handling code in Uri (#36865) * Make Vector256 dependent on AVX instruction set (#36895) * [Arm64] ASIMD Shift Intrinsics (#36830) * ShiftArithmetic * ShiftArithmeticRounded * ShiftArithmeticRoundedSaturate * ShiftArithmeticRoundedSaturateScalar * ShiftArithmeticRoundedScalar * ShiftArithmeticSaturate * ShiftArithmeticSaturateScalar * ShiftArithmeticScalar * ShiftLeftLogical * ShiftLeftLogicalSaturate * ShiftLeftLogicalSaturateScalar * ShiftLeftLogicalSaturateUnsigned * ShiftLeftLogicalSaturateUnsignedScalar * ShiftLeftLogicalScalar * ShiftLeftLogicalWideningLower * ShiftLeftLogicalWideningUpper * ShiftLogical * ShiftLogicalRounded * ShiftLogicalRoundedSaturate * ShiftLogicalRoundedSaturateScalar * ShiftLogicalRoundedScalar * ShiftLogicalSaturate * ShiftLogicalSaturateScalar * ShiftLogicalScalar * ShiftRightArithmetic * ShiftRightArithmeticAdd * ShiftRightArithmeticAddScalar * ShiftRightArithmeticNarrowingSaturateLower * ShiftRightArithmeticNarrowingSaturateUnsignedLower * ShiftRightArithmeticNarrowingSaturateUnsignedUpper * ShiftRightArithmeticNarrowingSaturateUpper * ShiftRightArithmeticRounded * ShiftRightArithmeticRoundedAdd * ShiftRightArithmeticRoundedAddScalar * ShiftRightArithmeticRoundedNarrowingSaturateLower * ShiftRightArithmeticRoundedNarrowingSaturateUnsignedLower * ShiftRightArithmeticRoundedNarrowingSaturateUnsignedUpper * ShiftRightArithmeticRoundedNarrowingSaturateUpper * ShiftRightArithmeticRoundedScalar * ShiftRightArithmeticScalar * ShiftRightLogical * ShiftRightLogicalAdd * ShiftRightLogicalAddScalar * ShiftRightLogicalNarrowingLower * ShiftRightLogicalNarrowingSaturateLower * ShiftRightLogicalNarrowingSaturateUpper * ShiftRightLogicalNarrowingUpper * ShiftRightLogicalRounded * ShiftRightLogicalRoundedAdd * ShiftRightLogicalRoundedAddScalar * ShiftRightLogicalRoundedNarrowingLower * ShiftRightLogicalRoundedNarrowingSaturateLower * ShiftRightLogicalRoundedNarrowingSaturateUpper * ShiftRightLogicalRoundedNarrowingUpper * ShiftRightLogicalRoundedScalar * ShiftRightLogicalScalar * SignExtendWideningLower * SignExtendWideningUpper * ZeroExtendWideningLower * ZeroExtendWideningUpper * ShiftArithmeticRoundedSaturateScalar * ShiftArithmeticSaturateScalar * ShiftLeftLogicalSaturateScalar * ShiftLeftLogicalSaturateUnsignedScalar * ShiftLogicalRoundedSaturateScalar * ShiftLogicalSaturateScalar * ShiftRightArithmeticNarrowingSaturateScalar * ShiftRightArithmeticNarrowingSaturateUnsignedScalar * ShiftRightArithmeticRoundedNarrowingSaturateScalar * ShiftRightArithmeticRoundedNarrowingSaturateUnsignedScalar * ShiftRightLogicalNarrowingSaturateScalar * ShiftRightLogicalRoundedNarrowingSaturateScalar * BigMul with 64-bit operands (#35975) * Unsigned 128-bit BigMul * Signed 128-bit BigMul * Added explanation of alg and reference to it * Removed else branch Co-authored-by: Adeel Mujahid <adeelbm@outlook.com> * Optimize BitOperations uses in CoreLib (#35650) * make Log2 branch-free * convert LeadingZeroCount uses to Log2 * switch Bmi1.TrailingZeroCount calls to BitOperations * [jit] Return the compiled method and an unbox trampoline from ldvirtftn when calling ldvirtftn on a valuetype method. (#36736) Fixes https://github.com/dotnet/runtime/issues/34379.  Co-authored-by: vargaz <vargaz@users.noreply.github.com> * Revert "Libunwind1.5rc2 (#36027)" (#36909) This reverts commit 8c6c7655bb7abc5c4ce769923aa5bea1daee6bd3. * Remove PreserveDependency from SR class (#36358) * Use TARGET_64BIT instead of set of TARGET_<arch> definitions for CLRConfig::EXTERNAL_CORECLR_PROFILER_PATH_64 and CLRConfig::EXTERNAL_CORECLR_PROFILER_PATH_32 checking. (#36924) * Follow ups for Math.BigMul (#36920) - Use more efficient and compact uint cast instead of bit masks where possible - Improved test coverage - Update comment in FastMod - Optimize BigMul for 32-bit platforms * Use BinaryPrimitives where possible. (#36881) * Fix exception message to indicate Cofactor is required. (#36821) * Disable GC/API/GC/AddThresholdTest on ARM64 (#36940) Issue #36850 * Add comments per code review feedback * Fixing emitAnyConst and related functions to better handle alignment (#36432) * Fixing emitAnyConst and related functions to better handle alignment * Don't use small alignment where it might not be supported * Applying formatting patch * Adding some comments explaining how emitAnyConst works * Fix output for user strings in R2RDump (#36935) Quote user strings and escape control characters, unpaired surrogates, and other unsafe characters. * [master] Update dependencies from mono/linker Microsoft/vstest dotnet/xharness (#36811) * Update dependencies from https://github.com/mono/linker build 20200520.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20269.1 -> To Version 5.0.0-preview.3.20270.1 * Update dependencies from https://github.com/microsoft/vstest build 20200521-01 Microsoft.NET.Test.Sdk From Version 16.7.0-preview-20200518-01 -> To Version 16.7.0-preview-20200521-01 * Update dependencies from https://github.com/dotnet/xharness build 20200520.2 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20270.1 -> To Version 1.0.0-prerelease.20270.2 * Update dependencies from https://github.com/mono/linker build 20200521.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20269.1 -> To Version 5.0.0-preview.3.20271.1 * Update dependencies from https://github.com/dotnet/xharness build 20200521.3 Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20270.1 -> To Version 1.0.0-prerelease.20271.3 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Remove volatile on field in ConcurrentDictionary.Tables (#36976) * configure.ac: check for sockaddr_in6 on Win32 (#36742) HAVE_STRUCT_SOCKADDR_IN6 needs to be defined to enable IPv6 features in mono/metadata/w32socket.c. Fixes TcpListener issues in wine-mono. Co-authored-by: w-flo <w-flo@users.noreply.github.com> * Update dependencies from https://github.com/dotnet/xharness build 20200525.1 (#36982) Microsoft.DotNet.XHarness.Tests.Runners From Version 1.0.0-prerelease.20271.3 -> To Version 1.0.0-prerelease.20275.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Timeout support for unprivileged Ping on Linux/BSD (#35461) * Added support of timeout when unprivileged ping is used * Fixed test according with new timeout parameter * Fixes code style issues * Fixed command-line flag for ping6 on BSD * Separated timeout test * Removed trailing whitespace * Workaround for OSX ping6 * Removed static local method * Suppress Ping stdout * Fixed variable name * Added test for zero timeout * Handle zero timeout correctly * Fix accumulating the offset in PemReader The preebEndIndex offset was a total value in to the PEM's contents, not a moving offset. This would cause it to advance past the contents of the PE…

swaroop-sridhar requested review from davidwrighton, janvorli, vitek-karas and jkotas September 4, 2019 08:46

vitek-karas reviewed Sep 4, 2019

View reviewed changes

janvorli reviewed Sep 4, 2019

View reviewed changes

jkotas reviewed Sep 4, 2019

View reviewed changes

swaroop-sridhar force-pushed the pass branch from e912d90 to bb6e54c Compare September 5, 2019 02:25

vitek-karas reviewed Sep 5, 2019

View reviewed changes

swaroop-sridhar force-pushed the pass branch from bb6e54c to 1d022e0 Compare September 11, 2019 10:34

swaroop-sridhar force-pushed the pass branch from 1d022e0 to b7601e5 Compare September 11, 2019 10:43

vitek-karas suggested changes Sep 11, 2019

View reviewed changes

vitek-karas approved these changes Sep 11, 2019

View reviewed changes

swaroop-sridhar and others added 5 commits September 16, 2019 01:24

Rename BundleLoc to BundleFileLocation

2e19196

Also fix an error during initialization of Bundle() structure.

Add a few type-casts to fix Windows failures

c7ec374

Fix a comment

c3f3014

swaroop-sridhar force-pushed the pass branch from e092ef6 to c3f3014 Compare September 16, 2019 08:25

Fix a GetLastError() bug

ca8fb68

Fix a bug that caused errno check after open() to be re-written

swaroop-sridhar force-pushed the pass branch from f6ffb37 to ca8fb68 Compare September 16, 2019 21:55

janvorli reviewed Sep 16, 2019

View reviewed changes

swaroop-sridhar merged commit 81c08e8 into dotnet:single-exe Sep 17, 2019

swaroop-sridhar added a commit to swaroop-sridhar/coreclr that referenced this pull request Sep 17, 2019

[Single-exe] A few small updates

6a4bf97

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in dotnet#26504

swaroop-sridhar added a commit to swaroop-sridhar/coreclr that referenced this pull request Sep 17, 2019

[Single-exe] A few small updates

55efcb3

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in dotnet#26504

swaroop-sridhar mentioned this pull request Sep 17, 2019

[Single-Exe] A few small updates #26754

Merged

swaroop-sridhar added a commit that referenced this pull request Sep 17, 2019

[Single-exe] A few small updates (#26754)

d3fcfb2

Make a few small improvements to the single-file bundle handling code. Implements changes suggested by @janvorli in #26504

swaroop-sridhar mentioned this pull request Jan 31, 2020

Rethink Host Interface to pass properties to CoreCLR dotnet/runtime#13432

Open

swaroop-sridhar mentioned this pull request May 7, 2020

Single-File: Run from Bundle dotnet/runtime#36052

Merged

		@@ -1,4 +1,5 @@
		include_directories(inc)
		include_directories(../../inc)

		ReleaseHolder<Assembly> pSystemAssembly;
		StackSString sCoreLib;

[Single-Exe] Run from bundle #26504

[Single-Exe] Run from bundle #26504

Conversation

swaroop-sridhar commented Sep 4, 2019 • edited Loading

Changes

Testing

Performance

Console Hello World

Startup of WebApiTemplate:

Steps to build single-exe app

vitek-karas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swaroop-sridhar commented Sep 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swaroop-sridhar commented Sep 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swaroop-sridhar commented Sep 11, 2019

swaroop-sridhar commented Sep 11, 2019

vitek-karas commented Sep 11, 2019

swaroop-sridhar commented Sep 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swaroop-sridhar commented Sep 17, 2019

swaroop-sridhar commented Sep 4, 2019 •

edited

Loading

Steps to build single-exe `app`

swaroop-sridhar commented Sep 11, 2019 •

edited

Loading