-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add .mvid section to PE and remove dependency on MetadataReader #19133
Changes from 3 commits
53a66be
67c31d3
51a3c3f
4de0d32
ec9b2d4
b1ac2aa
4c02a2d
7df9479
786dbe7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,7 +32,7 @@ Two mutually exclusive command-line parameters will be added to `csc.exe` and `v | |
- `/refout` | ||
- `/refonly` | ||
|
||
The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). | ||
The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). The filename for the ref assembly should generally match that of the primary assembly, but it can be in a different folder. | ||
|
||
The `/refonly` parameter is a flag that indicates that a ref assembly should be output instead of an implementation assembly. | ||
The `/refonly` parameter is not allowed together with the `/refout` parameter, as it doesn't make sense to have both the primary and secondary outputs be ref assemblies. Also, the `/refonly` parameter silently disables outputting PDBs, as ref assemblies cannot be executed. | ||
|
@@ -47,13 +47,14 @@ The `CoreCompile` target will support a new output, called `IntermediateRefAssem | |
The `Csc` task will support a new output, called `OutputRefAssembly`, which parallels the existing `OutputAssembly`. | ||
Both of those basically map to the `/refout` command-line parameter. | ||
|
||
An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match, then the destination file is left untouched. | ||
An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match (by comparing their MVIDs, see details below), then the destination file is left untouched. | ||
|
||
### CodeAnalysis APIs | ||
It is already possible to produce metadata-only assemblies by using `EmitOptions.EmitMetadataOnly`, which is used in IDE scenarios with cross-language dependencies. | ||
The compiler will be updated to honour the `EmitOptions.IncludePrivateMembers` flag as well. When combined with `EmitMetadataOnly` or a `metadataPeStream` in `Emit`, a ref assembly will be produced. | ||
The diagnostic check for emitting methods lacking a body (`void M();`) will be moved from declaration diagnostics to regular diagnostics, so that code will successfully emit with `EmitMetadataOnly`. | ||
The diagnostic check for emitting methods lacking a body (`void M();`) will be filtered from declaration diagnostics, so that code will successfully emit with `EmitMetadataOnly`. | ||
Later on, the `EmitOptions.TolerateErrors` flag will allow emitting error types as well. | ||
`Emit` is also modified to produce a new PE section called ".mvid" containing a copy of the MVID, when producing ref assemblies. This makes it easy for `CopyRefAssembly` to extract and compare MVIDs from ref assemblies. | ||
|
||
Going back to the 4 driving scenarios: | ||
1. For a regular compilation, `EmitMetadataOnly` is left to `false` and no `metadataPeStream` is passed into `Emit`. | ||
|
@@ -67,10 +68,10 @@ As mentioned above, there may be further refinements after C# 7.1: | |
- produce ref assemblies even when there are errors outside method bodies (emitting error types when `EmitOptions.TolerateErrors` is set) | ||
|
||
## Open questions | ||
- should explicit method implementations be included in ref assemblies? | ||
- should explicit method implementations be included in ref assemblies? (answer: no. The interfaces that are declared as implemented are what matter to consuming compilations) | ||
- Non-public attributes on public APIs (emit attribute based on accessibility rule) | ||
- ref assemblies and NoPia | ||
- `/refout` and `/addmodule`, should we disallow this combination? | ||
- `/refout` and `/addmodule`, should we disallow this combination? (answer: yes. This will not be supported in C# 7.1) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are these still open questions? If not, consider moving to another section in the document. #Resolved |
||
|
||
## Related issues | ||
- Produce ref assemblies from command-line and msbuild (https://github.com/dotnet/roslyn/issues/2184) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -262,9 +262,11 @@ internal static void Main() | |
|
||
VerifyEntryPoint(output, expectZero: false); | ||
VerifyMethods(output, new[] { "void C.Main()", "C..ctor()" }); | ||
VerifyMvid(output, hasMvidSection: false); | ||
|
||
VerifyEntryPoint(metadataOutput, expectZero: true); | ||
VerifyMethods(metadataOutput, new[] { "C..ctor()" }); | ||
VerifyMvid(metadataOutput, hasMvidSection: true); | ||
} | ||
|
||
void VerifyEntryPoint(MemoryStream stream, bool expectZero) | ||
|
@@ -275,6 +277,34 @@ void VerifyEntryPoint(MemoryStream stream, bool expectZero) | |
} | ||
} | ||
|
||
/// <summary> | ||
/// Extract the MVID using two different methods (PEReader and MvidReader) and compare them. | ||
/// We only expect an .mvid section in ref assemblies. | ||
/// </summary> | ||
private void VerifyMvid(MemoryStream stream, bool hasMvidSection) | ||
{ | ||
Guid mvidFromModuleDefinition; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Declaration can be moved into |
||
stream.Position = 0; | ||
using (var reader = new PEReader(stream)) | ||
{ | ||
var metadataReader = reader.GetMetadataReader(); | ||
mvidFromModuleDefinition = metadataReader.GetGuid(metadataReader.GetModuleDefinition().Mvid); | ||
|
||
stream.Position = 0; | ||
var mvidFromMvidReader = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(stream); | ||
|
||
if (hasMvidSection) | ||
{ | ||
Assert.Equal(mvidFromModuleDefinition, mvidFromMvidReader); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we checking the mvids are non-default? #Resolved |
||
} | ||
else | ||
{ | ||
Assert.NotEqual(Guid.Empty, mvidFromModuleDefinition); | ||
Assert.Equal(Guid.Empty, mvidFromMvidReader); | ||
} | ||
} | ||
} | ||
|
||
[Fact] | ||
public void EmitRefAssembly_PrivatePropertySetter() | ||
{ | ||
|
@@ -296,6 +326,8 @@ public class C | |
VerifyMethods(output, new[] { "System.Int32 C.<PrivateSetter>k__BackingField", "System.Int32 C.PrivateSetter.get", "void C.PrivateSetter.set", | ||
"C..ctor()", "System.Int32 C.PrivateSetter { get; private set; }" }); | ||
VerifyMethods(metadataOutput, new[] { "System.Int32 C.PrivateSetter.get", "C..ctor()", "System.Int32 C.PrivateSetter { get; }" }); | ||
VerifyMvid(output, hasMvidSection: false); | ||
VerifyMvid(metadataOutput, hasMvidSection: true); | ||
} | ||
} | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
// Copyright (c) Microsoft. All Rights Reserved. Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. | ||
|
||
using System; | ||
using System.Diagnostics; | ||
using System.IO; | ||
using System.Text; | ||
|
||
namespace Microsoft.CodeAnalysis.BuildTasks | ||
{ | ||
public sealed class MvidReader : BinaryReader | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are there callers that instantiate this type? If not, could this be a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll make the constructor private. |
||
{ | ||
public MvidReader(Stream stream) : base(stream) | ||
{ | ||
} | ||
|
||
public static Guid ReadAssemblyMvidOrEmpty(Stream stream) | ||
{ | ||
try | ||
{ | ||
var mvidReader = new MvidReader(stream); | ||
return mvidReader.TryFindMvid(); | ||
} | ||
catch (EndOfStreamException) | ||
{ | ||
} | ||
|
||
return Guid.Empty; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Feel like this is essentially hiding failure here. Consider using a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the exception handler still needed? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think so. If the ref assembly is truncated (there are section headers, but no actual sections), we risk reading past the end of the file. #Resolved |
||
} | ||
|
||
public Guid TryFindMvid() | ||
{ | ||
Guid empty = Guid.Empty; | ||
|
||
// DOS Header (64), DOS Stub (64), PE Signature (4), COFF Header (20), PE Header (224) | ||
if (BaseStream.Length < 64 + 64 + 4 + 20 + 224) | ||
{ | ||
return empty; | ||
} | ||
|
||
// DOS Header: PE (2) | ||
if (ReadUInt16() != 0x5a4d) | ||
{ | ||
return empty; | ||
} | ||
|
||
// DOS Header: Start (58) | ||
Skip(58); | ||
|
||
// DOS Header: Address of PE Signature | ||
MoveTo(ReadUInt32()); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It looks like we are moving to an unknown position without verifying if the destination makes sense. #Resolved There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. That's why this logic is wrapped in a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
But this method is public and it doesn't catch any exceptions. #Resolved There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer to avoid first-chance exceptions if at all possible. Just read the value and check against the stream length. #Resolved |
||
|
||
// PE Signature ('P' 'E' null null) | ||
if (ReadUInt32() != 0x00004550) | ||
{ | ||
return empty; | ||
} | ||
|
||
// COFF Header: Machine (2) | ||
Skip(2); | ||
|
||
// COFF Header: NumberOfSections (2) | ||
ushort sections = ReadUInt16(); | ||
|
||
// COFF Header: TimeDateStamp (4), PointerToSymbolTable (4), NumberOfSymbols (4) | ||
Skip(12); | ||
|
||
// COFF Header: OptionalHeaderSize (2) | ||
ushort optionalHeaderSize = ReadUInt16(); | ||
|
||
// COFF Header: Characteristics (2) | ||
Skip(2); | ||
|
||
// Optional header | ||
Skip(optionalHeaderSize); | ||
|
||
// Section headers | ||
return FindMvidInSections(sections); | ||
} | ||
|
||
private Guid FindMvidInSections(ushort count) | ||
{ | ||
for (int i = 0; i < count; i++) | ||
{ | ||
// Section: Name (8) | ||
byte[] name = ReadBytes(8); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Where is this helper defined? #Resolved There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In |
||
if (name.Length == 8 && name[0] == '.' && | ||
name[1] == 'm' && name[2] == 'v' && name[3] == 'i' && name[4] == 'd') | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
{ | ||
// Section: VirtualSize (4) | ||
uint virtualSize = ReadUInt32(); | ||
|
||
// Section: VirtualAddress (4), SizeOfRawData (4) | ||
Skip(8); | ||
|
||
// Section: PointerToRawData (4) | ||
uint pointerToRawData = ReadUInt32(); | ||
|
||
// The .mvid section only stores a Guid | ||
Debug.Assert(virtualSize == 16); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think we should verify the size, rather than simply assert it. #Resolved |
||
|
||
BaseStream.Position = pointerToRawData; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Should we use Move() here? #Resolved |
||
byte[] guidBytes = new byte[16]; | ||
BaseStream.Read(guidBytes, 0, 16); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Not checking how many bytes were read. #Resolved |
||
|
||
return new Guid(guidBytes); | ||
} | ||
else | ||
{ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: no need for else since the if-statement returns #Resolved |
||
// Section: VirtualSize (4), VirtualAddress (4), SizeOfRawData (4), | ||
// PointerToRawData (4), PointerToRelocations (4), PointerToLineNumbers (4), | ||
// NumberOfRelocations (2), NumberOfLineNumbers (2), Characteristics (4) | ||
Skip(4 + 4 + 4 + 4 + 4 + 4 + 2 + 2 + 4); | ||
} | ||
} | ||
|
||
return Guid.Empty; | ||
} | ||
|
||
public void Skip(int bytes) | ||
{ | ||
BaseStream.Seek(bytes, SeekOrigin.Current); | ||
} | ||
|
||
public void MoveTo(uint position) | ||
{ | ||
BaseStream.Seek(position, SeekOrigin.Begin); | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These methods can be |
||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure if this is accurate. #Closed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gafter may want to comment. From our discussion, it was ok to strip explicit method implementations.
That said, there is still a PROTOTYPE marker in this branch for this (see
GetExplicitImplementationOverrides
).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then let's not answer the question for now and discuss the issue offline. #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reverted the note. Keeping open question on explicit method implementations.
In reply to: 114207367 [](ancestors = 114207367)