Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add .mvid section to PE and remove dependency on MetadataReader #19133

Merged
merged 9 commits into from
May 2, 2017
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/compilers/CSharp/CommandLine.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
| ---- | ---- |
| **OUTPUT FILES** |
| `/out:`*file* | Specify output file name (default: base name of file with main class or first file)
| `/refout:`*file* | Specify the reference assembly's output file name
| `/target:exe` | Build a console executable (default) (Short form: `/t:exe`)
| `/target:winexe` | Build a Windows executable (Short form: `/t:winexe` )
| `/target:library` | Build a library (Short form: `/t:library`)
Expand Down Expand Up @@ -32,6 +33,7 @@
| `/debug`:{`full`|`pdbonly`|`portable`} | Specify debugging type (`full` is default, and enables attaching a debugger to a running program. `portable` is a cross-platform format)
| `/optimize`{`+`|`-`} | Enable optimizations (Short form: `/o`)
| `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output
| **ERRORS AND WARNINGS**
| `/warnaserror`{`+`|`-`} | Report all warnings as errors
| `/warnaserror`{`+`|`-`}`:`*warn list* | Report specific warnings as errors
Expand Down
2 changes: 2 additions & 0 deletions docs/compilers/Visual Basic/CommandLine.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
| ---- | ---- |
| **OUTPUT FILE**
| `/out:`*file* | Specifies the output file name.
| `/refout:`*file* | Specify the reference assembly's output file name
| `/target:exe` | Create a console application (default). (Short form: `/t`)
| `/target:winexe` | Create a Windows application.
| `/target:library` | Create a library assembly.
Expand Down Expand Up @@ -34,6 +35,7 @@
| `/debug:portable` | Emit debugging information in the portable format.
| `/debug:pdbonly` | Emit PDB file only.
| `/deterministic` | Produce a deterministic assembly (including module version GUID and timestamp)
| `/refonly | Produce a reference assembly, instead of a full assembly, as the primary output
| **ERRORS AND WARNINGS**
| `/nowarn` | Disable all warnings.
| `/nowarn:`*number_list* | Disable a list of individual warnings.
Expand Down
11 changes: 6 additions & 5 deletions docs/features/refout.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Two mutually exclusive command-line parameters will be added to `csc.exe` and `v
- `/refout`
- `/refonly`

The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below).
The `/refout` parameter specifies a file path where the ref assembly should be output. This translates to `metadataPeStream` in the `Emit` API (see details below). The filename for the ref assembly should generally match that of the primary assembly, but it can be in a different folder.

The `/refonly` parameter is a flag that indicates that a ref assembly should be output instead of an implementation assembly.
The `/refonly` parameter is not allowed together with the `/refout` parameter, as it doesn't make sense to have both the primary and secondary outputs be ref assemblies. Also, the `/refonly` parameter silently disables outputting PDBs, as ref assemblies cannot be executed.
Expand All @@ -47,13 +47,14 @@ The `CoreCompile` target will support a new output, called `IntermediateRefAssem
The `Csc` task will support a new output, called `OutputRefAssembly`, which parallels the existing `OutputAssembly`.
Both of those basically map to the `/refout` command-line parameter.

An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match, then the destination file is left untouched.
An additional task, called `CopyRefAssembly`, will be provided along with the existing `Csc` task. It takes a `SourcePath` and a `DestinationPath` and generally copies the file from the source over to the destination. But if it can determine that the contents of those two files match (by comparing their MVIDs, see details below), then the destination file is left untouched.

### CodeAnalysis APIs
It is already possible to produce metadata-only assemblies by using `EmitOptions.EmitMetadataOnly`, which is used in IDE scenarios with cross-language dependencies.
The compiler will be updated to honour the `EmitOptions.IncludePrivateMembers` flag as well. When combined with `EmitMetadataOnly` or a `metadataPeStream` in `Emit`, a ref assembly will be produced.
The diagnostic check for emitting methods lacking a body (`void M();`) will be moved from declaration diagnostics to regular diagnostics, so that code will successfully emit with `EmitMetadataOnly`.
The diagnostic check for emitting methods lacking a body (`void M();`) will be filtered from declaration diagnostics, so that code will successfully emit with `EmitMetadataOnly`.
Later on, the `EmitOptions.TolerateErrors` flag will allow emitting error types as well.
`Emit` is also modified to produce a new PE section called ".mvid" containing a copy of the MVID, when producing ref assemblies. This makes it easy for `CopyRefAssembly` to extract and compare MVIDs from ref assemblies.

Going back to the 4 driving scenarios:
1. For a regular compilation, `EmitMetadataOnly` is left to `false` and no `metadataPeStream` is passed into `Emit`.
Expand All @@ -67,10 +68,10 @@ As mentioned above, there may be further refinements after C# 7.1:
- produce ref assemblies even when there are errors outside method bodies (emitting error types when `EmitOptions.TolerateErrors` is set)

## Open questions
- should explicit method implementations be included in ref assemblies?
- should explicit method implementations be included in ref assemblies? (answer: no. The interfaces that are declared as implemented are what matter to consuming compilations)
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(answer: no. The interfaces that are declared as implemented are what matter to consuming compilations) [](start = 72, length = 103)

I am not sure if this is accurate. #Closed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gafter may want to comment. From our discussion, it was ok to strip explicit method implementations.

That said, there is still a PROTOTYPE marker in this branch for this (see GetExplicitImplementationOverrides).

Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said, there is still a PROTOTYPE marker in this branch for this (see GetExplicitImplementationOverrides).

Then let's not answer the question for now and discuss the issue offline. #Resolved

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted the note. Keeping open question on explicit method implementations.


In reply to: 114207367 [](ancestors = 114207367)

- Non-public attributes on public APIs (emit attribute based on accessibility rule)
- ref assemblies and NoPia
- `/refout` and `/addmodule`, should we disallow this combination?
- `/refout` and `/addmodule`, should we disallow this combination? (answer: yes. This will not be supported in C# 7.1)
Copy link
Member

@cston cston May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these still open questions? If not, consider moving to another section in the document. #Resolved


## Related issues
- Produce ref assemblies from command-line and msbuild (https://github.com/dotnet/roslyn/issues/2184)
Expand Down
3 changes: 3 additions & 0 deletions src/Compilers/CSharp/Test/Emit/CSharpCompilerEmitTest.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|AnyCPU'" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|AnyCPU'" />
<ItemGroup>
<Compile Include="..\..\..\Core\MSBuildTask\MvidReader.cs">
<Link>Emit\MvidReader.cs</Link>
</Compile>
<Compile Include="Attributes\AttributeTests.cs" />
<Compile Include="Attributes\AttributeTests_Assembly.cs" />
<Compile Include="Attributes\AttributeTests_CallerInfoAttributes.cs" />
Expand Down
32 changes: 32 additions & 0 deletions src/Compilers/CSharp/Test/Emit/Emit/CompilationEmitTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -262,9 +262,11 @@ internal static void Main()

VerifyEntryPoint(output, expectZero: false);
VerifyMethods(output, new[] { "void C.Main()", "C..ctor()" });
VerifyMvid(output, hasMvidSection: false);

VerifyEntryPoint(metadataOutput, expectZero: true);
VerifyMethods(metadataOutput, new[] { "C..ctor()" });
VerifyMvid(metadataOutput, hasMvidSection: true);
}

void VerifyEntryPoint(MemoryStream stream, bool expectZero)
Expand All @@ -275,6 +277,34 @@ void VerifyEntryPoint(MemoryStream stream, bool expectZero)
}
}

/// <summary>
/// Extract the MVID using two different methods (PEReader and MvidReader) and compare them.
/// We only expect an .mvid section in ref assemblies.
/// </summary>
private void VerifyMvid(MemoryStream stream, bool hasMvidSection)
{
Guid mvidFromModuleDefinition;
Copy link
Member

@cston cston May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declaration can be moved into using, to match mvidFromMvidReader. Alternatively, mvidFromMvidReader and compare could be moved outside the using. #Resolved

stream.Position = 0;
using (var reader = new PEReader(stream))
{
var metadataReader = reader.GetMetadataReader();
mvidFromModuleDefinition = metadataReader.GetGuid(metadataReader.GetModuleDefinition().Mvid);

stream.Position = 0;
var mvidFromMvidReader = BuildTasks.MvidReader.ReadAssemblyMvidOrEmpty(stream);

if (hasMvidSection)
{
Assert.Equal(mvidFromModuleDefinition, mvidFromMvidReader);
Copy link
Member

@cston cston May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we checking the mvids are non-default? #Resolved

}
else
{
Assert.NotEqual(Guid.Empty, mvidFromModuleDefinition);
Assert.Equal(Guid.Empty, mvidFromMvidReader);
}
}
}

[Fact]
public void EmitRefAssembly_PrivatePropertySetter()
{
Expand All @@ -296,6 +326,8 @@ public class C
VerifyMethods(output, new[] { "System.Int32 C.<PrivateSetter>k__BackingField", "System.Int32 C.PrivateSetter.get", "void C.PrivateSetter.set",
"C..ctor()", "System.Int32 C.PrivateSetter { get; private set; }" });
VerifyMethods(metadataOutput, new[] { "System.Int32 C.PrivateSetter.get", "C..ctor()", "System.Int32 C.PrivateSetter { get; }" });
VerifyMvid(output, hasMvidSection: false);
VerifyMvid(metadataOutput, hasMvidSection: true);
}
}

Expand Down
25 changes: 23 additions & 2 deletions src/Compilers/CSharp/Test/Emit/Emit/DeterministicTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,19 @@ namespace Microsoft.CodeAnalysis.CSharp.UnitTests.Emit
public class DeterministicTests : EmitMetadataTestBase
{
private Guid CompiledGuid(string source, string assemblyName, bool debug)
{
return CompiledGuid(source, assemblyName, options: debug ? TestOptions.DebugExe : TestOptions.ReleaseExe);
}

private Guid CompiledGuid(string source, string assemblyName, CSharpCompilationOptions options, EmitOptions emitOptions = null)
{
var compilation = CreateCompilation(source,
assemblyName: assemblyName,
references: new[] { MscorlibRef },
options: (debug ? TestOptions.DebugExe : TestOptions.ReleaseExe).WithDeterministic(true));
options: options.WithDeterministic(true));

Guid result = default(Guid);
base.CompileAndVerify(compilation, validator: a =>
base.CompileAndVerify(compilation, emitOptions: emitOptions, validator: a =>
{
var module = a.Modules[0];
result = module.GetModuleVersionIdOrThrow();
Expand Down Expand Up @@ -104,6 +109,22 @@ public static void Main(string[] args) {}
Assert.NotEqual(mvid3, mvid7);
}

[Fact]
public void RefAssembly()
{
var source =
@"class Program
{
public static void Main(string[] args) {}
CHANGE
}";
var emitRefAssembly = EmitOptions.Default.WithEmitMetadataOnly(true).WithIncludePrivateMembers(false);

var mvid1 = CompiledGuid(source.Replace("CHANGE", ""), "X1", TestOptions.DebugDll, emitRefAssembly);
var mvid2 = CompiledGuid(source.Replace("CHANGE", "private void M() { }"), "X1", TestOptions.DebugDll, emitRefAssembly);
Assert.Equal(mvid1, mvid2);
}

const string CompareAllBytesEmitted_Source = @"
using System;
using System.Linq;
Expand Down
6 changes: 1 addition & 5 deletions src/Compilers/Core/MSBuildTask/CopyRefAssembly.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

using System;
using System.IO;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

Expand Down Expand Up @@ -85,10 +83,8 @@ private bool Copy()
private Guid ExtractMvid(string path)
{
using (FileStream source = File.OpenRead(path))
using (var reader = new PEReader(source))
{
var metadataReader = reader.GetMetadataReader();
return metadataReader.GetGuid(metadataReader.GetModuleDefinition().Mvid);
return MvidReader.ReadAssemblyMvidOrEmpty(source);
}
}
}
Expand Down
1 change: 1 addition & 0 deletions src/Compilers/Core/MSBuildTask/MSBuildTask.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
</Compile>
<Compile Include="AssemblyResolution.cs" />
<Compile Include="CanonicalError.cs" />
<Compile Include="MvidReader.cs" />
<Compile Include="CopyRefAssembly.cs" />
<Compile Include="ValidateBootstrap.cs" />
<Compile Include="CommandLineBuilderExtension.cs" />
Expand Down
129 changes: 129 additions & 0 deletions src/Compilers/Core/MSBuildTask/MvidReader.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
// Copyright (c) Microsoft. All Rights Reserved. Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System;
using System.Diagnostics;
using System.IO;
using System.Text;

namespace Microsoft.CodeAnalysis.BuildTasks
{
public sealed class MvidReader : BinaryReader
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there callers that instantiate this type? If not, could this be a static class (with a single public method) that operates on a BinaryReader directly?

Copy link
Member Author

@jcouv jcouv May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make the constructor private.
I gave your suggestion a shot (making the class static, but it made the code worse). For instance, I need to add ReadUInt32 and ReadUInt16 wrapper methods, and then pass a BinaryReader to pretty much every line in this class. #Resolved

{
public MvidReader(Stream stream) : base(stream)
{
}

public static Guid ReadAssemblyMvidOrEmpty(Stream stream)
{
try
{
var mvidReader = new MvidReader(stream);
return mvidReader.TryFindMvid();
}
catch (EndOfStreamException)
{
}

return Guid.Empty;
Copy link
Member

@jaredpar jaredpar May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel like this is essentially hiding failure here. Consider using a TryReadAssemblyMvid pattern and potentially have a wrapper named ReadAssemblyMvidOrEmpty #Resolved

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jaredpar I renamed the methods. Let me know if that's better.


In reply to: 114092229 [](ancestors = 114092229)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the exception handler still needed?

Copy link
Member Author

@jcouv jcouv May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. If the ref assembly is truncated (there are section headers, but no actual sections), we risk reading past the end of the file. #Resolved

}

public Guid TryFindMvid()
{
Guid empty = Guid.Empty;

// DOS Header (64), DOS Stub (64), PE Signature (4), COFF Header (20), PE Header (224)
if (BaseStream.Length < 64 + 64 + 4 + 20 + 224)
{
return empty;
}

// DOS Header: PE (2)
if (ReadUInt16() != 0x5a4d)
{
return empty;
}

// DOS Header: Start (58)
Skip(58);

// DOS Header: Address of PE Signature
MoveTo(ReadUInt32());
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MoveTo(ReadUInt32()); [](start = 12, length = 21)

It looks like we are moving to an unknown position without verifying if the destination makes sense. #Resolved

Copy link
Member Author

@jcouv jcouv May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That's why this logic is wrapped in a try/catch (EndOfStreamException). #Resolved

Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That's why this logic is wrapped in a try/catch (EndOfStreamException).

But this method is public and it doesn't catch any exceptions. #Resolved

Copy link
Member

@KirillOsenkov KirillOsenkov May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to avoid first-chance exceptions if at all possible. Just read the value and check against the stream length. #Resolved


// PE Signature ('P' 'E' null null)
if (ReadUInt32() != 0x00004550)
{
return empty;
}

// COFF Header: Machine (2)
Skip(2);

// COFF Header: NumberOfSections (2)
ushort sections = ReadUInt16();

// COFF Header: TimeDateStamp (4), PointerToSymbolTable (4), NumberOfSymbols (4)
Skip(12);

// COFF Header: OptionalHeaderSize (2)
ushort optionalHeaderSize = ReadUInt16();

// COFF Header: Characteristics (2)
Skip(2);

// Optional header
Skip(optionalHeaderSize);

// Section headers
return FindMvidInSections(sections);
}

private Guid FindMvidInSections(ushort count)
{
for (int i = 0; i < count; i++)
{
// Section: Name (8)
byte[] name = ReadBytes(8);
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReadBytes [](start = 30, length = 9)

Where is this helper defined? #Resolved

Copy link
Member Author

@jcouv jcouv May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In System.IO.BinaryReader #Resolved

if (name.Length == 8 && name[0] == '.' &&
name[1] == 'm' && name[2] == 'v' && name[3] == 'i' && name[4] == 'd')
Copy link
Member

@tmat tmat May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name[4] == 'd' [](start = 74, length = 14)

&& name[5] == '\0' so that we don't match on ".mvidXXX" #Resolved

{
// Section: VirtualSize (4)
uint virtualSize = ReadUInt32();

// Section: VirtualAddress (4), SizeOfRawData (4)
Skip(8);

// Section: PointerToRawData (4)
uint pointerToRawData = ReadUInt32();

// The .mvid section only stores a Guid
Debug.Assert(virtualSize == 16);
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug.Assert(virtualSize == 16); [](start = 20, length = 32)

I think we should verify the size, rather than simply assert it. #Resolved


BaseStream.Position = pointerToRawData;
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BaseStream.Position = pointerToRawData; [](start = 20, length = 39)

Should we use Move() here? #Resolved

byte[] guidBytes = new byte[16];
BaseStream.Read(guidBytes, 0, 16);
Copy link
Contributor

@AlekseyTs AlekseyTs May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BaseStream.Read(guidBytes, 0, 16); [](start = 20, length = 34)

Not checking how many bytes were read. #Resolved


return new Guid(guidBytes);
}
else
{
Copy link
Member

@tmat tmat May 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: no need for else since the if-statement returns #Resolved

// Section: VirtualSize (4), VirtualAddress (4), SizeOfRawData (4),
// PointerToRawData (4), PointerToRelocations (4), PointerToLineNumbers (4),
// NumberOfRelocations (2), NumberOfLineNumbers (2), Characteristics (4)
Skip(4 + 4 + 4 + 4 + 4 + 4 + 2 + 2 + 4);
}
}

return Guid.Empty;
}

public void Skip(int bytes)
{
BaseStream.Seek(bytes, SeekOrigin.Current);
}

public void MoveTo(uint position)
{
BaseStream.Seek(position, SeekOrigin.Begin);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods can be private.

}
}
1 change: 0 additions & 1 deletion src/Compilers/Core/MSBuildTask/project.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
"System.IO.Pipes": "4.3.0",
"System.Linq": "4.3.0",
"System.Reflection": "4.3.0",
"System.Reflection.Metadata": "1.4.2",
"System.Security.AccessControl": "4.3.0",
"System.Security.Cryptography.Algorithms": "4.3.0",
"System.Security.Principal.Windows": "4.3.0",
Expand Down
1 change: 1 addition & 0 deletions src/Compilers/Core/Portable/CodeAnalysis.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
<Compile Include="Emit\AsyncMoveNextBodyDebugInfo.cs" />
<Compile Include="Emit\IteratorMoveNextBodyDebugInfo.cs" />
<Compile Include="Emit\StateMachineMoveNextDebugInfo.cs" />
<Compile Include="PEWriter\ExtendedPEBuilder.cs" />
<Compile Include="Serialization\IObjectWritable.cs" />
<Compile Include="Serialization\ObjectBinder.cs" />
<Compile Include="Serialization\ObjectBinderSnapshot.cs" />
Expand Down
Loading