Beginning real-world usage

This point marks a tiny milestone in my work in modifying Roslyn: I've used my modified Roslyn compiler to build a tiny program that is published and available for general usage. If you go to this page: https://merlinia.com/en/download-en

at the very bottom there is a program called WordscraperBreaker. You're welcome to download it and use it to improve your scores when playing Wordscraper on Facebook. (But don't cheat - share the program with your opponent, so you're still competing on a level playing field.)

If you examine WordscraperBreaker.exe with JetBrains dotPeek you will see how the names of all of the types, methods and fields have been "anonymized".

Including resources when testing modifications

The subject of this blog article is related to the fact that when I started to use my modified Roslyn compiler on the WordscraperBreaker program I became aware of a problem involving the "resources" used by a .Net WinForms program. The resource files were being properly handled when I ran my modified Roslyn compiler under Visual Studio or MSBuild, but not when I was working directly with Roslyn using my test setup. The problem was that this information was provided to csc.exe via command-line arguments, and my test program bypasses csc.exe in an attempt to be in "closer contact" with the C# compiler parts of Roslyn while I'm working on my modifications.

So I had to make some improvements to my test program, and that's what I'm describing in this article.

Specifying an icon for the program

One of the things that was missing when I built WordscraperBreaker.exe via my Roslyn test program was the program's icon. At first I had a lot of difficulty finding information about how to provide an icon for the program being built. (As I've mentioned before, there seems to be lots of information available on the internet about how to use Roslyn via the authorized APIs, but not so much about how to modify Roslyn or interface more directly with Roslyn.)

Fortunately, I finally found this: https://stackoverflow.com/questions/41111826/emit-win32icon-with-roslyn

Where would we be without Stack Overflow?

Embedded resources for a WinForms project

The other thing I was missing were the embedded resource files. This required an analysis of the .csproj file and the creation of a collection of ResourceDescription  objects. These are then used by Roslyn duing the emitting of the PE file. In the case of .resx files it was necessary to provide code to convert them into .resources files - that's the format that is actually embedded in the PE file.

One problem that I encountered here was that the resulting program would crash with the error "Could not load file or assembly 'System.Drawing, Version=4.0.0.0". I found it necessary to implement a rather kludgy scan of the .resources file data and simply replace text occurrences of "System.Drawing, Version=4.0.0.0" with "System.Drawing, Version=2.0.0.0". I think this problem was related to the fact that Visual Studio and MSBuild use a program called Resgen.exe, and I'm using the ResourceWriter class instead in an attempt to get better performance (no disk I/O).

One final problem involving the use of the embedded resources (which was also applicable for use via Visual Studio and MSBuild) was that when an anonymized module was produced, then the name of a .resources file had to be changed to match the name of the type it was associated with. So if class Merlinia.WordscraperBreaker.FormMain has been renamed to Merlinia.T2229$0001 then it is also necessary to rename Merlinia.WordscraperBreaker.FormMain.resources to Merlinia.T2229$0001.resources.

Program.cs

Here is the relevant code from the program I use to invoke Roslyn when testing my modifications:

/// <summary> /// Method to perform a single Roslyn C# compilation based on a .csproj file. /// </summary> private static bool CompileOneProgram(string csprojFilename) { // Check .csproj file exists if (!File.Exists(csprojFilename)) { DisplayErrorOrInfo(string.Format(CultureInfo.InvariantCulture, "File {0} does not exist.", csprojFilename)); return false; } // Create an MSBuildWorkspace object using (MSBuildWorkspace msBuildWorkspace = MSBuildWorkspace.Create()) { // Create a Project object that represents the project, and display any error messages Project msBuildWorkspaceProject = msBuildWorkspace.OpenProjectAsync(csprojFilename).Result; if (!msBuildWorkspace.Diagnostics.IsEmpty) { foreach (WorkspaceDiagnostic workspaceDiagnostic in msBuildWorkspace.Diagnostics) Console.WriteLine(workspaceDiagnostic.ToString()); Console.WriteLine(""); return false; } // Display some info just to be sure that the project was loaded OK Console.WriteLine("Assembly name = {0}", msBuildWorkspaceProject.AssemblyName); foreach (Document projectDocument in msBuildWorkspaceProject.Documents) Console.WriteLine(" " + projectDocument.Name); // Test for some error situations that have been (painfully) experienced. But these may // no longer be relevant now that msBuildWorkspace.Diagnostics is being queried. if (msBuildWorkspaceProject.OutputFilePath == "") { // See https://stackoverflow.com/a/49886334/253938 DisplayErrorOrInfo( "OutputFilePath = \"\"! Maybe due to not calling Microsoft.Build.Locator."); return false; } if (msBuildWorkspaceProject.MetadataReferences.Count == 0) { // See https://stackoverflow.com/a/39645040/253938 DisplayErrorOrInfo("No references! Probably something wrong with .csproj file."); } // Determine if the .csproj file includes <EmbeddedResource> elements and create a // collection of ResourceDescription objects if so string applicationIcon; IEnumerable<ResourceDescription> manifestResources = GetManifestResources(csprojFilename, out applicationIcon); // Create a CSharpCompilation object to perform the compilation of the project. It is // first at this point that code common with usage of csc.exe finally gets hit, and // first at this point that a few of the Yacks modifications to Roslyn come into play. Compilation roslynCompilation = msBuildWorkspaceProject.GetCompilationAsync().Result; // Do the C# compilation and produce an output file, or maybe two output files if the // YacksAnonymizeModule option is selected. (This previously used the Emit() method in // FileSystemExtensions, which simplified things somewhat, but that does not support // providing a Stream for an .ico file. Info about how to support the .ico file found // here: https://stackoverflow.com/questions/41111826/emit-win32icon-with-roslyn .) EmitResult emitResult; using (FileStream iconStreamOpt = applicationIcon == null ? null : File.OpenRead(applicationIcon)) using (Stream iconResourceOpt = roslynCompilation.CreateDefaultWin32Resources( true, true, null, iconStreamOpt)) using (FileStream outputStream = File.OpenWrite(msBuildWorkspaceProject.OutputFilePath)) { emitResult = roslynCompilation.Emit(outputStream, manifestResources: manifestResources, win32Resources: iconResourceOpt); } // If our compilation failed, we can discover exactly why if (!emitResult.Success) { foreach (Diagnostic roslynDiagnostic in emitResult.Diagnostics) Console.WriteLine(roslynDiagnostic.ToString()); Console.WriteLine(""); return false; } } // OK completion Console.WriteLine("Compilation complete."); Console.WriteLine(""); return true; } /// <summary> /// Method to analyze the .csproj file and determine if there are embedded resource files and /// to convert them into a collection of ResourceDescription objects. /// /// Embedded resource files are typically .resx files, but there are many other possibilities: /// .txt, .xml, graphics files, etc. This code has so far only been tested with .resx and .txt /// files. /// /// Parts of this code based on this Stack Overflow answer: /// https://stackoverflow.com/a/44142655/253938 /// </summary> /// <param name="csprojFileName">full path and filename for .csproj file</param> /// <param name="applicationIcon">full path and filename for ApplicationIcon, or null</param> /// <returns>collection of ResourceDescription objects, /// or null if no embedded resource files or error encountered</returns> [SuppressMessage("Microsoft.Design", "CA1031:DoNotCatchGeneralExceptionTypes")] private static IEnumerable<ResourceDescription> GetManifestResources(string csprojFileName, out string applicationIcon) { try { string baseDirectory = Path.GetDirectoryName(csprojFileName); Debug.Assert(baseDirectory != null); XDocument csprojAsXml = XDocument.Load(csprojFileName); XNamespace xmlNamespace = "http://schemas.microsoft.com/developer/msbuild/2003"; // Test for an <ApplicationIcon> element, process if so XElement applicationIconElement = GetXmlElement(csprojAsXml, xmlNamespace, "ApplicationIcon"); applicationIcon = applicationIconElement == null ? null : Path.Combine(baseDirectory, applicationIconElement.Value); // Check there is a <TargetFrameworkVersion> element and get the value string targetFrameworkVersion = GetAndCheckXmlElement(csprojAsXml, xmlNamespace, "TargetFrameworkVersion", csprojFileName); if (targetFrameworkVersion == null) return null; bool targetLessThan4 = string.CompareOrdinal(targetFrameworkVersion, "v4.") < 0; // Check there is a <RootNamespace> element and get the value string projectNamespace = GetAndCheckXmlElement(csprojAsXml, xmlNamespace, "RootNamespace", csprojFileName); if (projectNamespace == null) return null; // Process the <EmbeddedResource> elements List<ResourceDescription> resourceDescriptions = new List<ResourceDescription>(); foreach (XElement embeddedResourceElement in csprojAsXml.Descendants(xmlNamespace + "EmbeddedResource")) { XAttribute includeAttribute = embeddedResourceElement.Attribute("Include"); if (includeAttribute == null) continue; // Shouldn't be possible string resourceFilename = includeAttribute.Value; string resourceFullFilename = Path.Combine(baseDirectory, resourceFilename); if (resourceFilename.EndsWith(".resx", StringComparison.OrdinalIgnoreCase)) { string resourceName = resourceFilename.Remove(resourceFilename.Length - 5) + ".resources"; resourceDescriptions.Add( new ResourceDescription(projectNamespace + "." + resourceName, () => ProvideResourceDataForResx(resourceFullFilename, targetLessThan4), true)); } else { // This has currently only been tested with .txt file resources resourceDescriptions.Add( new ResourceDescription(projectNamespace + "." + resourceFilename, () => ProvideResourceDataForFile(resourceFullFilename), true)); } } return resourceDescriptions.Count == 0 ? null : resourceDescriptions; } catch (Exception e) { DisplayErrorOrInfo("Exception while processing embedded resource files: " + e.Message); applicationIcon = null; return null; } } /// <summary> /// Method to get an XML element from the .csproj file, returning it as a string. /// </summary> /// <returns>text from element, or null if not found (error message written)</returns> private static string GetAndCheckXmlElement(XDocument csprojAsXml, XNamespace xmlNamespace, string elementName, string csprojFileName) { XElement projectElement = GetXmlElement(csprojAsXml, xmlNamespace, elementName); if (projectElement != null) return projectElement.Value; DisplayErrorOrInfo(string.Format(CultureInfo.InvariantCulture, "Unable to find element {0} for project {1}.", elementName, csprojFileName)); return null; } /// <summary> /// Method to get an XML element from the .csproj file. May return null. /// </summary> private static XElement GetXmlElement(XDocument csprojAsXml, XNamespace xmlNamespace, string elementName) { return csprojAsXml.Descendants(xmlNamespace + elementName).FirstOrDefault(); } /// <summary> /// Method that gets called by ManagedResource.WriteData() in project CodeAnalysis during code /// emitting to get the data for an embedded resource file that is not a .resx file. Caller /// guarantees that the returned FileStream object gets disposed. /// </summary> /// <param name="resourceFullFilename">full path and filename for resource file to embed</param> /// <returns>FileStream providing binary image of a non-.resx file</returns> [SuppressMessage("Microsoft.Reliability", "CA2000:Dispose objects before losing scope")] private static FileStream ProvideResourceDataForFile(string resourceFullFilename) { return new FileStream(resourceFullFilename, FileMode.Open); } /// <summary> /// Method that gets called by ManagedResource.WriteData() in project CodeAnalysis during code /// emitting to get the data for an embedded .resx file. Caller guarantees that the returned /// MemoryStream object gets disposed. /// </summary> /// <param name="resourceFullFilename">full path and filename for .resx file to embed</param> /// <param name="targetLessThan4">true if necessary to change System.Drawing from 4.0.0.0 to 2.0.0.0</param> /// <returns>MemoryStream containing .resources file data for the .resx file</returns> [SuppressMessage("Microsoft.Reliability", "CA2000:Dispose objects before losing scope")] private static MemoryStream ProvideResourceDataForResx(string resourceFullFilename, bool targetLessThan4) { MemoryStream shortLivedBackingStream = new MemoryStream(); using (ResourceWriter resourceWriter = new ResourceWriter(shortLivedBackingStream)) { using (ResXResourceReader resourceReader = new ResXResourceReader(resourceFullFilename)) { IDictionaryEnumerator dictionaryEnumerator = resourceReader.GetEnumerator(); while (dictionaryEnumerator.MoveNext()) { string resourceKey = dictionaryEnumerator.Key as string; if (resourceKey != null) // Should not be possible resourceWriter.AddResource(resourceKey, dictionaryEnumerator.Value); } } } // Get reference to the buffer used by shortLivedBackingStream, which is now closed because // resourceWriter was disposed. If relevant, fix version number for System.Drawing. byte[] backingStreamBuffer = shortLivedBackingStream.GetBuffer(); if (targetLessThan4) ChangeSystemDrawingVersionNumber(backingStreamBuffer); // Create new MemoryStream because shortLivedBackingStream is closed return new MemoryStream(backingStreamBuffer); } /// <summary> /// Method to change the System.Drawing version number from "4.0.0.0" to "2.0.0.0" in the /// binary data that represents a .resources file. This implementation is based on the /// assumption that character data in the .resources file is in UTF-8 encoding. /// </summary> private static void ChangeSystemDrawingVersionNumber(byte[] dataBuffer) { byte[] byteArray1 = Encoding.UTF8.GetBytes("System.Drawing, Version=4.0.0.0"); byte[] byteArray2 = Encoding.UTF8.GetBytes("System.Drawing, Version=2.0.0.0"); for (int i = 0; i < dataBuffer.Length - byteArray1.Length; i++) if (ArrayEquals(byteArray1, dataBuffer, i)) Array.Copy(byteArray2, 0, dataBuffer, i, byteArray2.Length); } /// <summary> /// Method to test for a byte array in a larger byte array that is being searched. No error /// checking is done - it's assumed an indexing error is not possible. /// </summary> private static bool ArrayEquals(byte[] searchArray, byte[] searchedArray, int searchedArrayIndex) { for (int i = 0; i < searchArray.Length; i++) if (searchArray[i] != searchedArray[searchedArrayIndex + i]) return false; return true; }

MetadataWriter.cs

As mentioned above, it is necessary to modify the names of the .resources files when emitting an anonymized module. The following code is near the end of the PopulateManifestResourceTableRows() method:

//Yacks15: Maybe change resource name to match anonymized type name. string resourceName = GetResourceName(resource.Name); metadata.AddManifestResource( attributes: resource.IsPublic ? ManifestResourceAttributes.Public : ManifestResourceAttributes.Private, //name: GetStringHandleForNameAndCheckLength(resource.Name), name: GetStringHandleForNameAndCheckLength(resourceName), implementation: implementation, offset: GetManagedResourceOffset(resource, resourceDataWriter));

MetadataWriter.Yacks1.cs

This is a source file I've added to the Roslyn compiler. Here's the method called by the above code:

/// <summary> /// Method called when MetadataWriter is about to emit an embedded resource to the PE file. If /// an anonymized module is being produced then it is probably necessary to change the /// resource name so it matches the change made to the associated .Net type. /// </summary> private string GetResourceName(string resourceName) { if (EmittingAnonymized(resourceName) && resourceName.EndsWith(CDotResources, StringComparison.Ordinal)) { YacksCompilation yacksCompilation = module.CommonCompilation._YacksCompilation; string associatedTypeName = resourceName.Remove(resourceName.Length - CDotResources.Length); // First test if this is associated with a public class (persisted type number) YacksTypeInfo typeInfo = yacksCompilation.GetTypeInfoIfExists(associatedTypeName); if (typeInfo != null) return FormatResourceName(yacksCompilation, typeInfo.TypeNumber, true); // Now test if it's associated with a non-public class (non-persisted type number, may // be different for each compilation, but that's OK, new resource name can also change // for each compilation) int typeNumber = yacksCompilation.GetNonPersistedTypeNumber(associatedTypeName); if (typeNumber != -1) return FormatResourceName(yacksCompilation, typeNumber, false); } return resourceName; }

Conclusion

Some bits and pieces are missing from the above changes. To see everything involved in these modifications see this page: https://merlinia.com/en/download-en , where there is a .zip file containing all of my Roslyn modifications and the associated utility programs.

You must login to post a comment.
Loading comment... The comment will be refreshed after 00:00.

Be the first to comment.