Provide API to transform source to source

[alrz]

This is a different take on #15929 (I'm unsure if this is already possible using "Custom Tool" in VS).

I want to suggest instead of an "IL optimization step", we provide an API to transform the source itself, I believe it will bring more flexibility and IMO it is easier to develop. A lot of inlining optimizations like #15644, linq rewriting, etc can be done in this pass. The generated code will show up in solution explorer as a child item under the actual source, and debugging will work on those generated files.

This is a more general solution to "modifying generators" proposed at dotnet/csharplang#122, except that it does not incur any changes to the language. However, with source-only attributes this can be taken further to a lot of other use cases.

[miloush]

I have to agree that I would prefer this to IL post-processing when possible and I like the idea of great debugging experience.

The question is at what level you want to do the transform - the syntax tree? Or do you want to have already resolved things such as iterators, anonymous types etc.?

[AdamSpeight2008]

@alrz Are you suggesting macros? or templating?

[alrz]

@miloush

This does not happen in betwixt of the compiler passes, just that syntax tree (and possibly semantic model) will be passed to the generator to examine. Each generator will update part of the AST (per file) and return the rest as-is.

@AdamSpeight2008

No. This does not affect the language itself in any ways. But I can see that it can be used a templating mechanism as well.

---

I think the main challenge here is that when we should trigger generators. I think that should be once we save a file (and the final compilation), just like any other "custom tool" e.g. resx designer, except that it will run generators for every file since any change can affect semantics in other files as well.

[miloush]

Alright, do you have a scenario that you want to use this for?

[alrz]

@miloush

Basically, any optimization that you'd prefer to not defer to jit (since roslyn does not do many optimizations in front end), like inlining, loop optimizations, etc; and others that cannot be done on jit compiler like linq rewriting. You could use source-only attributes to hint a generator to do a custom transformation as well, see dotnet/csharplang#122 for some examples.

[Pilchie]

This is already possible using existing msbuild and VS extensibility - take a look at how T4 templates work for example.

[miloush]

@Pilchie do you know if there is an extensibility point if you wanted to transform the lowered tree (i.e. with state machines, anonymous types etc.)?

[Pilchie]

There is not.