Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Escape invalid characters when used in Java identifiers #659

Merged
merged 4 commits into from
Jun 16, 2020

Conversation

kelnos
Copy link
Member

@kelnos kelnos commented Jun 11, 2020

I ran into this when I wanted an enum value called integer?, and figured I'd add code for escaping all the various non-identifier characters, at least the ones found on a US keyboard (note that $ is actually legal in Java identifiers, so it's not included here).

This turned out to be a huge rabbit hole, because, despite the work we did last year to make naming better, there was still a lot of inconsistency and hackery around naming things. In order to make this a reasonable problem to solve, I added (to LanguageTerms) some new methods:

  • formatPackageName
  • formatTypeName
  • formatMethodName
  • formatMethodArgName
  • (we already had) formatEnumName

The bonus here is that as we add other languages with different naming conventions, it becomes trivial to have guardrail follow those naming conventions.

In addition, I set things up so some names (for example, the name of a response class) are generated by the core (by calling the above methods) and threading them into the interpreter methods so the interpreters don't keep rebuilding names in an error-prone way.

This doesn't fix everything. There are still some instances of .capitalize and .uncapitalized here and there. There are fewer instances than before, which is nice. And the core no longer does .toCamelCase, .toSnakeCase, or .toPascalCase anywhere. The one thing I am still not able to eradicate is ProtocolGenerator's transformProperty's needsCamelSnakeConversion, because doing so still breaks the conflicting-name fixer-upper, and making that continue to work would require more surgery. So, sadly, I will yet again punt on that.

This will unfortunately require some user code to change to continue to compile, which is detailed in MIGRATING.

Contributing to Twilio

All third-party contributors acknowledge that any contributions they provide will be made under the same open-source license that the open-source project is provided under.

  • I acknowledge that all my contributions will be made under the project's license.

@kelnos kelnos requested a review from blast-hardcheese June 11, 2020 23:26
@kelnos
Copy link
Member Author

kelnos commented Jun 11, 2020

Oh, also interesting, perhaps: BacktickTest is nearly obsolete, as almost all the names generated no longer need backticks. Strangely dashy-class is still there as-is, which I should perhaps dig into and fix before we merge this.

@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2020

Codecov Report

Merging #659 into master will increase coverage by 0.03%.
The diff coverage is 88.17%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #659      +/-   ##
==========================================
+ Coverage   81.13%   81.17%   +0.03%     
==========================================
  Files          78       78              
  Lines        5163     5195      +32     
  Branches      130      129       -1     
==========================================
+ Hits         4189     4217      +28     
- Misses        974      978       +4     
Impacted Files Coverage Δ
...ail/generators/Java/SpringMvcClientGenerator.scala 11.11% <ø> (ø)
...il/generators/Scala/EndpointsServerGenerator.scala 0.00% <ø> (ø)
.../guardrail/protocol/terms/client/ClientTerms.scala 0.00% <0.00%> (ø)
...l/protocol/terms/protocol/ModelProtocolTerms.scala 0.00% <0.00%> (ø)
.../guardrail/protocol/terms/server/ServerTerms.scala 0.00% <0.00%> (ø)
...ala/com/twilio/guardrail/terms/LanguageTerms.scala 0.00% <0.00%> (ø)
.../main/scala/com/twilio/guardrail/SwaggerUtil.scala 87.58% <50.00%> (ø)
...om/twilio/guardrail/generators/JavaGenerator.scala 75.24% <92.30%> (+0.75%) ⬆️
...n/src/main/scala/com/twilio/guardrail/Common.scala 98.80% <94.44%> (+0.01%) ⬆️
...n/scala/com/twilio/guardrail/ClientGenerator.scala 100.00% <100.00%> (+8.00%) ⬆️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 623fc43...5569961. Read the comment docs.

@kelnos
Copy link
Member Author

kelnos commented Jun 12, 2020

Ok, two things:

  1. Fixed up protocol definition class names not getting formatted. I'm not sure I did it in the best way, so I left it in its own commit so you can take a look at it separately.
  2. FINALLY I got rid of needCamelSnakeConversion. Super happy this is gone, finally finally finally.

@kelnos
Copy link
Member Author

kelnos commented Jun 12, 2020

Oh -- now BacktickTest doesn't actually have any backticks in the generated code anymore, heh. Should we remove it?

Also, apologies for the size of this PR... but I guess most of it is test changes/fixes.

@@ -31,30 +31,30 @@ class Issue126 extends AnyFunSuite with Matchers with SwaggerSpecRunner {

val handler = q"""
trait StoreHandler {
def getRoot(respond: StoreResource.getRootResponse.type)(): scala.concurrent.Future[StoreResource.getRootResponse]
def getRoot(respond: StoreResource.GetRootResponse.type)(): scala.concurrent.Future[StoreResource.GetRootResponse]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah. This is what you meant. Yes, this is quite unfortunate.

Copy link
Member

@blast-hardcheese blast-hardcheese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, and I request before we merge this that we have a scalafix rule ready to go along with this. I'm happy to write the rule, but I can't get to it this week. If you'd like to write the rule, https://github.com/blast-hardcheese/guardrail-scalafix-rules is not too unpleasant. https://youtu.be/InN_YdVUDnE?t=2507 is where I started writing a scalafix rule for the Tagless Final refactorer if you get stuck, I did everything on-stream.

for {
clientName <- formatTypeName(className.lastOption.getOrElse(""), Some("Client"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a great opportunity to push the string "Client" into each generator, further reducing the impedance mismatch when moving to support non-pascal-case languages.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, actually, and i even could have made the param value "client" or "CLIENT" or whatever; the formatter is responsible for doing the right thing for the language. I wanted the generators to not be responsible for naming things, as much as is reasonably possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, though hopefully my reticence towards hard-coded strings in core is understandable

@kelnos
Copy link
Member Author

kelnos commented Jun 15, 2020

I'll try to take a stab at the scalafix rules, but might not get to it before you have time.

This is some prep work for adding better identifier escaping to Java,
since Java doesn't have something like Scala's backticks for turning an
invalid identifier into a valid one.

Additionally, the core shouldn't be formatting names of things, since
conventions differ between languages.

This doesn't completely fix up every usage everywhere, but it should at
least remove things like calls to `.toSnakeCase` from the core, and
threads through some more pre-made names so the various generator
methods don't have to keep rebuilding them.
@kelnos kelnos force-pushed the java-escape-invalid-characters branch from 5569961 to 3e99b31 Compare June 16, 2020 19:58
kelnos added 3 commits June 16, 2020 13:52
This allows characters like `?`, `:`, etc. in things like query and form
parameter names, and object schema property names.
@kelnos kelnos force-pushed the java-escape-invalid-characters branch from 3e99b31 to 00a89e9 Compare June 16, 2020 20:52
@kelnos
Copy link
Member Author

kelnos commented Jun 16, 2020

I'm sure this will need a couple iterations to get right, but here is the scalafix rule I whipped up: guardrail-dev/guardrail-scalafix-rules#1

@kelnos kelnos merged commit 637e7e9 into guardrail-dev:master Jun 16, 2020
@kelnos kelnos deleted the java-escape-invalid-characters branch June 16, 2020 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants