Support full reconstruction of HCL from output dictionary

[weaversam8]

This is a follow up to #169, to more fully address the use case discussed in #23.

Along the way, I also found and fixed #171, #172, and #173. (To view those changes in isolation, please look at the diffs of the commits linked to those issues specifically.)

I also believe that this supersedes @zabacad's PR #143 to support nested interpolation. (Sorry Ian! I didn't see your PR and recreated your work - but we landed on similar approaches!)

I'm submitting this PR as a draft right now, because while this is technically working, there are a few more things I'd like to add. Namely:

• Better documentation about how to use this feature
• Better whitespace and formatting embedded in the AST generated from the dictionary
  • (this draft implementation produces syntactically equivalent output but it's very ugly)
• Helper functions to control the formatting of the generated output by embedding metadata in the Python dictionary
• Better error messages when parsing an invalid dictionary
• More tests for all of the above.

This PR adds:

• New API methods:
  • reverse_transform(hcl_dict) which returns an AST generated from a Python dictionary
• Lark grammar changes:
  • Changes to handle nested string interpolation (similar to Allow expressions in string interpolation #143)
  • Renamed several terminals to make them easier to reference when reconstructing
• BREAKING: changes to the output of the DictTransformer class to ensure it generates valid HCL expressions. (See HCL interpolations within dict output contain single quotes #172, I hope it's OK to include this since we just did a major version bump.)
• hcl2.reconstructor changes:
  • I added the HCLReverseTransformer class which aims to reverse the process that hcl2.transformer.DictTransformer takes (the bread and butter of this PR)
  • I overhauled the whitespace/formatting handling that I implemented in Support full reconstruction of HCL from parse tree #169 to make those decisions based on individual tokens/rules, rather than simply looking at the characters only
• Tests:
  • I added a simple test for HCL escape sequences for Terraform escape sequences are not processed when parsing strings #171
  • I updated the test files for the fixes I made for HCL interpolations within dict output contain single quotes #172 (see BREAKING note above)
  • I added a test for deeply nested string interpolation (Lark grammar does not support interpolations nested more than 2 times #173)
  • I modified test_load to print the full diff when a file does not match its expected JSON
  • I added test_reconstruct_dict to ensure that the HCL reconstructed from a Python dictionary is syntactically equivalent (whitespace checks will be added to this file soon.)

Still a draft PR, as described above, but open to hear any comments or suggestions!

[weaversam8]

OK, I think this is ready for a proper review. Since the draft, I added:

• More extensive documentation on how to use these features
• A rudimentary hcl2.Builder class allowing users to create fresh HCL documents from Python (the reason why this is necessary is described in the docs I added)
• Better formatting defaults on output from hcl2.reverse_transform(...).
• Tests for the hcl2.Builder class.

[weaversam8]

@kkozik-amplify - still reviewing this? any questions I can answer about my changes?

[kkozik-amplify]

@weaversam8 Yeah, need some more time. In the meantime - could you look into the errors that Codacy Analysis PR check reported?

[Nfsaavedra]

Hi!

While trying to use the code in this branch I found that when parsing the following file:

locals {
  tags = {
    application = "Modernisation Platform: ${terraform.workspace}"
  }
}

The program gets stuck in a regex match. I am almost sure it is related to the changes to better support string interpolation (that's why I'm using this branch).

[Nfsaavedra]

Suggested change

	STRING_LIT : "\"" (STRING_CHARS)* "\""
	STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string
	STRING_LIT : "\"" STRING_CHARS? "\""
	STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/ // any character except '"'

This seems to fix the problem, but I did not run the tests to check it. It does parse this file correctly tho.

[kkozik-amplify]

@Nfsaavedra Hey! Thanks for the suggestion. Since it's been some time already, I added your fix to the PR myself, all tests are passing.

[guilinden]

@weaversam8 any plans when this will be merged?

[Nfsaavedra]

Based on my understanding of the code, LGTM. Do the new tests consider the case where the previous version of the program would get stuck as described here? It's important to avoid regressions. If so, then I have nothing to add :)

[kkozik-amplify]

@Nfsaavedra I updated this test case to take issue you found into consideration.
It seems like the original regex would take longer to match non-interpolated string with each character, to the point where for 20 characters it would basically get stuck.

[Nfsaavedra]

Makes sense! :)