[Python] Support more testing model #2541

crisidev · 2023-04-04T15:41:12Z

Motivation and Context

Python should be tested and compliant with all the testing models used in Rust. To achieve this, there are several changes required in terms of customizations, constraint traits and general bug fixing for edge cases.

In languages that are not Rust, we swap non primitive symbols like DateTime, Blob, Document, etc with wrappers that are compatible with the target language.

This was done using a hack that was looking at the original symbol. If that differed from the new one, we called from() or into() to make rustc happy.

Description

This PR implements several changes:

Symbol swapping

In languages that are not Rust, we swap non primitive symbols like DateTime, Blob, Document, etc with wrappers that are compatible with the target language.

During deserialization the runtime parser requires the Rust symbols, while we are handed back the Python wrappers. Customizations are added call .map($type::from) for Option<T>, .into() for T, etc. The customizations are added to the JsonParserGenerator, in the ServerHttpBoundProtocolGenerator and in HttpBindingGenerator.

This should close #2479

Recursive shapes

Implements FromPyObject and IntoPy traits for Box<T> to support recursive shapes. This should close #2476

Naming obstacle

Ensure we use proper casing and escapes for reserved symbol in all Python code (see also #2552)

Constrained Blob

Rust Blobs can be constrained so we need to implement the From trait to allow conversion with the Python Blob type.

This should close #2551

Generate diff for all models

We now also generate diffs for all tested models. This should close #2483

Bug fixing

This PR also fixes a bug in the PythonServerUnionGenerator where we are missing a closing ) in a case of union Unit.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…DataSerializer` No implementation of the `Protocol` interface makes use of the `OperationShape` parameter in the `structuredDataParser` and `structuredDataSerializer` methods.

customizations for JsonParserGenerator and ServerHttpBoundProtocolGenerator. Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

github-actions · 2023-04-04T15:53:09Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
Server Test (ignoring whitespace)
No codegen difference in the Server Test Python

A new doc preview is ready to view.

crisidev · 2023-04-04T16:01:45Z

I need to fix a bunch of unit tests, but the code is almost ready.

github-actions · 2023-04-04T16:17:37Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
Server Test (ignoring whitespace)
No codegen difference in the Server Test Python

A new doc preview is ready to view.

github-actions · 2023-04-04T17:07:57Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

crisidev · 2023-04-05T10:00:15Z

Is the PR diff bot still broken? I would expect changes in Python..

github-actions · 2023-04-05T10:11:35Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

github-actions · 2023-04-05T11:39:06Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

github-actions · 2023-04-08T13:21:54Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2023-04-11T12:07:17Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2023-04-11T12:40:05Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2023-04-11T13:30:54Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

hlbarber · 2023-04-11T14:08:56Z

...rc/main/kotlin/software/amazon/smithy/rust/codegen/server/smithy/protocols/ServerRestJson.kt

-class ServerRestJsonFactory : ProtocolGeneratorFactory<ServerHttpBoundProtocolGenerator, ServerCodegenContext> {
-    override fun protocol(codegenContext: ServerCodegenContext): Protocol = ServerRestJsonProtocol(codegenContext)
+class ServerRestJsonFactory(
+    private val additionalParserCustomizations: List<JsonParserCustomization> = listOf(),


NIT: Maybe we should have a ServerCustomizations class to house all these arguments?

github-actions · 2023-04-11T19:28:17Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
Client Test (ignoring whitespace)
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

github-actions · 2023-04-11T19:45:35Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

github-actions · 2023-04-12T15:48:03Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK
No codegen difference in the Client Test
No codegen difference in the Server Test
Server Test Python (ignoring whitespace)
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

rcoh

codegen-core changes LGTM, small non-blocking comment on diff_lib

rcoh · 2023-04-13T13:43:59Z

tools/ci-scripts/codegen-diff/diff_lib.py

+                get_cmd_output(f"mv {target}/python/build/smithyprojections/{target}-python {OUTPUT_PATH}/")
+                get_cmd_output(f"mv {target}/typescript/build/smithyprojections/{target}-typescript {OUTPUT_PATH}/")


should this be target == target_codegen_typescript/python instead?

At the end it does the same.. I have some other rearchitectures of the diff lib in the next PR. I'll refactor this part in it as well if that's ok?

* Remove parameter from `Protocol`s `structuredDataParser`, `structuredDataSerializer` No implementation of the `Protocol` interface makes use of the `OperationShape` parameter in the `structuredDataParser` and `structuredDataSerializer` methods. * Remove the TypeConversionGenerator class in favor of using customizations for JsonParserGenerator and ServerHttpBoundProtocolGenerator. Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com> * Make the additionaParserCustomizations default to empty list * Fix merge conflict * Fix missing ; * Use better defaults when checking for customizations * Use better defaults when checking for customizations * Add HttpBindingCustomization and relax the datetime symbol check * Support recursive shapes and add a lot more models to the tests Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com> * Support naming obstacle course * Add support for constrained blobs conversions * Support constraint traits * Try to generate the full diff Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com> * A better way of checking if we need to go into the Timestamp branch * Remove wheels folder --------- Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com> Co-authored-by: david-perez <d@vidp.dev>

david-perez and others added 2 commits April 4, 2023 09:58

Remove parameter from Protocols structuredDataParser, `structured…

a7f86c5

…DataSerializer` No implementation of the `Protocol` interface makes use of the `OperationShape` parameter in the `structuredDataParser` and `structuredDataSerializer` methods.

Remove the TypeConversionGenerator class in favor of using

80bd255

customizations for JsonParserGenerator and ServerHttpBoundProtocolGenerator. Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

crisidev requested review from a team as code owners April 4, 2023 15:41

crisidev requested review from david-perez, drganjoo and unexge April 4, 2023 15:41

crisidev marked this pull request as draft April 4, 2023 15:41

crisidev added enhancement New feature or request python-server Python server SDK labels Apr 4, 2023

crisidev changed the title ~~[WRemove the TypeConversionGenerator class in favor of using customizations for JsonParserGenerator and ServerHttpBoundProtocolGenerator~~ [WIP][Python] Remove the TypeConversionGenerator class in favor of using customizations Apr 4, 2023

Make the additionaParserCustomizations default to empty list

146ec10

crisidev added 2 commits April 4, 2023 17:50

Merge branch 'main' into oxipy-testing-models

f125bc0

Fix merge conflict

3a92f64

Merge branch 'main' into oxipy-testing-models

b0fe123

Merge branch 'main' into oxipy-testing-models

e5d7256

crisidev changed the title ~~[WIP][Python] Remove the TypeConversionGenerator class in favor of using customizations~~ [Python] Remove the TypeConversionGenerator class in favor of using customizations Apr 5, 2023

crisidev marked this pull request as ready for review April 5, 2023 11:26

crisidev added 3 commits April 5, 2023 12:41

Fix missing ;

84fe43a

Use better defaults when checking for customizations

b050d37

Use better defaults when checking for customizations

aada4fb

Merge branch 'main' into oxipy-testing-models

5259205

unexge approved these changes Apr 11, 2023

View reviewed changes

crisidev added 2 commits April 11, 2023 12:49

Merge branch 'main' into oxipy-testing-models

0b34887

Merge branch 'main' into oxipy-testing-models

af419d5

Add support for constrained blobs conversions

87dbc4c

Support constraint traits

716fd59

crisidev added the needs-sdk-review label Apr 11, 2023

hlbarber approved these changes Apr 11, 2023

View reviewed changes

crisidev removed the needs-server-review label Apr 11, 2023

Merge branch 'main' into oxipy-testing-models

70fb1c4

Try to generate the full diff

740ddb9

Signed-off-by: Bigo <1781140+crisidev@users.noreply.github.com>

crisidev added 2 commits April 12, 2023 16:33

A better way of checking if we need to go into the Timestamp branch

6f39c1a

Remove wheels folder

1a555d1

rcoh approved these changes Apr 13, 2023

View reviewed changes

crisidev added this pull request to the merge queue Apr 13, 2023

Merged via the queue into main with commit d97defb Apr 13, 2023

crisidev deleted the oxipy-testing-models branch April 13, 2023 14:22

Velfi removed the needs-sdk-review label Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python] Support more testing model #2541

[Python] Support more testing model #2541

crisidev commented Apr 4, 2023 •

edited

Loading

github-actions bot commented Apr 4, 2023

crisidev commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

crisidev commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 8, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

hlbarber Apr 11, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 12, 2023

rcoh left a comment

rcoh Apr 13, 2023

crisidev Apr 13, 2023

		get_cmd_output(f"mv {target}/python/build/smithyprojections/{target}-python {OUTPUT_PATH}/")
		get_cmd_output(f"mv {target}/typescript/build/smithyprojections/{target}-typescript {OUTPUT_PATH}/")

[Python] Support more testing model #2541

[Python] Support more testing model #2541

Conversation

crisidev commented Apr 4, 2023 • edited Loading

Motivation and Context

Description

Symbol swapping

Recursive shapes

Naming obstacle

Constrained Blob

Generate diff for all models

Bug fixing

github-actions bot commented Apr 4, 2023

crisidev commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 4, 2023

crisidev commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 8, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

hlbarber Apr 11, 2023

Choose a reason for hiding this comment

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 11, 2023

github-actions bot commented Apr 12, 2023

rcoh left a comment

Choose a reason for hiding this comment

rcoh Apr 13, 2023

Choose a reason for hiding this comment

crisidev Apr 13, 2023

Choose a reason for hiding this comment

crisidev commented Apr 4, 2023 •

edited

Loading