Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mypy-protobuf #10914

Merged
merged 4 commits into from
Oct 22, 2023
Merged

Update mypy-protobuf #10914

merged 4 commits into from
Oct 22, 2023

Conversation

JelleZijlstra
Copy link
Member

See #10909 (comment)

I had to edit some sed commands so they worked for me on Linux.

The script also set the version of stubs/protobuf/METADATA.toml to 4.21.0; I changed it back.

I tried updating PROTOBUF_VERSION and PYTHON_PROTOBUF_VERSION to the latest (24.4/4.24.4), but the script failed; apparently they no longer distribute the Python library in the way the script expects.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@JelleZijlstra
Copy link
Member Author

cc @nipunn1313

@github-actions

This comment has been minimized.

@Avasam
Copy link
Collaborator

Avasam commented Oct 18, 2023

How about the sync_tensorflow_protobuf_stubs.sh script? I'm also unsure how to regenerate s2clientprotocol stubs

@JelleZijlstra
Copy link
Member Author

How about the sync_tensorflow_protobuf_stubs.sh script?

$ scripts/sync_tensorflow_protobuf_stubs.sh
++ dirname scripts/sync_tensorflow_protobuf_stubs.sh
+ cd scripts
+ cd ../stubs/tensorflow
+++ dirname scripts/sync_tensorflow_protobuf_stubs.sh
++ realpath scripts/..
realpath: scripts/..: No such file or directory
+ REPO_ROOT=

Don't have time to figure this out right now, sorry.

@Avasam
Copy link
Collaborator

Avasam commented Oct 18, 2023

The script also set the version of stubs/protobuf/METADATA.toml to 4.21.0; I changed it back.

Should the extra_description field still be updated to mention3.5.0 instead of 3.4.0 ?

@github-actions

This comment has been minimized.

@JelleZijlstra
Copy link
Member Author

@Avasam I get this locally now when running the TF script:

+ protoc --mypy_out relax_strict_optional_primitives:/home/jelle/repos/typeshed/stubs/tensorflow tensorflow/compiler/xla/autotune_results.proto tensorflow/compiler/xla/xla_data.proto tensorflow/compiler/xla/xla.proto tensorflow/compiler/xla/service/hlo_execution_profile_data.proto tensorflow/compiler/xla/service/hlo_profile_printer_data.proto tensorflow/compiler/xla/service/hlo.proto tensorflow/compiler/xla/service/metrics.proto tensorflow/compiler/xla/service/test_compilation_environment.proto tensorflow/core/example/example_parser_configuration.proto tensorflow/core/example/example.proto tensorflow/core/example/feature.proto tensorflow/core/framework/allocation_description.proto tensorflow/core/framework/api_def.proto tensorflow/core/framework/attr_value.proto tensorflow/core/framework/cost_graph.proto tensorflow/core/framework/dataset_metadata.proto tensorflow/core/framework/dataset_options.proto tensorflow/core/framework/dataset.proto tensorflow/core/framework/device_attributes.proto tensorflow/core/framework/full_type.proto tensorflow/core/framework/function.proto tensorflow/core/framework/graph.proto tensorflow/core/framework/graph_transfer_info.proto tensorflow/core/framework/kernel_def.proto tensorflow/core/framework/log_memory.proto tensorflow/core/framework/model.proto tensorflow/core/framework/node_def.proto tensorflow/core/framework/op_def.proto tensorflow/core/framework/optimized_function_graph.proto tensorflow/core/framework/reader_base.proto tensorflow/core/framework/resource_handle.proto tensorflow/core/framework/step_stats.proto tensorflow/core/framework/summary.proto tensorflow/core/framework/tensor_description.proto tensorflow/core/framework/tensor.proto tensorflow/core/framework/tensor_shape.proto tensorflow/core/framework/tensor_slice.proto tensorflow/core/framework/types.proto tensorflow/core/framework/variable.proto tensorflow/core/framework/versions.proto tensorflow/core/protobuf/autotuning.proto tensorflow/core/protobuf/bfc_memory_map.proto tensorflow/core/protobuf/cluster.proto tensorflow/core/protobuf/composite_tensor_variant.proto tensorflow/core/protobuf/config.proto tensorflow/core/protobuf/control_flow.proto tensorflow/core/protobuf/conv_autotuning.proto tensorflow/core/protobuf/core_platform_payloads.proto tensorflow/core/protobuf/critical_section.proto tensorflow/core/protobuf/data_service.proto tensorflow/core/protobuf/debug_event.proto tensorflow/core/protobuf/debug.proto tensorflow/core/protobuf/device_filters.proto tensorflow/core/protobuf/device_properties.proto tensorflow/core/protobuf/eager_service.proto tensorflow/core/protobuf/error_codes.proto tensorflow/core/protobuf/fingerprint.proto tensorflow/core/protobuf/graph_debug_info.proto tensorflow/core/protobuf/master.proto tensorflow/core/protobuf/master_service.proto tensorflow/core/protobuf/meta_graph.proto tensorflow/core/protobuf/named_tensor.proto tensorflow/core/protobuf/queue_runner.proto tensorflow/core/protobuf/remote_tensor_handle.proto tensorflow/core/protobuf/replay_log.proto tensorflow/core/protobuf/rewriter_config.proto tensorflow/core/protobuf/rpc_options.proto tensorflow/core/protobuf/saved_model.proto tensorflow/core/protobuf/saved_object_graph.proto tensorflow/core/protobuf/saver.proto tensorflow/core/protobuf/service_config.proto tensorflow/core/protobuf/snapshot.proto tensorflow/core/protobuf/struct.proto tensorflow/core/protobuf/tensor_bundle.proto tensorflow/core/protobuf/tensorflow_server.proto tensorflow/core/protobuf/trackable_object_graph.proto tensorflow/core/protobuf/transport_options.proto tensorflow/core/protobuf/verifier_config.proto tensorflow/core/protobuf/worker.proto tensorflow/core/protobuf/worker_service.proto tensorflow/core/protobuf/tpu/compilation_result.proto tensorflow/core/protobuf/tpu/compile_metadata.proto tensorflow/core/protobuf/tpu/dynamic_padding.proto tensorflow/core/protobuf/tpu/optimization_parameters.proto tensorflow/core/protobuf/tpu/topology.proto tensorflow/core/protobuf/tpu/tpu_embedding_configuration.proto tensorflow/core/util/event.proto tensorflow/core/util/example_proto_fast_parsing_test.proto tensorflow/core/util/memmapped_file_system.proto tensorflow/core/util/saved_tensor_slice.proto tensorflow/core/util/test_log.proto tensorflow/python/keras/protobuf/projector_config.proto tensorflow/python/keras/protobuf/saved_metadata.proto tensorflow/python/keras/protobuf/versions.proto tensorflow/tsl/protobuf/autotuning.proto tensorflow/tsl/protobuf/bfc_memory_map.proto tensorflow/tsl/protobuf/coordination_config.proto tensorflow/tsl/protobuf/coordination_service.proto tensorflow/tsl/protobuf/distributed_runtime_payloads.proto tensorflow/tsl/protobuf/dnn.proto tensorflow/tsl/protobuf/error_codes.proto tensorflow/tsl/protobuf/histogram.proto tensorflow/tsl/protobuf/rpc_options.proto tensorflow/tsl/protobuf/test_log.proto
google/protobuf/any.proto: File not found.
google/protobuf/duration.proto: File not found.
google/protobuf/wrappers.proto: File not found.
tensorflow/tsl/protobuf/dnn.proto: Import "google/protobuf/wrappers.proto" was not found or had errors.
tensorflow/tsl/protobuf/dnn.proto:139:3: "google.protobuf.UInt64Value" is not defined.
tensorflow/tsl/protobuf/autotuning.proto: Import "google/protobuf/any.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto: Import "google/protobuf/duration.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto: Import "tensorflow/tsl/protobuf/dnn.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto:55:7: "stream_executor.dnn.AlgorithmProto" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:77:3: "google.protobuf.Duration" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:85:5: "stream_executor.dnn.AlgorithmProto" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:92:3: "google.protobuf.Any" is not defined.
tensorflow/compiler/xla/autotune_results.proto: Import "tensorflow/tsl/protobuf/autotuning.proto" was not found or had errors.
tensorflow/compiler/xla/autotune_results.proto:42:5: "tensorflow.AutotuneResult" is not defined.

Do you know what's up with that? I indeed can't find a any.proto file in the TF repo.

@github-actions

This comment has been minimized.

@Avasam
Copy link
Collaborator

Avasam commented Oct 19, 2023

protoc --mypy_out relax_strict_optional_primitives

Not sure, I just ran it fresh locally and it seems to be just fine

@hmc-cs-mdrissi
Copy link
Contributor

hmc-cs-mdrissi commented Oct 19, 2023

I indeed can't find a any.proto file in the TF repo.

any.proto is similar to standard library protobuf file (or maybe like protobuf-dev in analogy to python-dev). It's not part of tensorflow. It's sign that your protobuf install includes compiler, but not protobuf related headers. Fix is something like apt install libprotobuf-dev.

One option to make the script more robust is maybe to instead use python -m grpc_tools.protoc and add grpcio-tools to requirements. grpcio-tools is pip package that contains protobuf compiler + related standard protobuf files like any.proto. I'm unsure there's currently documented way of how to get protoc here. pip install grpcio-tools==X would effectively pin well defined protoc version as each grpcio-tools version submodules a specific protoc version.

tensorflow/compiler/xla/xla_pb2.pyi
tensorflow/core/util/example_proto_fast_parsing_test_pb2.pyi

isort "$REPO_ROOT/stubs/tensorflow/tensorflow"
Copy link
Collaborator

@Avasam Avasam Oct 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If #10912 is merged first, this becomes:

Suggested change
isort "$REPO_ROOT/stubs/tensorflow/tensorflow"
ruff check "$REPO_ROOT/stubs/tensorflow/tensorflow" --fix-only

@nipunn1313
Copy link
Contributor

lgtm

@github-actions
Copy link
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

@JelleZijlstra JelleZijlstra merged commit f9f30cc into main Oct 22, 2023
@JelleZijlstra JelleZijlstra deleted the proto35 branch October 22, 2023 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants