Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): reduce copies in InsertObject() #11014

Merged

Conversation

coryan
Copy link
Contributor

@coryan coryan commented Mar 8, 2023

We can avoid a data copy if we use absl::string_view to contain the object payload.


This change is Reviewable

We can avoid a data copy if we use `absl::string_view` to contain the
object payload.
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Mar 8, 2023
@codecov
Copy link

codecov bot commented Mar 8, 2023

Codecov Report

Patch coverage: 98.48% and no project coverage change

Comparison is base (ab55c89) 93.77% compared to head (d7ad86e) 93.77%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #11014   +/-   ##
=======================================
  Coverage   93.77%   93.77%           
=======================================
  Files        1724     1724           
  Lines      155205   155294   +89     
=======================================
+ Hits       145546   145631   +85     
- Misses       9659     9663    +4     
Impacted Files Coverage Δ
.../cloud/storage/benchmarks/throughput_experiment.cc 0.00% <0.00%> (ø)
google/cloud/storage/client.cc 85.71% <100.00%> (ø)
google/cloud/storage/client.h 99.67% <100.00%> (+<0.01%) ⬆️
google/cloud/storage/client_object_test.cc 99.47% <100.00%> (ø)
google/cloud/storage/compose_many_test.cc 100.00% <100.00%> (ø)
google/cloud/storage/hashing_options.cc 100.00% <100.00%> (ø)
google/cloud/storage/hashing_options.h 100.00% <100.00%> (ø)
google/cloud/storage/internal/curl_client.cc 98.67% <100.00%> (ø)
google/cloud/storage/internal/curl_request.cc 100.00% <100.00%> (ø)
google/cloud/storage/internal/curl_request.h 100.00% <100.00%> (ø)
... and 20 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@coryan coryan marked this pull request as ready for review March 8, 2023 20:17
@coryan coryan requested a review from a team as a code owner March 8, 2023 20:17
handle_.SetOption(CURLOPT_UPLOAD, 0L);
if (!payload.empty()) {
handle_.SetOption(CURLOPT_POSTFIELDSIZE, payload.length());
handle_.SetOption(CURLOPT_POSTFIELDS, payload.c_str());
handle_.SetOption(CURLOPT_POSTFIELDS, payload.data());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

string_view::data() without string_view::size() is a red flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a .length() already, changed to .size().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh ... sorry I missed that.

inline std::string ComputeMD5Hash(char const* payload) {
auto p =
payload == nullptr ? absl::string_view{} : absl::string_view{payload};
return ComputeMD5Hash(std::move(p));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving a string_view is an anti-pattern, I believe (et seq).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed (everywhere I hope).

@@ -80,7 +81,16 @@ inline std::string UrlsafeBase64Encode(Collection const& bytes) {
StatusOr<std::vector<std::uint8_t>> UrlsafeBase64Decode(std::string const& str);

/// Compute the MD5 hash of @p payload
std::vector<std::uint8_t> MD5Hash(std::string const& payload);
std::vector<std::uint8_t> MD5Hash(absl::Span<char const> payload);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we need to introduce a Span overload?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. I am thinking that these should be receiving "bytes" as input, but we should do that in a separate PR.

handle_.SetOption(CURLOPT_UPLOAD, 0L);
if (!payload.empty()) {
handle_.SetOption(CURLOPT_POSTFIELDSIZE, payload.length());
handle_.SetOption(CURLOPT_POSTFIELDS, payload.c_str());
handle_.SetOption(CURLOPT_POSTFIELDS, payload.data());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh ... sorry I missed that.

@coryan coryan merged commit 3797cd0 into googleapis:main Mar 9, 2023
@coryan coryan deleted the feat-storage-avoid-copies-in-InsertObject branch March 9, 2023 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants