Support XML API for Resumable Upload

[anshulpatel25]

Expected Behavior

1. Generate signed URL using gsutil.

gsutil signurl -d 500m  -c 'text/plain' \
    -m RESUMABLE ~/Downloads/key.json \
    gs://<gcs_bucket>/<file_name>.txt

2. Use the signed URL with ResumableUpload

upload = ResumableUpload(signed_url, 1024*1024)
transport = session = Session()
response = upload.initiate(transport, stream, metadata, "text/plain", total_bytes=total_bytes)  

3. Upload should execute initiate method without giving an error and return the response object.

Actual Behavior

1. Generate signed URL using gsutil.

gsutil signurl -d 600m ~/Downloads/key.json gs://<gcs_bucket>/<file_name>.txt.

2. Use the signed URL with ResumableUpload

upload = ResumableUpload(signed_url, 1024*1024)
response = upload.initiate(transport, stream, metadata, "text/plain", total_bytes=total_bytes) # transport = session = Session()

3. Getting the following exception

Traceback (most recent call last):
  File "upload_test.py", line 19, in <module>
    response = upload.initiate(transport, stream, metadata, "text/plain", total_bytes=total_bytes)
  File "/Users/anshul.patel/.local/share/virtualenvs/virtualenv-python-domIBOZP/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 327, in initiate
    self._process_initiate_response(result)
  File "/Users/anshul.patel/.local/share/virtualenvs/virtualenv-python-domIBOZP/lib/python3.7/site-packages/google/resumable_media/_upload.py", line 451, in _process_initiate_response
    response, u'location', self._get_headers)
  File "/Users/anshul.patel/.local/share/virtualenvs/virtualenv-python-domIBOZP/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 64, in header_required
    response, u'Response headers must contain header', name)
google.resumable_media.common.InvalidResponse: ('Response headers must contain header', 'location')

It seems like the ResumableUpload doesn't interpret XML API endpoint.

Specifications

• Version: 3.7
• Platform: macOS Mojave 10.14.2

[antoncohen]

I think the key thing here is that the XML API requires the header x-goog-resumable: start, while the JSON API doesn't. If ResumableUpload.initiate() set the header x-goog-resumable: start it would work with the XML API. As far as I know the XML API is the only way to do signed URLs.

[tseaver]

@anshulpatel25 Does the upload work for you if you pass in the appropriate header to the constructor, e.g.:

xml_api_headers = {'x-goog-resumable': 'start'}
upload = ResumableUpload(signed_url, 1024*1024, headers=xml_api_headers)
response = upload.initiate(transport, stream, metadata, "text/plain", total_bytes=total_bytes)

[anshulpatel25]

Hey @tseaver

Only adding {'x-goog-resumable': 'start'} doesn't work, I also required to add content-type for it to work.

Working

>>> xml_api_headers
{'x-goog-resumable': 'start', 'content-type': 'text/plain'}
>>> upload = ResumableUpload(signed_url, 1024*1024, headers=xml_api_headers)
>>> p = upload.initiate(transport, stream, metadata, file_content_type)
......
HTTP/1.1" 201 0

Not Working

>>> xml_api_headers
{'x-goog-resumable': 'start'}
>>> upload = ResumableUpload(signed_url, 1024*1024, headers=xml_api_headers)
>>> x = upload.initiate(transport, stream, metadata, file_content_type)
......
google.resumable_media.common.InvalidResponse: ('Response headers must contain header', 'location')

Thanks

[tseaver]

@anshulpatel25 Thanks for the research! Looks like #99 needs a different approach, as the Content-Type for the JSON API is required to be 'application/json; charset=UTF-8' (the metadata is sent with the initiate request as JSON).

See the XML API docs for requirements:

• Empty body (no metadata).
• Content-Length: 0 header.
• x-goog-resumable: startheader
• Optional Content-Type: <XXX> header.