Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a query over an external data source / federated data source #3393

Closed
brandondutra opened this issue May 9, 2017 · 7 comments
Closed
Assignees
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release.

Comments

@brandondutra
Copy link

Bigquery supports running queries over csv files, but I don't see this option in this bigquery api.

See https://cloud.google.com/bigquery/external-data-sources
and
https://cloud.google.com/bigquery/external-data-cloud-storage

I see example for reading csv files into a table, but I would rather not have to manage creating and deleting temporary tables.

@dhermes dhermes added the api: bigquery Issues related to the BigQuery API. label May 9, 2017
@dhermes
Copy link
Contributor

dhermes commented May 9, 2017

AFAIK this is supported out of the box, but I'm not sure how.

@tswast Do you have any docs to this effect?

@tswast
Copy link
Contributor

tswast commented May 9, 2017

No samples yet for this. It's on my todo list.

@dhermes
Copy link
Contributor

dhermes commented May 9, 2017

@tswast Is there anything you'd like to see from us, e.g. a Client.from_foo() factory constructor that allows specifying scopes for the federated sources? (Or is that necessary?)

@tswast
Copy link
Contributor

tswast commented May 9, 2017

Yeah, it is necessary, thus: #3067 The Google Cloud Platform scope covers Cloud Storage, but not Drive or Sheets. (Not sure about Bigtable)

Regarding those Client.from_foo() functions, I'd want us to make sure we pass through all the supported arguments in the google-auth library.

As far is the external data source functionality goes, the Python client libs are missing a bit it seems.

Looks like QueryJob is missing the tableDefinitions parameter.

It also looks like Table is missing external data configuration.

You can create an external data table in the UI and then query it fine, though. Once the external table definition is defined you can query it like any other table.

@brandondutra
Copy link
Author

I cannot use the UI. I'm looking to replace Google's Datalab python/bigquery api with this one.

@lukesneeringer lukesneeringer added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Aug 9, 2017
@lukesneeringer
Copy link
Contributor

Hey all,
While triaging today, I deteremined that we have two issues for this, the other being #2372. I am closing this one in favor of that one.

@dhermes
Copy link
Contributor

dhermes commented Aug 11, 2017

@lukesneeringer Agree. Both this one and #2372 are BigQuery specific. We should probably have a more generic: "non-default scopes?" issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release.
Projects
None yet
Development

No branches or pull requests

4 participants