export to BigQuery with partition does not seem to work

[jennylover]

Hi,

When i tried to export asset data to the BigQuery, it is working fine. But, if I add partition option, created table does not seemed partitioned in BQ.

In the SDK, there are only 3 enums are available for partition_key, is there any way to get partition by DAY?

Thanks,

[busunkim96]

@lvvvvvf I see you authored the export to BigQuery samples, could you take a look?

[lvvvvvf]

Add Ziqi who's the poc for partitioned table

[busunkim96]

@Larittic-GG Could you take a look?

[Larittic-GG]

Sorry for the late reply.

First, for "created table does not seemed partitioned in BQ", can you recheck what the table details page? The Table Type row should display "Partitioned".

Secondly, for every value of the partition key enum other than PARTITION_KEY_UNSPECIFIED, the table is actually partitioned by daily granularity. See details about BigQuery partitioned table in https://cloud.google.com/bigquery/docs/partitioned-tables.

Let me know if you have more questions. Thanks!

[kbroughton]

I found it a bit confusing that the partition key was bound inside the output_config. This code works for me

` if partition_key:
partition_spec = asset_v1.PartitionSpec()
partition_spec.partition_key = partition_key
output_config.bigquery_destination.partition_spec = partition_spec

try:
if partition_key:
output_config.bigquery_destination.table = '%s_resource' % (bq_table)
else:
output_config.bigquery_destination.table = '%s_resource_%s' % (bq_table, read_time.strftime('%Y%m%d'))
logging.info('output_config {} '.format(output_config))

  response = client.export_assets(
      request={
          'parent': parent,
          'read_time': read_time,
          'content_type': asset_v1.ContentType.RESOURCE,
          'output_config': output_config
      }
  )

`

[parthea]

I'm going to close this issue due to inactivity. Please feel free to re-open it if you have additional questions.