Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow duplicate uniqueKeyPolicy in DocumentCollection - Part 2 #42244

Merged

Conversation

xinlian12
Copy link
Member

@xinlian12 xinlian12 commented Oct 9, 2024

Description

This PR address a customer-reported issue of a transient error of duplicate UniqueKeyPolicy.

Reported exception:

2024-08-19 12:11:41,861 ERROR Executor [Executor task launch worker for task 3.0 in stage 26.0 (TID 23809)]: Exception in task 3.0 in stage 26.0 (TID 23809)
java.lang.IllegalArgumentException: Unable to parse JSON {"id":"COLLECTIONNAME","indexingPolicy":{"indexingMode":"consistent","automatic":true,"includedPaths":[{"path":"/*"}],"excludedPaths":[{"path":"/\"_etag\"/?"}],"spatialIndexes":[{"path":"/*","types":["Point","LineString","Polygon","MultiPolygon"]}]},"partitionKey":{"paths":["/pk"],"kind":"Hash","systemKey":false},"defaultTtl":2764800,"conflictResolutionPolicy":{"mode":"LastWriterWins","conflictResolutionPath":"/_ts","conflictResolutionProcedure":""},"geospatialConfig":{"type":"Geography"},"_rid":"RID","_ts":1587070752,"_self":"dbs/DBRID/colls/COLLECTIONRID/","_etag":"\"ETAG\"","_docs":"docs/","_sprocs":"sprocs/","_triggers":"triggers/","_udfs":"udfs/","_conflicts":"conflicts/","uniqueKeyPolicy":{},"uniqueKeyPolicy":{}}
	at azure_cosmos_spark.com.azure.cosmos.implementation.JsonSerializable.fromJson(JsonSerializable.java:645)
	at azure_cosmos_spark.com.azure.cosmos.implementation.JsonSerializable.<init>(JsonSerializable.java:84)
	at azure_cosmos_spark.com.azure.cosmos.implementation.Resource.<init>(Resource.java:83)
	at azure_cosmos_spark.com.azure.cosmos.implementation.DocumentCollection.<init>(DocumentCollection.java:81)
	at azure_cosmos_spark.com.azure.cosmos.models.CosmosContainerProperties.<init>(CosmosContainerProperties.java:66)
	at azure_cosmos_spark.com.azure.cosmos.models.CosmosContainerProperties$1.create(CosmosContainerProperties.java:414)
	at azure_cosmos_spark.com.azure.cosmos.SparkBridgeInternal$.getContainerPropertiesFromCollectionCache(SparkBridgeInternal.scala:102)
	at com.azure.cosmos.spark.CosmosWriterBase.<init>(CosmosWriterBase.scala:58)
	at com.azure.cosmos.spark.CosmosWriter.<init>(CosmosWriter.scala:34)
	at com.azure.cosmos.spark.ItemsDataWriteFactory.createWriter(ItemsDataWriteFactory.scala:51)
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:439)
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:390)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

The duplicated "uniqueKeyPolicy":{},"uniqueKeyPolicy":{} is unexpected.

@xinlian12 xinlian12 requested review from kirankumarkolli and a team as code owners October 9, 2024 23:09
@github-actions github-actions bot added the Cosmos label Oct 9, 2024
@azure-sdk
Copy link
Collaborator

API change check

API changes are not detected in this pull request.

@xinlian12 xinlian12 merged commit 5d4b102 into Azure:main Oct 10, 2024
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants