Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per-partition automatic failover #41029

Draft
wants to merge 51 commits into
base: main
Choose a base branch
from

Conversation

jeet1995
Copy link
Member

@jeet1995 jeet1995 commented Jul 6, 2024

Background

Refer issue: #43143

Important classes introduced

GlobalPartitionEndpointManagerForPerPartitionAutomaticFailover

This class stores failover information for a partition key range and collection resource id combination. The failover information is stored using a ConcurrentHashMap where the key is of type PartitionKeyRangeWrapper and value is of type PartitionLevelFailoverInfo. Dependent classes use this information to override the available region for the partition key range and collection resource id combination encapsulated in PartitionKeyRangeWrapper.

Important Sequence diagrams

Reacting to relevant error codes

sequenceDiagram
    Direct transport -->> ClientRetryPolicy: A 403/3 or a 503 or a 408 could be returned from the transport.
    ClientRetryPolicy ->> GPEMForPerPartitionAutomaticFailover: tryMarkPartitionKeyRangeAsUnavailable 
Loading

Adding location override for a data-plane request routed to relevant partition key range

sequenceDiagram 
    CosmosContainer ->> RxDocumentClientImpl: executeSomeDataPlaneOperation (illustrative method name)
    CosmosAsyncContainer ->> RxDocumentClientImpl: executeSomeDataPlaneOperation (illustrative method name)
    RxDocumentClientImpl ->> RxDocumentClientImpl: setPartitionKeyRangeForSomeDataPlaneRequestForPerPartitionAutomaticFailover (illustrative method name)
    RxDocumentClientImpl ->> ClientRetryPolicy: onBeforeSendRequest
    ClientRetryPolicy ->> GlobalEndpointManager: resolveServiceEndoint
    ClientRetryPolicy ->> GPEMForPerPartitionAutomaticFailover: tryOverrideLocationForPartitionKeyRange
Loading

Important Activity Diagrams

Marking a partition key range as Unavailable for a given location

flowchart TD
    A[GPEMForPerPartitionAutomaticFailover:tryMarkPartitionKeyRangeAsUnavailable] --> B{Is per-partition automatic failover enabled?}
    B --> |Yes|C{Is readonly request?}
    B --> |No|D[Return]
    C -->|Yes|D
    C -->|No|E{Is count of applicable read endpoints <= 1?}
    E -->|Yes|D
    E -->|No|F{Is resource type a Document or operation type and execution of stored procedure?}
    F -->|No|D
    F -->|Yes|G{Is the account a multi-write account?}
    G -->|Yes|D
    G -->|No|H[Conclude per-partition automatic failover is applicable]
    H -->I[Mark location as Unavailable and mark next account-level read location as location to override.]
Loading

Client-level settings

To enable per-partition automatic failover from the perspective of the client instance, use the below system property.

System.setProperty("COSMOS.IS_PER_PARTITION_AUTOMATIC_FAILOVER_ENABLED", "true");

Testing outcomes

In all cases, an upsert workload is run for some arbitrary duration. This workload is run against a Strong consistency single-write account with 3 regions - NorthCentralUs, WestUs2, CentralUs.

Inject quorum loss into a server partition in the primary region as a warm client is running the workload.

In this scenario, complete quorum loss is injected into a server partition in NorthCentralUs. The upsert operation against the affected server partition is failed over to WestUs2 from which a success is received.

{
    "userAgent": "azsdk-java-cosmos/4.65.0-beta.1 Windows11/10.0 JRE/17.0.13",
    "activityId": "ce196f6b-6c66-41f2-b4f3-03f6d275287d",
    "requestLatencyInMs": 124322,
    "requestStartTimeUTC": "2024-12-01T16:31:45.894292Z",
    "requestEndTimeUTC": "2024-12-01T16:33:50.216905100Z",
    "responseStatisticsList": [
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-northcentralus1-be1.documents-test.windows-int.net:14306/apps/52408f3e-7918-434b-b59e-5a16313cec2c/services/a777cc3a-919a-4580-b77e-5dee3b560003/partitions/51df8419-15a3-4135-84ca-9115e25c60da/replicas/133770483916092352p/",
                "lsn": 10476,
                "quorumAckedLSN": 10476,
                "currentReplicaSetSize": 3,
                "globalCommittedLsn": 10476,
                "partitionKeyRangeId": null,
                "isValid": true,
                "statusCode": 410,
                "subStatusCode": 21005,
                "isGone": true,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": -1,
                "sessionToken": null,
                "backendLatencyInMs": 0.083,
                "retryAfterInMs": null,
                "exceptionMessage": "[\"The requested resource is no longer available at the server.\"]",
                "exceptionResponseHeaders": "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Sat, 30 Nov 2024 23:01:54.618 GMT, x-ms-request-duration-ms=0.083, lsn=10476, x-ms-schemaversion=1.18, x-ms-transport-request-id=514, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=10476, x-ms-quorum-acked-lsn=10476, x-ms-activity-id=563df6b7-3893-48cf-93f5-8cecc5b4e6de, x-ms-xp-role=1, x-ms-global-Committed-lsn=10476, x-ms-cosmos-llsn=10476, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=21005}",
                "replicaStatusList": {
                    "Ignoring": [
                        "14009:S:Connected",
                        "14303:S:Connected",
                        "14002:S:Connected"
                    ],
                    "Attempting": [
                        "14306:P:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:31:45.897901500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:31:45.897901500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:31:45.897901500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:31:45.897901500Z",
                        "durationInMilliSecs": 1.147
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:31:45.899048500Z",
                        "durationInMilliSecs": 24.8687
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:31:45.923917200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:31:45.923917200Z",
                        "durationInMilliSecs": 15.6799
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:31:45.939597100Z",
                        "durationInMilliSecs": 1.353
                    }
                ],
                "rntbdRequestLengthInBytes": 418,
                "rntbdResponseLengthInBytes": 259,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 259,
                "channelStatistics": {
                    "channelId": "17573c20",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:31:45.892860400Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:31:45.894Z",
                    "lastRequestTime": "2024-12-01T16:31:45.866Z",
                    "createdTime": "2024-12-01T16:31:06.781255Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:31:45.941946600Z",
            "requestStartTimeUTC": "2024-12-01T16:31:45.897901500Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14004/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488721995992p/",
                "lsn": 10476,
                "quorumAckedLSN": 10476,
                "currentReplicaSetSize": 3,
                "globalCommittedLsn": 10476,
                "partitionKeyRangeId": null,
                "isValid": true,
                "statusCode": 403,
                "subStatusCode": 3,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": -1,
                "sessionToken": ":-1#10476",
                "backendLatencyInMs": 0.207,
                "retryAfterInMs": null,
                "exceptionMessage": "[\"The requested operation cannot be performed at this region\"]",
                "exceptionResponseHeaders": "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Mon, 25 Nov 2024 22:54:36.599 GMT, x-ms-request-duration-ms=0.207, x-ms-session-token=:-1#10476, lsn=10476, x-ms-schemaversion=1.18, x-ms-transport-request-id=515, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=10476, x-ms-quorum-acked-lsn=10476, x-ms-activity-id=ff398259-6077-4973-8ac4-3e7128b833e8, x-ms-xp-role=4, x-ms-global-Committed-lsn=10476, x-ms-cosmos-llsn=10476, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=3}",
                "replicaStatusList": {
                    "Ignoring": [
                        "14308:S:Unknown",
                        "14300:S:Unknown",
                        "14307:S:Unknown"
                    ],
                    "Attempting": [
                        "14004:P:Unknown"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:32:46.895774Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:32:46.895774Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:32:46.895774Z",
                        "durationInMilliSecs": 344.7475
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:32:47.240521500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:32:47.240521500Z",
                        "durationInMilliSecs": 36.0849
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:32:47.276606400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:32:47.276606400Z",
                        "durationInMilliSecs": 1.0156
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:32:47.277622Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 418,
                "rntbdResponseLengthInBytes": 277,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 277,
                "channelStatistics": {
                    "channelId": "b460d5af",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:32:47.240521500Z",
                    "waitForConnectionInit": true
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 0,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:32:46.895Z",
                    "lastRequestTime": "2024-12-01T16:32:46.895Z",
                    "createdTime": "2024-12-01T16:32:46.895774Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:32:47.278582Z",
            "requestStartTimeUTC": "2024-12-01T16:32:46.895774Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-westus2-be1.documents-test.windows-int.net:14005/apps/82e77298-6922-4b21-a250-f420f9efbd9e/services/726bb931-5b5c-488b-bc6a-37be8b795601/partitions/fadc715d-9328-456d-8386-2e680f786f96/replicas/133770493853234127p/",
                "lsn": 10476,
                "quorumAckedLSN": 10476,
                "currentReplicaSetSize": 3,
                "globalCommittedLsn": 10476,
                "partitionKeyRangeId": null,
                "isValid": true,
                "statusCode": 403,
                "subStatusCode": 3,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": -1,
                "sessionToken": ":-1#10476",
                "backendLatencyInMs": 0.075,
                "retryAfterInMs": null,
                "exceptionMessage": "[\"The requested operation cannot be performed at this region\"]",
                "exceptionResponseHeaders": "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Sat, 30 Nov 2024 23:03:15.376 GMT, x-ms-request-duration-ms=0.075, x-ms-session-token=:-1#10476, lsn=10476, x-ms-schemaversion=1.18, x-ms-transport-request-id=516, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=10476, x-ms-quorum-acked-lsn=10476, x-ms-activity-id=18ee0c3c-ecfc-4f57-9a4f-754ca44c6bab, x-ms-xp-role=4, x-ms-global-Committed-lsn=10476, x-ms-cosmos-llsn=10476, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=3}",
                "replicaStatusList": {
                    "Ignoring": [
                        "14303:S:Unknown",
                        "14004:S:Unknown",
                        "14001:S:Unknown"
                    ],
                    "Attempting": [
                        "14005:P:Unknown"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:32:47.818232700Z",
                        "durationInMilliSecs": 1.1088
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:32:47.819341500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:32:47.819341500Z",
                        "durationInMilliSecs": 428.374
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:32:48.247715500Z",
                        "durationInMilliSecs": 1.992
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:32:48.249707500Z",
                        "durationInMilliSecs": 72.915
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:32:48.322622500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:32:48.322622500Z",
                        "durationInMilliSecs": 1.0009
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:32:48.323623400Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 420,
                "rntbdResponseLengthInBytes": 277,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 277,
                "channelStatistics": {
                    "channelId": "957f6b5a",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:32:48.245707800Z",
                    "waitForConnectionInit": true
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 0,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:32:47.818Z",
                    "lastRequestTime": "2024-12-01T16:32:47.818Z",
                    "createdTime": "2024-12-01T16:32:47.818232700Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:32:48.323623400Z",
            "requestStartTimeUTC": "2024-12-01T16:32:47.818232700Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14004/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488721995992p/",
                "lsn": 10478,
                "quorumAckedLSN": 10478,
                "currentReplicaSetSize": 3,
                "globalCommittedLsn": 10478,
                "partitionKeyRangeId": null,
                "isValid": true,
                "statusCode": 403,
                "subStatusCode": 3,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": -1,
                "sessionToken": ":-1#10478",
                "backendLatencyInMs": 0.096,
                "retryAfterInMs": null,
                "exceptionMessage": "[\"The requested operation cannot be performed at this region\"]",
                "exceptionResponseHeaders": "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Mon, 25 Nov 2024 22:54:36.599 GMT, x-ms-request-duration-ms=0.096, x-ms-session-token=:-1#10478, lsn=10478, x-ms-schemaversion=1.18, x-ms-transport-request-id=517, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=10478, x-ms-quorum-acked-lsn=10478, x-ms-activity-id=b99b3238-05b5-4d75-87b6-14cc03c7c792, x-ms-xp-role=4, x-ms-global-Committed-lsn=10478, x-ms-cosmos-llsn=10478, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=3}",
                "replicaStatusList": {
                    "Ignoring": [
                        "14308:S:Unknown",
                        "14300:S:Unknown",
                        "14307:S:Unknown"
                    ],
                    "Attempting": [
                        "14004:P:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:33:49.355385300Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:33:49.355385300Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:33:49.355385300Z",
                        "durationInMilliSecs": 1.4194
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:33:49.356804700Z",
                        "durationInMilliSecs": 28.9217
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:33:49.385726400Z",
                        "durationInMilliSecs": 34.054
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:33:49.419780400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:33:49.419780400Z",
                        "durationInMilliSecs": 0.9998
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:33:49.420780200Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 422,
                "rntbdResponseLengthInBytes": 277,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 277,
                "channelStatistics": {
                    "channelId": "b460d5af",
                    "channelTaskQueueSize": 0,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:32:47.276606400Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:32:46.895Z",
                    "lastRequestTime": "2024-12-01T16:32:46.895Z",
                    "createdTime": "2024-12-01T16:32:46.895774Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:33:49.420780200Z",
            "requestStartTimeUTC": "2024-12-01T16:33:49.355385300Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-westus2-be1.documents-test.windows-int.net:14005/apps/82e77298-6922-4b21-a250-f420f9efbd9e/services/726bb931-5b5c-488b-bc6a-37be8b795601/partitions/fadc715d-9328-456d-8386-2e680f786f96/replicas/133770493853234127p/",
                "lsn": 10479,
                "quorumAckedLSN": 10478,
                "currentReplicaSetSize": 4,
                "globalCommittedLsn": 10478,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 200,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 10.29,
                "itemLSN": -1,
                "sessionToken": "0:-1#10479",
                "backendLatencyInMs": 5.434,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [
                        "14303:S:Unknown",
                        "14004:S:Unknown",
                        "14001:S:Unknown"
                    ],
                    "Attempting": [
                        "14005:P:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:33:49.428775900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:33:49.428775900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:33:49.428775900Z",
                        "durationInMilliSecs": 0.9385
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:33:49.429714400Z",
                        "durationInMilliSecs": 64.6587
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:33:49.494373100Z",
                        "durationInMilliSecs": 74.3993
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:33:49.568772400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:33:49.568772400Z",
                        "durationInMilliSecs": 1.2393
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:33:49.570011700Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 422,
                "rntbdResponseLengthInBytes": 449,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "957f6b5a",
                    "channelTaskQueueSize": 0,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:32:48.322622500Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:32:47.818Z",
                    "lastRequestTime": "2024-12-01T16:32:47.818Z",
                    "createdTime": "2024-12-01T16:32:47.818232700Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:33:49.570011700Z",
            "requestStartTimeUTC": "2024-12-01T16:33:49.428775900Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        }
    ],
    "supplementalResponseStatisticsList": [
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-westus2-be1.documents-test.windows-int.net:14001/apps/82e77298-6922-4b21-a250-f420f9efbd9e/services/726bb931-5b5c-488b-bc6a-37be8b795601/partitions/fadc715d-9328-456d-8386-2e680f786f96/replicas/133770493853234129s/",
                "lsn": 10479,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 10479,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#10479",
                "backendLatencyInMs": 0.302,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14005:P:Connected",
                        "14303:S:Unknown",
                        "14004:S:Unknown",
                        "14001:S:Unknown"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T16:33:49.570588400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T16:33:49.570588400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T16:33:49.570588400Z",
                        "durationInMilliSecs": 516.8994
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T16:33:50.087487800Z",
                        "durationInMilliSecs": 26.7806
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T16:33:50.114268400Z",
                        "durationInMilliSecs": 102.6367
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T16:33:50.216905100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T16:33:50.216905100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T16:33:50.216905100Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "c29ac23c",
                    "channelTaskQueueSize": 0,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T16:33:50.087487800Z",
                    "waitForConnectionInit": true
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 0,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T16:33:49.570Z",
                    "lastRequestTime": "2024-12-01T16:33:49.570Z",
                    "createdTime": "2024-12-01T16:33:49.570588400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T16:33:50.216905100Z",
            "requestStartTimeUTC": "2024-12-01T16:33:49.570588400Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        }
    ],
    "addressResolutionStatistics": {
        "3a6cb451-6e8b-449f-aa06-deda285e2fc2": {
            "startTimeUTC": "2024-12-01T16:32:50.051223Z",
            "endTimeUTC": "2024-12-01T16:32:50.084312800Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "5d72eba7-7458-482d-b265-61447165281a": {
            "startTimeUTC": "2024-12-01T16:32:47.286591200Z",
            "endTimeUTC": "2024-12-01T16:32:47.818232700Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": false,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "54717938-c77e-419a-83d7-0314e1a4d342": {
            "startTimeUTC": "2024-12-01T16:32:48.992528900Z",
            "endTimeUTC": "2024-12-01T16:32:49.033836700Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "256022e2-8120-4433-a589-ae476030b11c": {
            "startTimeUTC": "2024-12-01T16:31:49.057578700Z",
            "endTimeUTC": "2024-12-01T16:31:49.091246400Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "ce256d9f-8099-4f64-8d25-b92d9d549d62": {
            "startTimeUTC": "2024-12-01T16:32:46.298811700Z",
            "endTimeUTC": "2024-12-01T16:32:46.334753Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "effc6bd1-b37e-44e4-923c-7417811b180d": {
            "startTimeUTC": "2024-12-01T16:33:34.262698800Z",
            "endTimeUTC": "2024-12-01T16:33:34.297385Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "dc6191c7-96bd-498c-b795-e7891fcc7702": {
            "startTimeUTC": "2024-12-01T16:32:56.127582300Z",
            "endTimeUTC": "2024-12-01T16:32:56.161588200Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "68220201-1bbb-4313-bcbe-b99394dd098b": {
            "startTimeUTC": "2024-12-01T16:33:04.171663700Z",
            "endTimeUTC": "2024-12-01T16:33:04.209054200Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "27a0e865-dbde-4c3e-9c7f-e057de061fab": {
            "startTimeUTC": "2024-12-01T16:31:53.097779700Z",
            "endTimeUTC": "2024-12-01T16:31:53.132249400Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "01bb65a1-689b-439d-8b87-476d048ec2cb": {
            "startTimeUTC": "2024-12-01T16:33:49.311376Z",
            "endTimeUTC": "2024-12-01T16:33:49.346039400Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "84c63ce5-1798-4f7b-8a08-14cdde149568": {
            "startTimeUTC": "2024-12-01T16:31:45.954549900Z",
            "endTimeUTC": "2024-12-01T16:31:45.992403600Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "07463c91-7504-4274-973f-11f1de6044b5": {
            "startTimeUTC": "2024-12-01T16:31:47.009769400Z",
            "endTimeUTC": "2024-12-01T16:31:47.042997700Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "51abd5b4-5312-4c81-a23f-32c19f9db4a4": {
            "startTimeUTC": "2024-12-01T16:32:46.448755600Z",
            "endTimeUTC": "2024-12-01T16:32:46.894767Z",
            "targetEndpoint": "https://abhm-ppaf-testing-centralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": false,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "3f98badf-9fb7-45a4-88f5-6ce1495235ea": {
            "startTimeUTC": "2024-12-01T16:32:52.089978400Z",
            "endTimeUTC": "2024-12-01T16:32:52.125074400Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "5ee08058-9945-4ba7-8e4d-5bf8a968149f": {
            "startTimeUTC": "2024-12-01T16:32:31.242114700Z",
            "endTimeUTC": "2024-12-01T16:32:31.281500200Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "a2f85445-69a8-46d9-90b2-0cce4dee9fba": {
            "startTimeUTC": "2024-12-01T16:33:19.220346600Z",
            "endTimeUTC": "2024-12-01T16:33:19.257456500Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "e7fd6da9-896f-476f-9f60-4c6116835040": {
            "startTimeUTC": "2024-12-01T16:32:01.144695900Z",
            "endTimeUTC": "2024-12-01T16:32:01.177759500Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "6287ac62-c8aa-4f96-8102-59df14534b24": {
            "startTimeUTC": "2024-12-01T16:32:16.188582300Z",
            "endTimeUTC": "2024-12-01T16:32:16.224654500Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        }
    },
    "regionsContacted": [
        "central us",
        "north central us",
        "west us 2"
    ],
    "retryContext": {
        "statusAndSubStatusCodes": [
            [
                410,
                21005
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                503,
                0
            ],
            [
                403,
                3
            ],
            [
                403,
                3
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                503,
                0
            ],
            [
                403,
                3
            ]
        ],
        "retryCount": 23,
        "retryLatency": 124276
    },
    "metadataDiagnosticsContext": {
        "metadataDiagnosticList": [
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:31:45.954549900Z",
                "endTimeUTC": "2024-12-01T16:31:45.992403600Z",
                "durationinMS": 37
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:31:47.009769400Z",
                "endTimeUTC": "2024-12-01T16:31:47.042997700Z",
                "durationinMS": 33
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:31:49.057578700Z",
                "endTimeUTC": "2024-12-01T16:31:49.091246400Z",
                "durationinMS": 33
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:31:53.097779700Z",
                "endTimeUTC": "2024-12-01T16:31:53.132249400Z",
                "durationinMS": 34
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:01.144695900Z",
                "endTimeUTC": "2024-12-01T16:32:01.177759500Z",
                "durationinMS": 33
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:16.188582300Z",
                "endTimeUTC": "2024-12-01T16:32:16.224654500Z",
                "durationinMS": 36
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:31.242114700Z",
                "endTimeUTC": "2024-12-01T16:32:31.281500200Z",
                "durationinMS": 39
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:46.298811700Z",
                "endTimeUTC": "2024-12-01T16:32:46.334753Z",
                "durationinMS": 35
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:46.448755600Z",
                "endTimeUTC": "2024-12-01T16:32:46.893766500Z",
                "durationinMS": 445
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:47.286591200Z",
                "endTimeUTC": "2024-12-01T16:32:47.818232700Z",
                "durationinMS": 531
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:48.992528900Z",
                "endTimeUTC": "2024-12-01T16:32:49.033836700Z",
                "durationinMS": 41
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:50.051223Z",
                "endTimeUTC": "2024-12-01T16:32:50.084312800Z",
                "durationinMS": 33
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:52.089978400Z",
                "endTimeUTC": "2024-12-01T16:32:52.125074400Z",
                "durationinMS": 35
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:32:56.127582300Z",
                "endTimeUTC": "2024-12-01T16:32:56.161588200Z",
                "durationinMS": 34
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:33:04.171663700Z",
                "endTimeUTC": "2024-12-01T16:33:04.209054200Z",
                "durationinMS": 37
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:33:19.220346600Z",
                "endTimeUTC": "2024-12-01T16:33:19.257456500Z",
                "durationinMS": 37
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:33:34.262698800Z",
                "endTimeUTC": "2024-12-01T16:33:34.297385Z",
                "durationinMS": 34
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T16:33:49.311376Z",
                "endTimeUTC": "2024-12-01T16:33:49.346039400Z",
                "durationinMS": 34
            }
        ]
    },
    "serializationDiagnosticsContext": {
        "serializationDiagnosticsList": [
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:31:45.895366500Z",
                "endTimeUTC": "2024-12-01T16:31:45.895366500Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:32:46.447323800Z",
                "endTimeUTC": "2024-12-01T16:32:46.447323800Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:32:47.284596100Z",
                "endTimeUTC": "2024-12-01T16:32:47.284596100Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:32:48.990517300Z",
                "endTimeUTC": "2024-12-01T16:32:48.990517300Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:33:49.355385300Z",
                "endTimeUTC": "2024-12-01T16:33:49.355385300Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T16:33:49.427776100Z",
                "endTimeUTC": "2024-12-01T16:33:49.427776100Z",
                "durationInMilliSecs": 0.0
            }
        ]
    },
    "gatewayStatisticsList": [],
    "samplingRateSnapshot": 1.0,
    "bloomFilterInsertionCountSnapshot": 0,
    "systemInformation": {
        "usedMemory": "175371 KB",
        "availableMemory": "12333813 KB",
        "systemCpuLoad": "empty",
        "availableProcessors": 8
    },
    "clientCfgs": {
        "id": 1,
        "machineId": "uuid:0bb75aba-382c-47e3-a24d-400179178303",
        "connectionMode": "DIRECT",
        "numberOfClients": 0,
        "excrgns": "[]",
        "clientEndpoints": {
            "https://abhm-ppaf-testing.documents-test.windows-int.net:443/": 1
        },
        "connCfg": {
            "rntbd": "(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)",
            "gw": "(cps:1000, nrto:PT1M, icto:PT1M, p:false)",
            "other": "(ed: true, cs: false, rv: true)"
        },
        "consistencyCfg": "(consistency: null, mm: true, prgns: [northcentralus,westus2,centralus])",
        "proactiveInitCfg": "",
        "e2ePolicyCfg": "",
        "sessionRetryCfg": "",
        "partitionLevelCircuitBreakerCfg": "(cb: false, type: CONSECUTIVE_EXCEPTION_COUNT_BASED, rexcntt: 10, wexcntt: 5)"
    }
}

Inject quorum loss into a server partition in the primary region after which a cold client starts running the workload.

Inject quorum loss into a server partition and master in the primary region after which a cold client starts running the workload.

In this scenario, a server partition and master partition in NorthCentralUs are injected with quorum loss. Post this, a workload is started. From the diagnostics, a success response is obtained from WestUs2.

{
    "userAgent" : "azsdk-java-cosmos/4.65.0-beta.1 Windows11/10.0 JRE/17.0.13",
    "activityId" : "766cb240-a7bc-4a41-a5e3-d02542849bbd",
    "requestLatencyInMs" : 82064,
    "requestStartTimeUTC" : "2024-12-01T18:00:35.518703800Z",
    "requestEndTimeUTC" : "2024-12-01T18:01:57.583185300Z",
    "responseStatisticsList" : [ {
      "storeResult" : {
        "storePhysicalAddress" : "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14004/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488721995992p/",
        "lsn" : 15635,
        "quorumAckedLSN" : 15635,
        "currentReplicaSetSize" : 3,
        "globalCommittedLsn" : 15635,
        "partitionKeyRangeId" : null,
        "isValid" : true,
        "statusCode" : 403,
        "subStatusCode" : 3,
        "isGone" : false,
        "isNotFound" : false,
        "isInvalidPartition" : false,
        "isThroughputControlRequestRateTooLarge" : false,
        "requestCharge" : 0.0,
        "itemLSN" : -1,
        "sessionToken" : ":-1#15635",
        "backendLatencyInMs" : 0.202,
        "retryAfterInMs" : null,
        "exceptionMessage" : "[\"The requested operation cannot be performed at this region\"]",
        "exceptionResponseHeaders" : "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Mon, 25 Nov 2024 22:54:36.599 GMT, x-ms-request-duration-ms=0.202, x-ms-session-token=:-1#15635, lsn=15635, x-ms-schemaversion=1.18, x-ms-transport-request-id=3, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=15635, x-ms-quorum-acked-lsn=15635, x-ms-activity-id=ce7b8d38-fb1c-4213-ba58-d36455bb94c4, x-ms-xp-role=4, x-ms-global-Committed-lsn=15635, x-ms-cosmos-llsn=15635, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=3}",
        "replicaStatusList" : {
          "Ignoring" : [ "14308:S:Unknown", "14300:S:Unknown", "14307:S:Unknown" ],
          "Attempting" : [ "14004:P:Unknown" ]
        },
        "transportRequestTimeline" : [ {
          "eventName" : "created",
          "startTimeUTC" : "2024-12-01T18:00:54.270315900Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "queued",
          "startTimeUTC" : "2024-12-01T18:00:54.270315900Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "channelAcquisitionStarted",
          "startTimeUTC" : "2024-12-01T18:00:54.270315900Z",
          "durationInMilliSecs" : 428.4701
        }, {
          "eventName" : "pipelined",
          "startTimeUTC" : "2024-12-01T18:00:54.698786Z",
          "durationInMilliSecs" : 5.3216
        }, {
          "eventName" : "transitTime",
          "startTimeUTC" : "2024-12-01T18:00:54.704107600Z",
          "durationInMilliSecs" : 102.162
        }, {
          "eventName" : "decodeTime",
          "startTimeUTC" : "2024-12-01T18:00:54.806269600Z",
          "durationInMilliSecs" : 16.033
        }, {
          "eventName" : "received",
          "startTimeUTC" : "2024-12-01T18:00:54.822302600Z",
          "durationInMilliSecs" : 6.0454
        }, {
          "eventName" : "completed",
          "startTimeUTC" : "2024-12-01T18:00:54.828348Z",
          "durationInMilliSecs" : 0.0
        } ],
        "rntbdRequestLengthInBytes" : 420,
        "rntbdResponseLengthInBytes" : 277,
        "requestPayloadLengthInBytes" : 10,
        "responsePayloadLengthInBytes" : 277,
        "channelStatistics" : {
          "channelId" : "3329d987",
          "channelTaskQueueSize" : 0,
          "pendingRequestsCount" : 0,
          "lastReadTime" : "2024-12-01T18:00:54.698786Z",
          "waitForConnectionInit" : true
        },
        "serviceEndpointStatistics" : {
          "availableChannels" : 0,
          "acquiredChannels" : 0,
          "executorTaskQueueSize" : 0,
          "inflightRequests" : 1,
          "lastSuccessfulRequestTime" : "2024-12-01T18:00:54.274Z",
          "lastRequestTime" : "2024-12-01T18:00:54.274Z",
          "createdTime" : "2024-12-01T18:00:54.270315900Z",
          "isClosed" : false,
          "cerMetrics" : { }
        }
      },
      "requestResponseTimeUTC" : "2024-12-01T18:00:54.828348Z",
      "requestStartTimeUTC" : "2024-12-01T18:00:54.270315900Z",
      "requestResourceType" : "Document",
      "requestOperationType" : "Upsert",
      "requestSessionToken" : null,
      "e2ePolicyCfg" : null,
      "excludedRegions" : null,
      "sessionTokenEvaluationResults" : [ ],
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    }, {
      "storeResult" : {
        "storePhysicalAddress" : "rntbd://cdb-ms-test61-westus2-be1.documents-test.windows-int.net:14303/apps/82e77298-6922-4b21-a250-f420f9efbd9e/services/726bb931-5b5c-488b-bc6a-37be8b795601/partitions/fadc715d-9328-456d-8386-2e680f786f96/replicas/133770493853234128p/",
        "lsn" : 15636,
        "quorumAckedLSN" : 15635,
        "currentReplicaSetSize" : 4,
        "globalCommittedLsn" : 15635,
        "partitionKeyRangeId" : "0",
        "isValid" : true,
        "statusCode" : 200,
        "subStatusCode" : 0,
        "isGone" : false,
        "isNotFound" : false,
        "isInvalidPartition" : false,
        "isThroughputControlRequestRateTooLarge" : false,
        "requestCharge" : 10.29,
        "itemLSN" : -1,
        "sessionToken" : "0:-1#15636",
        "backendLatencyInMs" : 4.966,
        "retryAfterInMs" : null,
        "replicaStatusList" : {
          "Ignoring" : [ "14005:S:Unknown", "14004:S:Unknown", "14001:S:Unknown" ],
          "Attempting" : [ "14303:P:Unknown" ]
        },
        "transportRequestTimeline" : [ {
          "eventName" : "created",
          "startTimeUTC" : "2024-12-01T18:01:56.555157600Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "queued",
          "startTimeUTC" : "2024-12-01T18:01:56.555157600Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "channelAcquisitionStarted",
          "startTimeUTC" : "2024-12-01T18:01:56.555157600Z",
          "durationInMilliSecs" : 699.1599
        }, {
          "eventName" : "pipelined",
          "startTimeUTC" : "2024-12-01T18:01:57.254317500Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "transitTime",
          "startTimeUTC" : "2024-12-01T18:01:57.254317500Z",
          "durationInMilliSecs" : 155.9729
        }, {
          "eventName" : "decodeTime",
          "startTimeUTC" : "2024-12-01T18:01:57.410290400Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "received",
          "startTimeUTC" : "2024-12-01T18:01:57.410290400Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "completed",
          "startTimeUTC" : "2024-12-01T18:01:57.410290400Z",
          "durationInMilliSecs" : 0.0
        } ],
        "rntbdRequestLengthInBytes" : 416,
        "rntbdResponseLengthInBytes" : 449,
        "requestPayloadLengthInBytes" : 10,
        "responsePayloadLengthInBytes" : 0,
        "channelStatistics" : {
          "channelId" : "535bce0f",
          "channelTaskQueueSize" : 1,
          "pendingRequestsCount" : 0,
          "lastReadTime" : "2024-12-01T18:01:57.254317500Z",
          "waitForConnectionInit" : true
        },
        "serviceEndpointStatistics" : {
          "availableChannels" : 0,
          "acquiredChannels" : 0,
          "executorTaskQueueSize" : 0,
          "inflightRequests" : 1,
          "lastSuccessfulRequestTime" : "2024-12-01T18:01:56.558Z",
          "lastRequestTime" : "2024-12-01T18:01:56.558Z",
          "createdTime" : "2024-12-01T18:01:56.555157600Z",
          "isClosed" : false,
          "cerMetrics" : { }
        }
      },
      "requestResponseTimeUTC" : "2024-12-01T18:01:57.410290400Z",
      "requestStartTimeUTC" : "2024-12-01T18:01:56.555157600Z",
      "requestResourceType" : "Document",
      "requestOperationType" : "Upsert",
      "requestSessionToken" : null,
      "e2ePolicyCfg" : null,
      "excludedRegions" : null,
      "sessionTokenEvaluationResults" : [ ],
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    } ],
    "supplementalResponseStatisticsList" : [ {
      "storeResult" : {
        "storePhysicalAddress" : "rntbd://cdb-ms-test61-westus2-be1.documents-test.windows-int.net:14303/apps/82e77298-6922-4b21-a250-f420f9efbd9e/services/726bb931-5b5c-488b-bc6a-37be8b795601/partitions/fadc715d-9328-456d-8386-2e680f786f96/replicas/133770493853234128p/",
        "lsn" : 15636,
        "quorumAckedLSN" : 15636,
        "currentReplicaSetSize" : 4,
        "globalCommittedLsn" : 15636,
        "partitionKeyRangeId" : "0",
        "isValid" : true,
        "statusCode" : 204,
        "subStatusCode" : 0,
        "isGone" : false,
        "isNotFound" : false,
        "isInvalidPartition" : false,
        "isThroughputControlRequestRateTooLarge" : false,
        "requestCharge" : 0.0,
        "itemLSN" : 1,
        "sessionToken" : "0:-1#15636",
        "backendLatencyInMs" : 0.222,
        "retryAfterInMs" : null,
        "replicaStatusList" : {
          "Ignoring" : [ ],
          "Attempting" : [ "14005:S:Unknown", "14303:P:Connected", "14004:S:Unknown", "14001:S:Unknown" ]
        },
        "transportRequestTimeline" : [ {
          "eventName" : "created",
          "startTimeUTC" : "2024-12-01T18:01:57.421737700Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "queued",
          "startTimeUTC" : "2024-12-01T18:01:57.421737700Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "channelAcquisitionStarted",
          "startTimeUTC" : "2024-12-01T18:01:57.421737700Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "pipelined",
          "startTimeUTC" : "2024-12-01T18:01:57.421737700Z",
          "durationInMilliSecs" : 9.8862
        }, {
          "eventName" : "transitTime",
          "startTimeUTC" : "2024-12-01T18:01:57.431623900Z",
          "durationInMilliSecs" : 151.5614
        }, {
          "eventName" : "decodeTime",
          "startTimeUTC" : "2024-12-01T18:01:57.583185300Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "received",
          "startTimeUTC" : "2024-12-01T18:01:57.583185300Z",
          "durationInMilliSecs" : 0.0
        }, {
          "eventName" : "completed",
          "startTimeUTC" : "2024-12-01T18:01:57.583185300Z",
          "durationInMilliSecs" : 0.0
        } ],
        "rntbdRequestLengthInBytes" : 363,
        "rntbdResponseLengthInBytes" : 356,
        "requestPayloadLengthInBytes" : 0,
        "responsePayloadLengthInBytes" : 0,
        "channelStatistics" : {
          "channelId" : "535bce0f",
          "channelTaskQueueSize" : 1,
          "pendingRequestsCount" : 0,
          "lastReadTime" : "2024-12-01T18:01:57.410290400Z",
          "waitForConnectionInit" : false
        },
        "serviceEndpointStatistics" : {
          "availableChannels" : 1,
          "acquiredChannels" : 0,
          "executorTaskQueueSize" : 0,
          "inflightRequests" : 2,
          "lastSuccessfulRequestTime" : "2024-12-01T18:01:56.558Z",
          "lastRequestTime" : "2024-12-01T18:01:56.558Z",
          "createdTime" : "2024-12-01T18:01:56.555157600Z",
          "isClosed" : false,
          "cerMetrics" : { }
        }
      },
      "requestResponseTimeUTC" : "2024-12-01T18:01:57.583185300Z",
      "requestStartTimeUTC" : "2024-12-01T18:01:57.421737700Z",
      "requestResourceType" : "DocumentCollection",
      "requestOperationType" : "Head",
      "requestSessionToken" : null,
      "e2ePolicyCfg" : null,
      "excludedRegions" : null,
      "sessionTokenEvaluationResults" : [ ],
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    } ],
    "addressResolutionStatistics" : {
      "1c8b849b-06c4-4bd1-b4f6-595cdb07776e" : {
        "startTimeUTC" : "2024-12-01T18:01:26.003796800Z",
        "endTimeUTC" : "2024-12-01T18:01:26.162705Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "4fb5d649-9d21-4f44-ac94-393f690ad06f" : {
        "startTimeUTC" : "2024-12-01T18:01:41.171459500Z",
        "endTimeUTC" : "2024-12-01T18:01:41.376221Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : true,
        "inflightRequest" : false
      },
      "ac0faa54-c837-49c3-8526-67d6901ea259" : {
        "startTimeUTC" : "2024-12-01T18:00:54.847772800Z",
        "endTimeUTC" : "2024-12-01T18:00:55.521935300Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : "io.netty.handler.timeout.ReadTimeoutException",
        "forceRefresh" : false,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "f41ddb6f-3f5b-404a-ab75-f7242fcbf3dc" : {
        "startTimeUTC" : "2024-12-01T18:01:11.622232Z",
        "endTimeUTC" : "2024-12-01T18:01:11.735035600Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "a709ed36-7b0e-4278-9b35-3a430931134c" : {
        "startTimeUTC" : "2024-12-01T18:01:17.880367900Z",
        "endTimeUTC" : "2024-12-01T18:01:17.991635900Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "dd1ffbf8-4800-4ff4-b602-acb6ed64bf08" : {
        "startTimeUTC" : "2024-12-01T18:01:55.903341200Z",
        "endTimeUTC" : "2024-12-01T18:01:56.005965Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : true,
        "inflightRequest" : false
      },
      "78bedfd5-1099-47ca-8787-bc6efe46d6ca" : {
        "startTimeUTC" : "2024-12-01T18:00:55.521935300Z",
        "endTimeUTC" : "2024-12-01T18:01:00.780591400Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : "io.netty.handler.timeout.ReadTimeoutException",
        "forceRefresh" : false,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "41d5032f-eff2-4916-9209-c4b787bb322d" : {
        "startTimeUTC" : "2024-12-01T18:01:01.796334700Z",
        "endTimeUTC" : "2024-12-01T18:01:10.330952800Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : false,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "a9e95e42-0703-4aae-8144-4fb3e1217ae5" : {
        "startTimeUTC" : "2024-12-01T18:00:53.836835400Z",
        "endTimeUTC" : "2024-12-01T18:00:54.270315900Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-centralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : false,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "0d486161-5643-4735-9c7a-152b9a45bfa8" : {
        "startTimeUTC" : "2024-12-01T18:01:55.386902900Z",
        "endTimeUTC" : "2024-12-01T18:01:55.900867700Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : "io.netty.handler.timeout.ReadTimeoutException",
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : true,
        "inflightRequest" : false
      },
      "d01a366e-c2cd-4736-9f92-12d120892b36" : {
        "startTimeUTC" : "2024-12-01T18:01:10.330952800Z",
        "endTimeUTC" : "2024-12-01T18:01:10.607683900Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "3f8d5ca1-2b14-4dc4-8f2f-d30848f87868" : {
        "startTimeUTC" : "2024-12-01T18:01:13.755569800Z",
        "endTimeUTC" : "2024-12-01T18:01:13.871497600Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : true,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      },
      "1d029734-67f2-439b-a229-4cd0f2f51bef" : {
        "startTimeUTC" : "2024-12-01T18:01:56.018464700Z",
        "endTimeUTC" : "2024-12-01T18:01:56.555157600Z",
        "targetEndpoint" : "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
        "exceptionMessage" : null,
        "forceRefresh" : false,
        "forceCollectionRoutingMapRefresh" : false,
        "inflightRequest" : false
      }
    },
    "regionsContacted" : [ "central us", "north central us", "west us 2" ],
    "retryContext" : {
      "statusAndSubStatusCodes" : [ [ 403, 3 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 410, 0 ], [ 503, 0 ] ],
      "retryCount" : 10,
      "retryLatency" : 62754
    },
    "metadataDiagnosticsContext" : {
      "metadataDiagnosticList" : [ {
        "metaDataName" : "CONTAINER_LOOK_UP",
        "startTimeUTC" : "2024-12-01T18:00:35.539077800Z",
        "endTimeUTC" : "2024-12-01T18:00:35.648073300Z",
        "durationinMS" : 108
      }, {
        "metaDataName" : "PARTITION_KEY_RANGE_LOOK_UP",
        "startTimeUTC" : "2024-12-01T18:00:35.657063700Z",
        "endTimeUTC" : "2024-12-01T18:00:53.817062200Z",
        "durationinMS" : 18159
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:00:53.836835400Z",
        "endTimeUTC" : "2024-12-01T18:00:54.270315900Z",
        "durationinMS" : 433
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:01.796334700Z",
        "endTimeUTC" : "2024-12-01T18:01:10.330952800Z",
        "durationinMS" : 8534
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:10.330952800Z",
        "endTimeUTC" : "2024-12-01T18:01:10.607683900Z",
        "durationinMS" : 276
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:11.622232Z",
        "endTimeUTC" : "2024-12-01T18:01:11.735035600Z",
        "durationinMS" : 112
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:13.755569800Z",
        "endTimeUTC" : "2024-12-01T18:01:13.871497600Z",
        "durationinMS" : 115
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:17.880367900Z",
        "endTimeUTC" : "2024-12-01T18:01:17.991635900Z",
        "durationinMS" : 111
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:26.003796800Z",
        "endTimeUTC" : "2024-12-01T18:01:26.162705Z",
        "durationinMS" : 158
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:41.171459500Z",
        "endTimeUTC" : "2024-12-01T18:01:41.376221Z",
        "durationinMS" : 204
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:55.903341200Z",
        "endTimeUTC" : "2024-12-01T18:01:56.005965Z",
        "durationinMS" : 102
      }, {
        "metaDataName" : "SERVER_ADDRESS_LOOKUP",
        "startTimeUTC" : "2024-12-01T18:01:56.018464700Z",
        "endTimeUTC" : "2024-12-01T18:01:56.555157600Z",
        "durationinMS" : 536
      } ]
    },
    "serializationDiagnosticsContext" : {
      "serializationDiagnosticsList" : [ {
        "serializationType" : "PARTITION_KEY_FETCH_SERIALIZATION",
        "startTimeUTC" : "2024-12-01T18:00:35.648073300Z",
        "endTimeUTC" : "2024-12-01T18:00:35.651091400Z",
        "durationInMilliSecs" : 3.0181
      }, {
        "serializationType" : "PARTITION_KEY_FETCH_SERIALIZATION",
        "startTimeUTC" : "2024-12-01T18:00:54.847772800Z",
        "endTimeUTC" : "2024-12-01T18:00:54.847772800Z",
        "durationInMilliSecs" : 0.0
      }, {
        "serializationType" : "PARTITION_KEY_FETCH_SERIALIZATION",
        "startTimeUTC" : "2024-12-01T18:01:56.005965Z",
        "endTimeUTC" : "2024-12-01T18:01:56.005965Z",
        "durationInMilliSecs" : 0.0
      } ]
    },
    "gatewayStatisticsList" : [ {
      "sessionToken" : null,
      "operationType" : "Upsert",
      "resourceType" : "Document",
      "statusCode" : 408,
      "subStatusCode" : 10002,
      "requestCharge" : 0.0,
      "requestTimeline" : null,
      "partitionKeyRangeId" : null,
      "responsePayloadSizeInBytes" : 0,
      "exceptionResponseHeaders" : "{x-ms-substatus=10002}",
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    }, {
      "sessionToken" : null,
      "operationType" : "Upsert",
      "resourceType" : "Document",
      "statusCode" : 408,
      "subStatusCode" : 10002,
      "requestCharge" : 0.0,
      "requestTimeline" : null,
      "partitionKeyRangeId" : null,
      "responsePayloadSizeInBytes" : 0,
      "exceptionResponseHeaders" : "{x-ms-substatus=10002}",
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    }, {
      "sessionToken" : null,
      "operationType" : "Upsert",
      "resourceType" : "Document",
      "statusCode" : 408,
      "subStatusCode" : 10002,
      "requestCharge" : 0.0,
      "requestTimeline" : null,
      "partitionKeyRangeId" : null,
      "responsePayloadSizeInBytes" : 0,
      "exceptionResponseHeaders" : "{x-ms-substatus=10002}",
      "locationToLocationSpecificHealthContext" : {
        "v" : null
      }
    } ],
    "samplingRateSnapshot" : 1.0,
    "bloomFilterInsertionCountSnapshot" : 0,
    "systemInformation" : {
      "usedMemory" : "41417 KB",
      "availableMemory" : "12467767 KB",
      "systemCpuLoad" : "(2024-12-01T18:01:47.287520100Z 6.3%), (2024-12-01T18:01:52.300384200Z 7.0%), (2024-12-01T18:01:57.286141100Z 9.5%), (2024-12-01T18:02:14.872390500Z 17.3%), (2024-12-01T18:02:14.872390500Z 17.3%), (2024-12-01T18:02:14.872390500Z 17.3%)",
      "availableProcessors" : 8
    },
    "clientCfgs" : {
      "id" : 1,
      "machineId" : "uuid:3c866e29-2802-4d93-982d-260c28e77b18",
      "connectionMode" : "DIRECT",
      "numberOfClients" : 1,
      "excrgns" : "[]",
      "clientEndpoints" : {
        "https://abhm-ppaf-testing.documents-test.windows-int.net:443/" : 1
      },
      "connCfg" : {
        "rntbd" : "(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)",
        "gw" : "(cps:1000, nrto:PT1M, icto:PT1M, p:false)",
        "other" : "(ed: true, cs: false, rv: true)"
      },
      "consistencyCfg" : "(consistency: null, mm: true, prgns: [northcentralus,westus2,centralus])",
      "proactiveInitCfg" : "",
      "e2ePolicyCfg" : "",
      "sessionRetryCfg" : "",
      "partitionLevelCircuitBreakerCfg" : "(cb: false, type: CONSECUTIVE_EXCEPTION_COUNT_BASED, rexcntt: 10, wexcntt: 5)"
    }
  }

Inject quorum loss into a server partition in the primary region and secondary region where a warm client is running the workload.

In this scenario, quorum loss is injected into a server partition belonging to the same partition set in NorthCentralUs and WestUs2 while the workload is running. A success response is obtained from CentralUs.

{
    "userAgent": "azsdk-java-cosmos/4.65.0-beta.1 Windows11/10.0 JRE/17.0.13",
    "activityId": "d1ec6f40-86a4-4786-a1c8-b0517bc22690",
    "requestLatencyInMs": 78777,
    "requestStartTimeUTC": "2024-12-01T18:35:29.675533300Z",
    "requestEndTimeUTC": "2024-12-01T18:36:48.453339200Z",
    "responseStatisticsList": [
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-northcentralus1-be1.documents-test.windows-int.net:14009/apps/52408f3e-7918-434b-b59e-5a16313cec2c/services/a777cc3a-919a-4580-b77e-5dee3b560003/partitions/51df8419-15a3-4135-84ca-9115e25c60da/replicas/133774529066660643p/",
                "lsn": 16456,
                "quorumAckedLSN": 16456,
                "currentReplicaSetSize": 3,
                "globalCommittedLsn": 16455,
                "partitionKeyRangeId": null,
                "isValid": true,
                "statusCode": 403,
                "subStatusCode": 3,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": -1,
                "sessionToken": ":-1#16456",
                "backendLatencyInMs": 0.132,
                "retryAfterInMs": null,
                "exceptionMessage": "[\"The requested operation cannot be performed at this region\"]",
                "exceptionResponseHeaders": "{x-ms-current-replica-set-size=4, x-ms-last-state-change-utc=Sun, 01 Dec 2024 18:36:37.687 GMT, x-ms-request-duration-ms=0.132, x-ms-session-token=:-1#16456, lsn=16456, x-ms-schemaversion=1.18, x-ms-transport-request-id=1897, x-ms-number-of-read-regions=2, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=16456, x-ms-quorum-acked-lsn=16456, x-ms-activity-id=f271207d-077e-4cd2-9817-eeb6d38cb45f, x-ms-xp-role=1, x-ms-global-Committed-lsn=16455, x-ms-cosmos-llsn=16456, x-ms-serviceversion= version=2.14.0.0, x-ms-substatus=3}",
                "replicaStatusList": {
                    "Ignoring": [
                        "14002:S:Unknown",
                        "14306:S:Unknown",
                        "14303:S:Unknown"
                    ],
                    "Attempting": [
                        "14009:P:Unknown"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:46.391816Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:46.391816Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:46.391816Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:46.391816Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:46.391816Z",
                        "durationInMilliSecs": 115.5733
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:46.507389300Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:46.507389300Z",
                        "durationInMilliSecs": 1.0044
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:46.508393700Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 418,
                "rntbdResponseLengthInBytes": 277,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 277,
                "channelStatistics": {
                    "channelId": "8c773000",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:31:17.952406400Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:31:17.954Z",
                    "lastRequestTime": "2024-12-01T18:31:17.840Z",
                    "createdTime": "2024-12-01T18:27:52.358589500Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:46.508393700Z",
            "requestStartTimeUTC": "2024-12-01T18:36:46.391816Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14004/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488721995992p/",
                "lsn": 16624,
                "quorumAckedLSN": 16623,
                "currentReplicaSetSize": 4,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 200,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 10.29,
                "itemLSN": -1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 6.494,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [
                        "14307:S:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ],
                    "Attempting": [
                        "14004:P:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                        "durationInMilliSecs": 65.0832
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:46.576927400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:46.576927400Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:46.576927400Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 418,
                "rntbdResponseLengthInBytes": 449,
                "requestPayloadLengthInBytes": 10,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "d0ba20b6",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:35:13.543790200Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:35:13.547Z",
                    "lastRequestTime": "2024-12-01T18:35:13.440Z",
                    "createdTime": "2024-12-01T18:31:18.419193300Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:46.576927400Z",
            "requestStartTimeUTC": "2024-12-01T18:36:46.511844200Z",
            "requestResourceType": "Document",
            "requestOperationType": "Upsert",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        }
    ],
    "supplementalResponseStatisticsList": [
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14307/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621134s/",
                "lsn": 16624,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.25,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:46.937338100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:46.937338100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:46.937338100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:46.937338100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:46.937338100Z",
                        "durationInMilliSecs": 119.7574
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.057095500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.057095500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.057095500Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "a01aec9d",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:46.840855900Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:46.841Z",
                    "lastRequestTime": "2024-12-01T18:36:46.791Z",
                    "createdTime": "2024-12-01T18:34:31.303234800Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.057095500Z",
            "requestStartTimeUTC": "2024-12-01T18:36:46.937338100Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14004/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488721995992p/",
                "lsn": 16624,
                "quorumAckedLSN": 16624,
                "currentReplicaSetSize": 4,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.324,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.105307900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.105307900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.105307900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.105307900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.105307900Z",
                        "durationInMilliSecs": 112.9941
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.218302Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.218302Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.218302Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 356,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "d0ba20b6",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:46.904576200Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:46.905Z",
                    "lastRequestTime": "2024-12-01T18:36:46.856Z",
                    "createdTime": "2024-12-01T18:31:18.419193300Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.218302Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.105307900Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14300/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621132s/",
                "lsn": 16622,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16622",
                "backendLatencyInMs": 0.29,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.264564100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.264564100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.264564100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.264564100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.264564100Z",
                        "durationInMilliSecs": 111.4072
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.375971300Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.375971300Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.375971300Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "de519081",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:46.643613200Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:46.642Z",
                    "lastRequestTime": "2024-12-01T18:36:46.577Z",
                    "createdTime": "2024-12-01T18:34:30.243557400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.375971300Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.264564100Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14300/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621132s/",
                "lsn": 16622,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16622",
                "backendLatencyInMs": 0.246,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.424420900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.424420900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.424420900Z",
                        "durationInMilliSecs": 0.9993
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.425420200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.425420200Z",
                        "durationInMilliSecs": 120.0626
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.545482800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.545482800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.545482800Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "de519081",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:47.375971300Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:47.377Z",
                    "lastRequestTime": "2024-12-01T18:36:47.264Z",
                    "createdTime": "2024-12-01T18:34:30.243557400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.545482800Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.424420900Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14308/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621133s/",
                "lsn": 16624,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.233,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.575853100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.575853100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.575853100Z",
                        "durationInMilliSecs": 1.1238
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.576976900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.576976900Z",
                        "durationInMilliSecs": 112.5789
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.689555800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.689555800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.689555800Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "a7fd8601",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:46.776831500Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:46.776Z",
                    "lastRequestTime": "2024-12-01T18:36:46.723Z",
                    "createdTime": "2024-12-01T18:34:29.619546700Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.689555800Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.575853100Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14300/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621132s/",
                "lsn": 16622,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16622",
                "backendLatencyInMs": 0.255,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.723838600Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.723838600Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.723838600Z",
                        "durationInMilliSecs": 0.9585
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.724797100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.724797100Z",
                        "durationInMilliSecs": 115.653
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:47.840450100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:47.840450100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:47.840450100Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "de519081",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:47.545482800Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:47.544Z",
                    "lastRequestTime": "2024-12-01T18:36:47.423Z",
                    "createdTime": "2024-12-01T18:34:30.243557400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:47.840450100Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.723838600Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14308/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621133s/",
                "lsn": 16624,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.253,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:47.888566800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:47.888566800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:47.888566800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:47.888566800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:47.888566800Z",
                        "durationInMilliSecs": 119.342
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:48.007908800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:48.007908800Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:48.007908800Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "a7fd8601",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:47.689555800Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:47.689Z",
                    "lastRequestTime": "2024-12-01T18:36:47.575Z",
                    "createdTime": "2024-12-01T18:34:29.619546700Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:48.007908800Z",
            "requestStartTimeUTC": "2024-12-01T18:36:47.888566800Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14307/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621134s/",
                "lsn": 16624,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.253,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:48.042342100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:48.042342100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:48.042342100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:48.042342100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:48.042342100Z",
                        "durationInMilliSecs": 108.0828
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:48.150424900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:48.150424900Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:48.150424900Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "a01aec9d",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:47.057095500Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:47.059Z",
                    "lastRequestTime": "2024-12-01T18:36:46.937Z",
                    "createdTime": "2024-12-01T18:34:31.303234800Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:48.150424900Z",
            "requestStartTimeUTC": "2024-12-01T18:36:48.042342100Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14300/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621132s/",
                "lsn": 16622,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16621,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16622",
                "backendLatencyInMs": 0.238,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:48.188819600Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:48.188819600Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:48.188819600Z",
                        "durationInMilliSecs": 1.5235
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:48.190343100Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:48.190343100Z",
                        "durationInMilliSecs": 111.5159
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:48.301859Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:48.301859Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:48.301859Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "de519081",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:47.840450100Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:47.845Z",
                    "lastRequestTime": "2024-12-01T18:36:47.723Z",
                    "createdTime": "2024-12-01T18:34:30.243557400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:48.301859Z",
            "requestStartTimeUTC": "2024-12-01T18:36:48.188819600Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        },
        {
            "storeResult": {
                "storePhysicalAddress": "rntbd://cdb-ms-test61-centralus1-be1.documents-test.windows-int.net:14300/apps/aaf1f673-9801-41cf-9a9e-1b4fce153138/services/13acee5e-383f-46f9-960e-46102bc8eaec/partitions/8440d781-6545-4ded-ad3e-57fab1699dcd/replicas/133770488772621132s/",
                "lsn": 16624,
                "quorumAckedLSN": -1,
                "currentReplicaSetSize": -1,
                "globalCommittedLsn": 16624,
                "partitionKeyRangeId": "0",
                "isValid": true,
                "statusCode": 204,
                "subStatusCode": 0,
                "isGone": false,
                "isNotFound": false,
                "isInvalidPartition": false,
                "isThroughputControlRequestRateTooLarge": false,
                "requestCharge": 0.0,
                "itemLSN": 1,
                "sessionToken": "0:-1#16624",
                "backendLatencyInMs": 0.194,
                "retryAfterInMs": null,
                "replicaStatusList": {
                    "Ignoring": [],
                    "Attempting": [
                        "14307:S:Connected",
                        "14004:P:Connected",
                        "14300:S:Connected",
                        "14308:S:Connected"
                    ]
                },
                "transportRequestTimeline": [
                    {
                        "eventName": "created",
                        "startTimeUTC": "2024-12-01T18:36:48.341147500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "queued",
                        "startTimeUTC": "2024-12-01T18:36:48.341147500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "channelAcquisitionStarted",
                        "startTimeUTC": "2024-12-01T18:36:48.341147500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "pipelined",
                        "startTimeUTC": "2024-12-01T18:36:48.341147500Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "transitTime",
                        "startTimeUTC": "2024-12-01T18:36:48.341147500Z",
                        "durationInMilliSecs": 112.1917
                    },
                    {
                        "eventName": "decodeTime",
                        "startTimeUTC": "2024-12-01T18:36:48.453339200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "received",
                        "startTimeUTC": "2024-12-01T18:36:48.453339200Z",
                        "durationInMilliSecs": 0.0
                    },
                    {
                        "eventName": "completed",
                        "startTimeUTC": "2024-12-01T18:36:48.453339200Z",
                        "durationInMilliSecs": 0.0
                    }
                ],
                "rntbdRequestLengthInBytes": 363,
                "rntbdResponseLengthInBytes": 320,
                "requestPayloadLengthInBytes": 0,
                "responsePayloadLengthInBytes": 0,
                "channelStatistics": {
                    "channelId": "de519081",
                    "channelTaskQueueSize": 1,
                    "pendingRequestsCount": 0,
                    "lastReadTime": "2024-12-01T18:36:48.301859Z",
                    "waitForConnectionInit": false
                },
                "serviceEndpointStatistics": {
                    "availableChannels": 1,
                    "acquiredChannels": 0,
                    "executorTaskQueueSize": 0,
                    "inflightRequests": 1,
                    "lastSuccessfulRequestTime": "2024-12-01T18:36:48.302Z",
                    "lastRequestTime": "2024-12-01T18:36:48.188Z",
                    "createdTime": "2024-12-01T18:34:30.243557400Z",
                    "isClosed": false,
                    "cerMetrics": {}
                }
            },
            "requestResponseTimeUTC": "2024-12-01T18:36:48.453339200Z",
            "requestStartTimeUTC": "2024-12-01T18:36:48.341147500Z",
            "requestResourceType": "DocumentCollection",
            "requestOperationType": "Head",
            "requestSessionToken": null,
            "e2ePolicyCfg": null,
            "excludedRegions": null,
            "sessionTokenEvaluationResults": [],
            "locationToLocationSpecificHealthContext": {
                "v": null
            }
        }
    ],
    "addressResolutionStatistics": {
        "280c2632-93a9-4fb9-a98c-defbe49fd9e0": {
            "startTimeUTC": "2024-12-01T18:35:45.775970300Z",
            "endTimeUTC": "2024-12-01T18:35:45.944273100Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "18ae5d6e-bbbc-4608-a02d-b9c6fd4a5a78": {
            "startTimeUTC": "2024-12-01T18:36:00.959491500Z",
            "endTimeUTC": "2024-12-01T18:36:01.111902300Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "a5092306-c373-46f7-b420-f600e94c09c9": {
            "startTimeUTC": "2024-12-01T18:36:34.024504600Z",
            "endTimeUTC": "2024-12-01T18:36:34.136988800Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "50fd9110-d66a-4f74-927f-d38267e70a15": {
            "startTimeUTC": "2024-12-01T18:36:30.322659200Z",
            "endTimeUTC": "2024-12-01T18:36:30.492738600Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "e6df90a5-5898-4b92-8a48-cd806088c41f": {
            "startTimeUTC": "2024-12-01T18:36:46.273294300Z",
            "endTimeUTC": "2024-12-01T18:36:46.391816Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "a59044b1-521c-4795-a65f-e8d61c73426e": {
            "startTimeUTC": "2024-12-01T18:36:30.492738600Z",
            "endTimeUTC": "2024-12-01T18:36:30.875202Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "8784db94-0b40-4ae9-a5b0-7b999115f7ca": {
            "startTimeUTC": "2024-12-01T18:35:37.604908100Z",
            "endTimeUTC": "2024-12-01T18:35:37.767642300Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "45be1ddd-8ada-4df7-8353-2ba1be0cc096": {
            "startTimeUTC": "2024-12-01T18:36:16.141085600Z",
            "endTimeUTC": "2024-12-01T18:36:16.311616300Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "4f71d914-b897-440a-9552-7d012f91e2e9": {
            "startTimeUTC": "2024-12-01T18:36:31.891448300Z",
            "endTimeUTC": "2024-12-01T18:36:32.005519700Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "2322f050-2d68-4462-a39d-64ef8f0b188a": {
            "startTimeUTC": "2024-12-01T18:35:31.276789800Z",
            "endTimeUTC": "2024-12-01T18:35:31.436274800Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "ba8d0582-47af-465e-a0e6-8d68f4cca58d": {
            "startTimeUTC": "2024-12-01T18:35:29.675533300Z",
            "endTimeUTC": "2024-12-01T18:35:30.259456500Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": false,
            "inflightRequest": false
        },
        "f2644726-2be8-4aff-9416-cc5dc0886e3a": {
            "startTimeUTC": "2024-12-01T18:35:33.452454700Z",
            "endTimeUTC": "2024-12-01T18:35:33.595638Z",
            "targetEndpoint": "https://abhm-ppaf-testing-westus2.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        },
        "6b0befca-5b5d-4d7a-82a2-c8b81d86f93b": {
            "startTimeUTC": "2024-12-01T18:36:38.143783900Z",
            "endTimeUTC": "2024-12-01T18:36:38.258384700Z",
            "targetEndpoint": "https://abhm-ppaf-testing-northcentralus.documents-test.windows-int.net:443/addresses/?$resolveFor=dbs%2FawCdCQ%3D%3D%2Fcolls%2FawCdCbPQFEs%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0",
            "exceptionMessage": null,
            "forceRefresh": true,
            "forceCollectionRoutingMapRefresh": true,
            "inflightRequest": false
        }
    },
    "regionsContacted": [
        "central us",
        "north central us"
    ],
    "retryContext": {
        "statusAndSubStatusCodes": [
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                503,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                410,
                0
            ],
            [
                403,
                3
            ]
        ],
        "retryCount": 16,
        "retryLatency": 78777
    },
    "metadataDiagnosticsContext": {
        "metadataDiagnosticList": [
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:35:29.675533300Z",
                "endTimeUTC": "2024-12-01T18:35:30.259456500Z",
                "durationinMS": 583
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:35:31.276789800Z",
                "endTimeUTC": "2024-12-01T18:35:31.436274800Z",
                "durationinMS": 159
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:35:33.452454700Z",
                "endTimeUTC": "2024-12-01T18:35:33.595638Z",
                "durationinMS": 143
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:35:37.604908100Z",
                "endTimeUTC": "2024-12-01T18:35:37.767642300Z",
                "durationinMS": 162
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:35:45.775970300Z",
                "endTimeUTC": "2024-12-01T18:35:45.944273100Z",
                "durationinMS": 168
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:00.959491500Z",
                "endTimeUTC": "2024-12-01T18:36:01.111902300Z",
                "durationinMS": 152
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:16.141085600Z",
                "endTimeUTC": "2024-12-01T18:36:16.311616300Z",
                "durationinMS": 170
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:30.322659200Z",
                "endTimeUTC": "2024-12-01T18:36:30.492738600Z",
                "durationinMS": 170
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:30.492738600Z",
                "endTimeUTC": "2024-12-01T18:36:30.875202Z",
                "durationinMS": 382
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:31.891448300Z",
                "endTimeUTC": "2024-12-01T18:36:32.005519700Z",
                "durationinMS": 114
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:34.024504600Z",
                "endTimeUTC": "2024-12-01T18:36:34.136988800Z",
                "durationinMS": 112
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:38.143783900Z",
                "endTimeUTC": "2024-12-01T18:36:38.258384700Z",
                "durationinMS": 114
            },
            {
                "metaDataName": "SERVER_ADDRESS_LOOKUP",
                "startTimeUTC": "2024-12-01T18:36:46.273294300Z",
                "endTimeUTC": "2024-12-01T18:36:46.391816Z",
                "durationinMS": 118
            }
        ]
    },
    "serializationDiagnosticsContext": {
        "serializationDiagnosticsList": [
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T18:35:29.675533300Z",
                "endTimeUTC": "2024-12-01T18:35:29.675533300Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T18:36:30.492738600Z",
                "endTimeUTC": "2024-12-01T18:36:30.492738600Z",
                "durationInMilliSecs": 0.0
            },
            {
                "serializationType": "PARTITION_KEY_FETCH_SERIALIZATION",
                "startTimeUTC": "2024-12-01T18:36:46.511844200Z",
                "endTimeUTC": "2024-12-01T18:36:46.511844200Z",
                "durationInMilliSecs": 0.0
            }
        ]
    },
    "gatewayStatisticsList": [],
    "samplingRateSnapshot": 1.0,
    "bloomFilterInsertionCountSnapshot": 0,
    "systemInformation": {
        "usedMemory": "42709 KB",
        "availableMemory": "12466475 KB",
        "systemCpuLoad": "empty",
        "availableProcessors": 8
    },
    "clientCfgs": {
        "id": 1,
        "machineId": "uuid:c908f46a-876a-4f94-8cb0-a1c288983e55",
        "connectionMode": "DIRECT",
        "numberOfClients": 0,
        "excrgns": "[]",
        "clientEndpoints": {
            "https://abhm-ppaf-testing.documents-test.windows-int.net:443/": 1
        },
        "connCfg": {
            "rntbd": "(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)",
            "gw": "(cps:1000, nrto:PT1M, icto:PT1M, p:false)",
            "other": "(ed: true, cs: false, rv: true)"
        },
        "consistencyCfg": "(consistency: null, mm: true, prgns: [northcentralus,westus2,centralus])",
        "proactiveInitCfg": "",
        "e2ePolicyCfg": "",
        "sessionRetryCfg": "",
        "partitionLevelCircuitBreakerCfg": "(cb: false, type: CONSECUTIVE_EXCEPTION_COUNT_BASED, rexcntt: 10, wexcntt: 5)"
    }
}

Inject quorum loss into a server partition in the primary region and secondary region after which a cold client starts running the workload.

@github-actions github-actions bot added the Cosmos label Jul 6, 2024
@jeet1995 jeet1995 changed the title Per partition automatic failover [NO REVIEW]: Per-partition automatic failover Jul 6, 2024
@jeet1995
Copy link
Member Author

jeet1995 commented Jul 6, 2024

/azp run java - cosmos - tests

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@azure-sdk
Copy link
Collaborator

azure-sdk commented Jul 31, 2024

API change check

APIView has identified API level changes in this PR and created following API reviews.

com.azure:azure-cosmos

jeet1995 added 21 commits August 1, 2024 15:49
…rPartitionAutomaticFailover

# Conflicts:
#	sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/GlobalPartitionEndpointManagerForPerPartitionCircuitBreakerTests.java
#	sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/faultinjection/SessionRetryOptionsTests.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/LocationSpecificHealthContextTransitionHandler.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/query/ChangeFeedFetcher.java
…rPartitionAutomaticFailover

# Conflicts:
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java
…rPartitionAutomaticFailover

# Conflicts:
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java
…rPartitionAutomaticFailover

# Conflicts:
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/CosmosClientBuilder.java
#	sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java
@jeet1995 jeet1995 changed the title [NO REVIEW]: Per-partition automatic failover Per-partition automatic failover Dec 3, 2024
@@ -193,7 +193,7 @@ public Mono<ShouldRetryResult> shouldRetry(Exception exception) {
Duration timeout;
boolean forceRefreshAddressCache;
if (isNonRetryableException(exception)) {
logger.debug("Operation will NOT be retried. Current attempt {}, Exception: ", this.attemptCount,
logger.warn("Operation will NOT be retried. Current attempt {}, Exception: ", this.attemptCount,
Copy link
Member Author

@jeet1995 jeet1995 Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment for self: revert warn to debug prior to merge.

@@ -209,7 +209,9 @@ public Mono<ShouldRetryResult> shouldRetry(Exception exception) {
this.attemptCount,
exception);

return Mono.just(ShouldRetryResult.noRetry(
exceptionToThrow = logAndWrapExceptionWithLastRetryWithException(exception);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea is to wrap non-retriable (within the region) 410s as 503s.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants