Added API to inject custom serializer for point read and query

[FabianMeiswinkel]

Description

The goal of this PR is to add a new API that allows customers to extend the built-in serialization. Today, the Cosmso DB client has multiple methods where the customer can specify the PoJo type of documents/items to be returned. For example...

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
package com.azure.cosmos;

public class CosmosAsyncContainer {
    [...]

/**
     * Reads an item by itemId.
     * <br/>
     * This operation is used to retrieve a single item from a container based on its unique identifier (ID) and partition key.
     * The readItem operation provides direct access to a specific item using its unique identifier, which consists of the item's ID and the partition key value. This operation is efficient for retrieving a known item by its ID and partition key without the need for complex querying.
     * <p>
     * After subscription the operation will be performed.
     * The {@link Mono} upon successful completion will contain an item response with the read item.
     * <!-- src_embed com.azure.cosmos.CosmosAsyncContainer.readItem -->
     * <pre>
     * &#47;&#47; Read an item
     * cosmosAsyncContainer.readItem&#40;passenger.getId&#40;&#41;, new PartitionKey&#40;passenger.getId&#40;&#41;&#41;, Passenger.class&#41;
     *     .flatMap&#40;response -&gt; Mono.just&#40;response.getItem&#40;&#41;&#41;&#41;
     *     .subscribe&#40;passengerItem -&gt; System.out.println&#40;passengerItem&#41;, throwable -&gt; &#123;
     *         CosmosException cosmosException = &#40;CosmosException&#41; throwable;
     *         cosmosException.printStackTrace&#40;&#41;;
     *     &#125;&#41;;
     * &#47;&#47; ...
     * </pre>
     * <!-- end com.azure.cosmos.CosmosAsyncContainer.readItem -->
     *
     * @param <T> the type parameter.
     * @param itemId the item id.
     * @param partitionKey the partition key.
     * @param itemType the item type.
     * @return an {@link Mono} containing the Cosmos item response with the read item or an error.
     */
    public <T> Mono<CosmosItemResponse<T>> readItem(String itemId, PartitionKey partitionKey, Class<T> itemType) {
        return readItem(itemId, partitionKey, ModelBridgeInternal.createCosmosItemRequestOptions(partitionKey), itemType);
    }

    /**
     * Reads an item by itemId using a configured {@link CosmosItemRequestOptions}.
     * <br/>
     * This operation is used to retrieve a single item from a container based on its unique identifier (ID) and partition key.
     * The readItem operation provides direct access to a specific item using its unique identifier, which consists of the item's ID and the partition key value. This operation is efficient for retrieving a known item by its ID and partition key without the need for complex querying.
     * <p>
     * After subscription the operation will be performed.
     * The {@link Mono} upon successful completion will contain a Cosmos item response with the read item.
     *
     * @param <T> the type parameter.
     * @param itemId the item id.
     * @param partitionKey the partition key.
     * @param options the request (Optional) {@link CosmosItemRequestOptions}.
     * @param itemType the item type.
     * @return an {@link Mono} containing the Cosmos item response with the read item or an error.
     */
    public <T> Mono<CosmosItemResponse<T>> readItem(
        String itemId, PartitionKey partitionKey,
        CosmosItemRequestOptions options, Class<T> itemType) {
        [...]
    }

    [...]
}

Today, the serialization into the Class Pojo type happens with Jackson's ObjectMapper and pre-set serialization settings. The only way around this right now, is to use com.fasterxml.jackson.databind.node.ObjectNode as the Pojo type - this would still do the json parsing with the pre-defined setting, but customers can use their own ObjectMapper to customize actual serialization/deserialization. But even this approach has some drawbacks - for example it wouldn't be easily possible to do custom payload transformations - for example to put the Json-payload into an envelope, extract id, partition key and maybe some queriable properties and compress the remaining part etc. So, the goal of this PR is, to add an API allowing those payload transformations and also allow overriding serialization settings on a per-operation or client-wide.

The limitation of not allowing to modify json parsing -setting is maintained - this is needed to make sure that the json payload store in the database is readable in the service. Cosmos DB is - and stays - a json database after all.

CosmosItemSerializer trait

As a representation of a Cosmos DB item Map<String, Object> is used - why a generic map, not Jackson's ObjectNode or similar? The goal is to allow serialization to even use different json stacks (gson instead of Jackson etc.) - so, an abstraction of a json tree is needed - ideally one where efficient conversions exist. Map<String, Object> is a reasonably good compromise here.

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

package com.azure.cosmos;

/**
 * The {@link CosmosItemSerializer} allows customizing the serialization of Cosmos Items - either to transform payload (for
 * example wrap/unwrap in custom envelopes) or use custom serialization settings or json serializer stacks.
 */
public abstract class CosmosItemSerializer {
    private final static ObjectMapper objectMapper = Utils.getSimpleObjectMapper();

    /**
     * Gets the default Cosmos item serializer. This serializer is used by default when no custom serializer is
     * specified on request options or the {@link CosmosClientBuilder}
     */
    public final static CosmosItemSerializer DEFAULT_SERIALIZER = new DefaultCosmosItemSerializer();

    /**
     * Used to instantiate subclasses
     */
    protected CosmosItemSerializer() {
    }

    /**
     * Used to serialize a POJO into a json tree
     * @param item the POJO to be serialized
     * @return the json tree that will be used as payload in Cosmos DB items
     * @param <T> The type of the POJO
     */
    public abstract <T> Map<String, Object> serialize(T item);

    /**
     * Used to deserialize the json tree stored in the Cosmos DB item as a POJO
     * @param jsonNodeMap the json tree from the Cosmos DB item
     * @param classType The type of the POJO
     * @return The deserialized POJO
     * @param <T> The type of the POJO
     */
    public abstract  <T> T deserialize(Map<String, Object> jsonNodeMap, Class<T> classType);
}

Per-client serialization defaults

In the CosmsoClientBuilder a new API is added, which allows specifying the default serialization options. If not specified, the default serialization settings are the same as the -pre-set settings today.

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
package com.azure.cosmos;

@ServiceClientBuilder(serviceClients = {CosmosClient.class, CosmosAsyncClient.class})
public class CosmosClientBuilder implements
    TokenCredentialTrait<CosmosClientBuilder>,
    AzureKeyCredentialTrait<CosmosClientBuilder>,
    EndpointTrait<CosmosClientBuilder> {
    /**
     * Sets a custom serializer that should be used for conversion between POJOs and Json payload stored in the
     * Cosmos DB service. The custom serializer can also be specified in request options. If defined here and
     * in request options the serializer defined in request options will be used.
     * @param serializer the custom serialzier to be used for payload transformations
     * @return current CosmosClientBuilder
     */
    public CosmosClientBuilder setCustomSerializer(CosmosItemSerializer serializer) {
        this.defaultCustomSerializer = serializer;

        return this;
    }

    [...]
}

Per-operation serialization overrides

Like many other configuration settings, the serialization behavior can be overridden in request options. When no serialization settings are specified in request options, the default of the client is used. If specified, the request options always override whatever is defined at the client level.

The request-option level override of serialization settings is possible in the following classes:

• com.azure.cosmos.models.CosmosItemRequestOptions (all point operations - including patch)
• com.azure.cosmos.models.CosmosQueryRequestOptions (for queries and read all)
• com.azure.cosmos.models.CosmosChangeFeedRequestOptions (for change feed pull model)
• com.azure.cosmos.models.CosmosReadManyRequestOptions (for read many API)
• com.azure.cosmos.models.CosmosBatchRequestOptions (for transactional batch)
• com.azure.cosmos.models.CosmosBulkExecutionOptions (for bulk operations)

All of the above request options would expose a getCustomSerializer and sertCustomSerializer method like below...

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
package com.azure.cosmos.models;

public class CosmosItemRequestOptions {

    /**
     * Gets the custom item serializer defined for this instance of request options
     * @return the custom item serializer
     */
    public CosmosItemSerializer getCustomSerializer() {
        return this.customSerializer;
    }

    /**
     * Allows specifying a custom item serializer to be used for this operation. If the serializer
     * on the request options is null, the serializer on CosmosClientBuilder is used. If both serializers
     * are null (the default), an internal Jackson ObjectMapper is ued for serialization/deserialization.
     * @param itemSerializerOverride the custom item serializer for this operation
     * @return  the CosmosItemRequestOptions.
     */
    public CosmosItemRequestOptions setCustomSerializer(CosmosItemSerializer itemSerializerOverride) {
        this.customSerializer = itemSerializerOverride;

        return this;
    }

    [...]
}

Sequencing in azure-cosmos-encryption

The custom serialization can also be used for azure-cosmos-encryption. But one question worth calling out here is what the order of processing would be:

• OriginalPayload --> Encryption --> Custom serialization or OriginalPayload --> custom serialization --> encryption

To maintain the configuration of the encryption policy based on the structure in the database the latter OriginalPayload --> custom serialization --> encryption is used. If the custom serialization for example wraps the payload in an envelope the encryption policy would be based on the structure in the database (with the envelope), so the custom serialization needs to be applied first - before any encryption.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[FabianMeiswinkel]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[FabianMeiswinkel]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

com.azure:azure-cosmos
com.azure:azure-cosmos-encryption

[FabianMeiswinkel]

/azp run java - cosmos - tests

[FabianMeiswinkel]

/azp run java - cosmos - spark

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

LGTM, great changes, thanks

[FabianMeiswinkel]

/azp run java - cosmos - tests

[FabianMeiswinkel]

/azp run java - cosmos - spark

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[sonnyhcl]

We found this is a breaking change when upgrade azure-cosmos from 4.58 to 4.59 since it requires an extra parameter, though it's always CosmosItemSerializer.DEFAULT_SERIALIZER.
Is it a missing in release note to call out this API breaking change?

[FabianMeiswinkel]

It is not a breaking change - because this is an internal API. The implementation package is not exported and is not part of public surface are. Breaking changes in the implementation package can and will happen rather frequently - even in hotfix versions - and it is simply a bug to take a direct dependency on it - modules are not enforced in Java 8 technically - but more recent Java versions would even prevent your app to use these APIs.