Add UCX history schema and table for storing UCX's artifact

[JCZuurmond]

Changes

Add UCX history schema and table for storing UCX's artifact. The artifacts are, amongst other things, going to be used for migration progress tracking.

Linked issues

Resolves #2572

Functionality

• modified existing command: databricks labs ucx create-ucx-catalog

Tests

• manually tested
• added unit tests
• added integration tests

[JCZuurmond]

@nfx and @asnare : Please provide input. Especially on the dataclass attributes and the naming

[JCZuurmond]

Missing feature on lsql to reuse SchemaDeployer here: databrickslabs/lsql#294

[asnare]

@nfx and @asnare : Please provide input. Especially on the dataclass attributes and the naming

This is the current version that I'm using (as an intermediate version) on #2743:

@dataclass(frozen=True, kw_only=True)
class HistoricalRecord:
    workspace_id: int
    """The identifier of the workspace where this record was generated."""

    run_id: str
    """An identifier of the workflow run that generated this record."""

    snapshot_id: str
    """An identifier that is unique to the records produced for a given snapshot."""

    run_start_time: dt.datetime
    """When this record was generated."""

    object_type: str
    """The inventory table for which this record was generated."""

    object_type_version: int
    """Versioning of inventory table, for forward compatibility."""

    object_id: list[str]
    """The type-specific identifier for this inventory record."""

    object_data: str
    """Type-specific JSON-encoded data of the inventory record."""

    failures: list[str]
    """The list of problems associated with the object that this inventory record covers."""

    owner: str
    """The identity of the account that created this inventory record."""

[asnare]

I've also added some notes on the feature issue: #2572 (comment) &
#2572 (comment)

[JCZuurmond]

Both `_id

@nfx and @asnare : Please provide input. Especially on the dataclass attributes and the naming

This is the current version that I'm using (as an intermediate version) on #2743:

@dataclass(frozen=True, kw_only=True)
class HistoricalRecord:
    workspace_id: int
    """The identifier of the workspace where this record was generated."""

    run_id: str
    """An identifier of the workflow run that generated this record."""

    snapshot_id: str
    """An identifier that is unique to the records produced for a given snapshot."""

    run_start_time: dt.datetime
    """When this record was generated."""

    object_type: str
    """The inventory table for which this record was generated."""

    object_type_version: int
    """Versioning of inventory table, for forward compatibility."""

    object_id: list[str]
    """The type-specific identifier for this inventory record."""

    object_data: str
    """Type-specific JSON-encoded data of the inventory record."""

    failures: list[str]
    """The list of problems associated with the object that this inventory record covers."""

    owner: str
    """The identity of the account that created this inventory record."""

Thanks! I have update the data class: e80556c

Note:

• The run_id and snapshot_id are int. Do you know if they should be str instead?
• On owner see: [FEATURE] Create a ucx.history table #2572 (comment)

[asnare]

• The run_id and snapshot_id are int. Do you know if they should be str instead?

run_id: Can definitely be an integer.

snapshot_id: I was populating this with a UUID, but I'm also fine with it being an int.

[nfx]

can it be just the ucx version str? if not str, then it should be list[int]

[asnare]

Elsewhere I wrote:

Typically it's easier to version serialisation formats independently of the software version they're used in. (Corner cases with software upgrades and downgrades are also handled better.)

Technically we don't need a version for the first version (ie, what we're doing now) and can introduce the version field when it's needed. However schema changes are painful, so having the field from the start is worth the cost of having it from the start even though it's not needed yet.

After a version change for a type, during deserialisation we only need to have logic for "this is how we decode version 0, this is how we decode version 1, etc." and it's all tightly contained/testable/etc. With the exception of some hash-based versioning schemes, this is typically how all serde handle this problem.

If we use the UCX version then this suddenly becomes based on semantic range-checks. (And downgrades lead to failed deserialisation because the 'current' version is never specified, but isn't necessarily the 'latest', and that range has an open upper bound leading to a version by a future release not being detected as unsupported.)

[nfx]

we can

version_map[tuple[int,int,int], int] = {
   (0,37,0): 1,
   (0,39,0): 2,
   (0,52,0): 3,
}

[asnare]

Does this handle future versions, including later versions installed in other workspaces attached to the same metastore?

If we (a reader) are on version 0,54,0 then we'll assume version 3 (which is fine) but break if 0,55,0 introduced version 4 because that's indistinguishable to us: we will treat it was 3 (and fail) because only 0,55,0 or later knows version 4 exists.

Just encoding the integer avoids this because we can detect a future record (and that it cannot be decoded by the current version).

[nfx]

actually, better option would be:

migrate_after_versions[tuple[int,int,int], Callable[[dict[str,str]], dict[str,str]] = {
   (0,37,0): lambda x: x | {'foo': "bar"},
   (0,39,0): remove_x_column,
   (0,52,0): convert_some_datatype,
}

we sort by version tuple and get a diff between from databricks.labs.ucx.__about__ import __version__ and apply those. if we get a version that we don't understand - we warn and skip.

[asnare]

Just to aid my own understanding, and try to work through ambiguities, as far as I can tell there are two ways of interpreting a version tied to the UCX release:

• The records contain the first version that understands the serialised form. That is, version 0.37.0, 0.38.0, etc. write 0.37.0 in the record, until version 0.39.0 at which point version 0.39.0, 0.40, etc. all start writing 0.39.0. This continues until 0.52.0, etc.

  Under this protocol we can detect future/unknown versions because they're higher than __about__.__version__. We can also read past versions. It does however have a fairly large caveat: for all data classes we need to know the earliest version at which the 'current' version was introduced. This needs to be known in advance of the software release while we're writing and testing it and hard-coded somewhere. (We also need something special during development to ensure that we can read our own records.)

• The records contain the actual version that wrote the record.

  Under this protocol we can detect past versions, but not handle future/unknown versions because there's no way to know if a future release introduced a new version in some way. For safety we could just always ignore (and warn on) versions later than __about__.__version__. The main caveat here relates to when a downgrade happens. Mixed installs with a shared metastore could end up quite noisy (if not filtering out records from other workspaces).

@nfx: Is this also your understanding?

[nfx]

The records contain the actual version that wrote the record.

this is the intended way. let's store the version as ucx_version: list[int] and source it from __about__.__version__ and use databricks.sdk.mixins.compute.SemVer.parse to parse it.

[asnare]

That's clear, thanks.

[nfx]

lgtm

[github-actions]

✅ 29/29 passed, 1 flaky, 58m9s total

Flaky tests:

• 🤪 test_running_real_remove_backup_groups_job (8m40.994s)

_{Running from acceptance #6577}