Feature Proposal: Support MySQL as metadata store in Databuilder

[xuan616]

My workplace is going to use Amundsen with MySQL as metadata store but currently, Amundsen only supports Graph DB as the backend metadata store. We plan to do the dev work to integrate Amundsen with MySQL. There would be some work to do in databuilder and metaservice and I created this issue for any possible work needed to do in databuilder.

Expected Behavior or Use Case

The workflow would be similar to the existing one and backend store would have relational DB options, like MySQL

Service or Ingestion ETL

Data loader, publisher and models in Databuilder

Possible Implementation

1. Currently, Amundsen does not support any relational DB as backend store, so we have to support ORM first with 'SQLAlchemy' for relational DBs. Considering the possible metadata schema change, we have to add a feature to support DB version management with 'alembic'.

2. We may need new models that work with ORM and probably different fields from the existing model. I would also like to know if it is possible to update all current models to let them work with both graph and relational DBs? I assume we still use current ETL logic for the metadata to be pushed to MySQL, so for the model used for mysql, is it good to develop new iterator working for each row instead of 'node' and 'relationship' in model?

3. Add data loader that work for MySQL

4. Add MySQL publisher

All the above is possible implementation looking forward a better solution. If I missed anything or the proposal is not proper, please advise. Thanks.

Example Screenshots (if appropriate):

Context

We are not sure if there is a plan to support relation DB from Amundsen side and in order to use Amundsen and MySQL as the metadata engine in my workplace, we plan to do the dev work to support MySQL as backend store.

[feng-tao]

will take a look next week.

[feng-tao]

also we should create a rfc in https://github.com/amundsen-io/rfcs . And it would be good to be more explicit about how to map the graph model to ORM model .

In terms of databuilder, @AndrewCiambrone has a proposal( amundsen-io/rfcs#5) to make the entity more generic.

[feng-tao]

here are a few additional thoughts:
1). we initial target graph model because it is easier to model metadata entity connection through a graph (e.g we could easily add more additional structure metadata(e.g update time, create time, etc) for the entity by just adding a new node with connection to the entity(e.g dataset). It would be good to understand what is ORM model looks like (e.g will it be a denormalized model which each entity has all the metadata inside a single schema?) This would be good to be more explicit for the RFC.
2) Depends on 1, I haven't thought about how to modify the existing model to an ORM model. cc @jinhyukchang as well

[xuan616]

@feng-tao Thanks for your reply. Yes, Graph DB has some advantages on metadata store and it is good to know the work of @AndrewCiambrone, which will bring more generic graph db models. For the ORM model, we will probably have to make a trade-off between the normalized and denormalized models for metadata store due to some concerns, like performance, consistency issues. Also, the model will need support schema upgrade to handle possible metadata structure change. I am trying to figure out how to map graph DB model to ORM model, probably with the incoming abstraction layer for an explicit plan.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Golodhros]

Done in https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/mysql_metadata_extractor.py