Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Updated Roadmap document #827

Merged
merged 1 commit into from
Dec 3, 2020
Merged

docs: Updated Roadmap document #827

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 55 additions & 50 deletions docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,107 +1,112 @@
# Amundsen Roadmap

The following roadmap gives an overview of what we are currently working on and what we want to tackle next. This helps potential contributors understand the current status of your project and where it's going next, as well as giving a chance to be part of the planning.

## Amundsen Mission
> *To organize all information about data and make it universally actionable*

## Vision for 2020
> *Centralize a comprehensive and actionable map of all our data resources that can be leveraged to solve a growing number of use cases and workflows*
> _To organize all information about data and make it universally actionable_

## Short Term - Our Current focus
## Vision for 2021

#### Provide Rich metadata to make data trust worthy
*What*: Enrich table detail page with additional structure metadata / programmatic description.
> _Centralize a comprehensive and actionable map of all our data resources that can be leveraged to solve a growing number of use cases and workflows_

*Status*: tech spec WIP
## Short Term - Our Current focus

#### Native lineage integration
*What*: We want to create a native lineage integration in Amundsen, to better surface how data assets interact with each other.

*Status*: tech spec out
_What_: We want to create a native lineage integration in Amundsen, to better surface how data assets interact with each other.
Copy link
Member

@feng-tao feng-tao Nov 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are only targeting table level lineage in short term, we may want to make it explicit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want both, and we have designs for both


_Status_: designs complete

#### Integrate with Data Quality system
*What*: Integrate with different data quality systems to provide quality score.

*Status*: planning
_What_: Integrate with different data quality systems to provide quality score.

_Status_: in progress

## Mid Term - Our Next steps
#### Improve search ranking
*What*: Update search ranking to be informed by "badges" that may exist on data sets e.g. deprecated, etc.

*Status*: planning
_What_: Overhaul search ranking to improve results.

#### Notifications when a table evolves
*What*: Notify users in Amundsen (akin to Facebook notifications or similar) when a table evolves. Owners of data and consumers of data will likely need to be notified of different things.
_Status_: planning

*Status*: planning has not started
#### Show distinct column values

#### Commonly joined tables / browsing the data model
*What*: As a data user, I would like to see commonly joined tables and how to join them.
One option would be to show commonly joined tables and showing example join queries. Another option would be to provide a navigational experience for data model, showing foreign keys and which tables they come from.
_What_: When a column has a limited set of possible values, we want to make then easily discoverable.

*Status*: planning has not started
_Status_: implementation started

## Mid Term - Our Next steps

#### Curated navigation experience
*What*: Currently Amundsen's experience is very focussed on search. However, especially for new users, an experience where they are able to navigate through the data hierarchy is very important. This item proposes to revamp the navigational experience in Amundsen (currently, barebones - based on tags) to do justice to the user need to browse through data sets when they don't know what to even search for.

*Status*: planning
_What_: Currently Amundsen's experience is very focussed on search. However, especially for new users, an experience where they are able to navigate through the data hierarchy is very important. This item proposes to revamp the navigational experience in Amundsen (currently, barebones - based on tags) to do justice to the user need to browse through data sets when they don't know what to even search for.

#### Push ingest API
*What*: We want to create a push API so that it is as easy as possible for a new data resource type to be ingested
_Status_: planning

*Status*: implementation has started (around 80% complete)
#### Notifications when a table evolves

#### GET Rest API
*What*: enable users to access our data map programmatically through a Rest API
_What_: Notify users in Amundsen (akin to Facebook notifications or similar) when a table evolves. Owners of data and consumers of data will likely need to be notified of different things.

*Status*: implementation has started
_Status_: planning has not started

#### Granular Access Control
*What*: we want to have a more granular control of the access. For example, only certain types of people would be able to see certain types of metadata/functionality
#### Commonly joined tables / browsing the data model

_What_: As a data user, I would like to see commonly joined tables and how to join them.
One option would be to show commonly joined tables and showing example join queries. Another option would be to provide a navigational experience for data model, showing foreign keys and which tables they come from.

*Status*: implementation has not started
_Status_: planning has not started

#### Show distinct column values
*What*: When a column has a limited set of possible values, we want to make then easily discoverable
#### Push ingest API

_What_: We want to create a push API so that it is as easy as possible for a new data resource type to be ingested

*Status*: implementation has not started
_Status_: not planned

#### Granular Access Control

#### “Order by” for columns
*What*: we want to help users make sense of what are the columns people use in the tables we index. Within a frequently used table, a column might not be used anymore because it is know to be deprecated
_What_: we want to have a more granular control of the access. For example, only certain types of people would be able to see certain types of metadata/functionality

*Status*: implementation has not started
_Status_: not planned

#### Versioning system
*What*: We want to create a versioning system for our indexed resources, to be able to index different versions of the same resource. This is especially required for machine learning purposes.

*Status*: implementation has not started
_What_: We want to create a versioning system for our indexed resources, to be able to index different versions of the same resource. This is especially required for machine learning purposes.

_Status_: not planned

#### Index Processes
*What*: we want to index ETLs and pipelines from our Machine Learning Engine

*Status*: implementation has not started
_What_: we want to index ETLs and pipelines from our Machine Learning Engine

_Status_: not planned

#### Index Teams
*What*: We want to add teams pages to enable users to see what are the important tables and dashboard a team uses

*Status*: implementation has not started
_What_: We want to add teams pages to enable users to see what are the important tables and dashboard a team uses

_Status_: not planned

#### Index Services
*What*: With our microservices architecture, we want to index services and show how these services interact with data artifacts

*Status*: implementation has not started
_What_: With our microservices architecture, we want to index services and show how these services interact with data artifacts

_Status_: not planned

#### Index S3 buckets
*What*: add these new resource types to our data map and create resource pages for them

*Status*: implementation has not started
_What_: add these new resource types to our data map and create resource pages for them

_Status_: not planned

#### Index Pub/Sub systems
*What*: We want to make our pub/sub systems discoverable

*Status*: implementation has not started
_What_: We want to make our pub/sub systems discoverable

_Status_: not planned

## How to Get Involved

Let us know in the [Slack channel](https://app.slack.com/client/TGFR0CZM3/CGFBVT23V) if you are interested in taking a stab at leading the development of one of these features.

You can also jump right in by tackling one of our issues labeled as ['help wanted'](https://github.com/amundsen-io/amundsen/labels/help%20wanted) or, if you are new to Amundsen, try one of our ['good first issue'](https://github.com/amundsen-io/amundsen/labels/good%20first%20issue) tickets.