Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partitions for some MySQL tables are not migrated to ClickHouse #461

Open
Selfeer opened this issue Feb 9, 2024 · 4 comments
Open

Partitions for some MySQL tables are not migrated to ClickHouse #461

Selfeer opened this issue Feb 9, 2024 · 4 comments
Labels
lightweight Issues related to Lightweight version mysql p3

Comments

@Selfeer
Copy link
Collaborator

Selfeer commented Feb 9, 2024

When creating a table in MySQL with the following structure, in some cases it seems that the partition is not migrated to the ClickHouse.

For example if we create a partitioned table in MySQL,

mysql> CREATE TABLE test3 (
    ->     order_id INT AUTO_INCREMENT,
    ->     product_name VARCHAR(255),
    ->     quantity INT,
    ->     order_date DATE,
    ->     PRIMARY KEY (order_id, order_date)
    -> ) 
    -> ENGINE = InnoDB
    -> PARTITION BY RANGE( YEAR(order_date) ) (
    ->     PARTITION p2020 VALUES LESS THAN (2021),
    ->     PARTITION p2021 VALUES LESS THAN (2022),
    ->     PARTITION p2022 VALUES LESS THAN (2023),
    ->     PARTITION p2023 VALUES LESS THAN (2024),
    ->     PARTITION p2024 VALUES LESS THAN (2025)
    -> );

In ClickHouse we get a table that is not PARTITIONED BY

SHOW CREATE TABLE test3;

CREATE TABLE test.test3
(
    `order_id` Int32,
    `product_name` Nullable(String),
    `quantity` Nullable(Int32),
    `order_date` Date32,
    `_sign` Int8,
    `_version` UInt64
)
ENGINE = ReplacingMergeTree(_version)
PRIMARY KEY (order_id, order_date)
ORDER BY (order_id, order_date)
SETTINGS index_granularity = 8192
@Selfeer
Copy link
Collaborator Author

Selfeer commented Feb 9, 2024

Related to: #332

@aadant
Copy link
Collaborator

aadant commented Feb 14, 2024

@Selfeer actually it would work if you use range columns which is recommended in MySQL

@Selfeer
Copy link
Collaborator Author

Selfeer commented Feb 14, 2024

@aadant but shouldn't it still work in the given example?

@aadant
Copy link
Collaborator

aadant commented Feb 16, 2024

The partitioning support is limited to the RANGE COLUMN case with date / datetime. Any other scheme is skipped.
It should be easy to support though.
You can start with the python code if you wish to contribute or just alter this table.

YEAR(order_date) can return NULL, there is no future partition. MySQL will probably scan the first partition as a result …

@subkanthi subkanthi added this to the 2.3.0 milestone Aug 2, 2024
@subkanthi subkanthi added mysql lightweight Issues related to Lightweight version labels Aug 2, 2024
@subkanthi subkanthi added the p3 label Aug 16, 2024
@subkanthi subkanthi modified the milestones: 2.3.0, 2.4.0 Aug 19, 2024
@subkanthi subkanthi removed this from the 2.4.0 milestone Sep 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lightweight Issues related to Lightweight version mysql p3
Projects
None yet
Development

No branches or pull requests

3 participants