Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] mysql load large data local INFILE causes primary/secondary switch #8928

Closed
JashBook opened this issue Feb 14, 2025 · 1 comment
Closed
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@JashBook
Copy link
Collaborator

Describe the bug
mysql load large data local INFILE causes primary/secondary switch

kbcli version
Kubernetes: v1.30.6
KubeBlocks: 0.9.3-beta.24
kbcli: 0.9.3-beta.3

To Reproduce
Steps to reproduce the behavior:

  1. create mysql
kubectl apply -f - <<EOF
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: mysql-cluster
  namespace: default
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: mysql
      componentDef: mysql-8.0
      serviceVersion: 8.0.39
      replicas: 2
      resources:
        requests:
          cpu: 8
          memory: 16Gi
        limits:
          cpu: 16
          memory: 32Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 2Ti
EOF
kbcli cluster expose mysql-cluster --auto-approve --force=true --type internet --enable true --components mysql
kubectl get secret mysql-cluster-mysql-account-root -ojsonpath='{.data.username}' | base64 -d
kubectl get secret mysql-cluster-mysql-account-root -ojsonpath='{.data.password}' | base64 -d


kbcli cluster list-instances mysql-cluster 
NAME                    NAMESPACE   CLUSTER         COMPONENT   STATUS    ROLE        ACCESSMODE   AZ   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                                             CREATED-TIME                 
mysql-cluster-mysql-0   default     mysql-cluster   mysql       Running   primary     <none>       0    8 / 16               16Gi / 32Gi             data:2Ti   aks-backuplarge-28239716-vmss000003/10.224.0.7   Feb 14,2025 13:07 UTC+0800   
mysql-cluster-mysql-1   default     mysql-cluster   mysql       Running   secondary   <none>       0    8 / 16               16Gi / 32Gi             data:2Ti   aks-backuplarge-28239716-vmss000000/10.224.0.9   Feb 14,2025 13:07 UTC+0800   
➜  ~ 
  1. create tpch table
kbcli cluster connect mysql-cluster
create database backupdb;
use backupdb;
CREATE TABLE IF NOT EXISTS CUSTOMER ( C_CUSTKEY     INTEGER NOT NULL,C_NAME        VARCHAR(25) NOT NULL,C_ADDRESS     VARCHAR(40) NOT NULL,C_NATIONKEY   INTEGER NOT NULL,C_PHONE       CHAR(15) NOT NULL,C_ACCTBAL     DECIMAL(15,2)   NOT NULL,C_MKTSEGMENT  CHAR(10) NOT NULL,C_COMMENT     VARCHAR(117) NOT NULL);

  1. load data
echo "use backupdb;load data local INFILE './tpch/customer.tbl' INTO TABLE CUSTOMER FIELDS TERMINATED BY '|';" | mysql --local-infile=1 -h*** -uroot -P3306 -p'***' -A
  1. See error
kbcli cluster list-instances mysql-cluster
NAME                    NAMESPACE   CLUSTER         COMPONENT   STATUS    ROLE        ACCESSMODE   AZ   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                                             CREATED-TIME                 
mysql-cluster-mysql-0   default     mysql-cluster   mysql       Running   secondary   <none>       0    8 / 16               16Gi / 32Gi             data:2Ti   aks-backuplarge-28239716-vmss000003/10.224.0.7   Feb 14,2025 13:07 UTC+0800   
mysql-cluster-mysql-1   default     mysql-cluster   mysql       Running   primary     <none>       0    8 / 16               16Gi / 32Gi             data:2Ti   aks-backuplarge-28239716-vmss000000/10.224.0.9   Feb 14,2025 13:07 UTC+0800   

pod 0

kbcli cluster connect -i  mysql-cluster-mysql-0
mysql> select count(*) from backupdb.CUSTOMER;
+-----------+
| count(*)  |
+-----------+
| 150000000 |
+-----------+
1 row in set (21.81 sec)

mysql> show master status\G;
*************************** 1. row ***************************
             File: mysql-cluster-mysql-0-bin.000004
         Position: 2070389
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: ac63c90e-ea91-11ef-b5b7-d2339c44a7b1:1-3977,
b1e91723-ea91-11ef-953f-16220e5454d8:1-5074
1 row in set (0.00 sec)

ERROR: 
No query specified

show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: mysql-cluster-mysql-1.mysql-cluster-mysql-headless
                  Master_User: kbreplicator
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-cluster-mysql-1-bin.000003
          Read_Master_Log_Pos: 3667593
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 2048375
        Relay_Master_Log_File: mysql-cluster-mysql-1-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 3667593
              Relay_Log_Space: 2048579
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: Yes
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: b1e91723-ea91-11ef-953f-16220e5454d8
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: b1e91723-ea91-11ef-953f-16220e5454d8:1-5107
            Executed_Gtid_Set: ac63c90e-ea91-11ef-b5b7-d2339c44a7b1:1-3977,
b1e91723-ea91-11ef-953f-16220e5454d8:1-5107
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

ERROR: 
No query specified

pod 1

kbcli cluster connect -i  mysql-cluster-mysql-1
mysql> select count(*) from backupdb.CUSTOMER;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)


mysql> show master status\G;
*************************** 1. row ***************************
             File: mysql-cluster-mysql-1-bin.000003
         Position: 3655563
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: ac63c90e-ea91-11ef-b5b7-d2339c44a7b1:1-3975,
b1e91723-ea91-11ef-953f-16220e5454d8:1-5077
1 row in set (0.00 sec)

ERROR: 
No query specified

mysql> show slave status\G;
Empty set, 1 warning (0.00 sec)

ERROR: 
No query specified

Expected behaviorA clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@JashBook JashBook added the kind/bug Something isn't working label Feb 14, 2025
@xuriwuyun
Copy link
Contributor

Image
When executing 'LOAD DATA LOCAL INFILE' SQL with a large file, the master server becomes overloaded, making it inaccessible.

@github-actions github-actions bot added this to the Release 0.9.3 milestone Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants