Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restarting or removing the clickhouse-operator will cause Clickhouse to restart #855

Closed
czhfe opened this issue Jan 4, 2022 · 6 comments
Closed

Restarting or removing the clickhouse-operator will cause Clickhouse to restart #855

czhfe opened this issue Jan 4, 2022 · 6 comments
Labels
in testing

Comments

@czhfe
Copy link

czhfe commented Jan 4, 2022

Restarting or removing the clickhouse-operator will cause Clickhouse to restart

image
@czhfe
Copy link
Author

czhfe commented Jan 5, 2022

The logs are as follows:

I0104 07:16:33.770622       1 announcer.go:226] clickhouse_operator.go:87:Run():start
I0104 07:16:33.770712       1 announcer.go:224] clickhouse_operator.go:95:Run():Starting clickhouse-operator. Version:0.17.0 GitSHA:9731c23e BuiltAt:2021-12-01T18:48:33
I0104 07:16:33.772778       1 announcer.go:218] Parameters num: 11
I0104 07:16:33.772792       1 announcer.go:218] OPERATOR_CONTAINER_CPU_LIMIT=1
I0104 07:16:33.772795       1 announcer.go:218] OPERATOR_CONTAINER_CPU_REQUEST=1
I0104 07:16:33.772797       1 announcer.go:218] OPERATOR_CONTAINER_MEM_LIMIT=1073741824
I0104 07:16:33.772799       1 announcer.go:218] OPERATOR_CONTAINER_MEM_REQUEST=134217728
I0104 07:16:33.772801       1 announcer.go:218] OPERATOR_POD_IP=10.0.1.82
I0104 07:16:33.772803       1 announcer.go:218] OPERATOR_POD_NAME=altinity-clickhouse-operator-6c95d65d75-2mlts
I0104 07:16:33.772806       1 announcer.go:218] OPERATOR_POD_NAMESPACE=kube-system
I0104 07:16:33.772810       1 announcer.go:218] OPERATOR_POD_NODE_NAME=k8s-node4
I0104 07:16:33.772812       1 announcer.go:218] OPERATOR_POD_SERVICE_ACCOUNT=altinity-clickhouse-operator
I0104 07:16:33.772814       1 announcer.go:218] WATCH_NAMESPACE=
I0104 07:16:33.772816       1 announcer.go:218] WATCH_NAMESPACES=
I0104 07:16:33.775078       1 announcer.go:220] File-based ClickHouseOperatorConfigurations
I0104 07:16:33.775127       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace:
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-server)\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: kube-system
CHCredentialsSecretName: altinity-clickhouse-operator-secret
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold:
VModule:
LogBacktraceAt:
ReconcileThreadsNumber: 10
IncludeIntoPropagationAnnotations (0):
ExcludeFromPropagationAnnotations (0):
IncludeIntoPropagationLabels (0):
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
terminationGracePeriod: 30
I0104 07:16:33.842367       1 announcer.go:220] Unified (but not post-processed yet) CHOP config
I0104 07:16:33.842410       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace:
WatchNamespaces (0):
CHCommonConfigsPath: config.d
CHHostConfigsPath: conf.d
CHUsersConfigsPath: users.d
CHCommonConfigs (0):
CHHostConfigs (0):
CHUsersConfigs (0):
CHITemplatesPath: templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-server)\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: kube-system
CHCredentialsSecretName: altinity-clickhouse-operator-secret
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold:
VModule:
LogBacktraceAt:
ReconcileThreadsNumber: 10
IncludeIntoPropagationAnnotations (0):
ExcludeFromPropagationAnnotations (0):
IncludeIntoPropagationLabels (0):
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
terminationGracePeriod: 30
I0104 07:16:33.842863       1 announcer.go:220] Final CHOP config
I0104 07:16:33.842972       1 announcer.go:220] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>error</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>2d4cc92a0f88766f27f3d496c59a23dd7796c0dd0f23820758948a8ca25346da</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-server)\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: kube-system
CHCredentialsSecretName: altinity-clickhouse-operator-secret
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold:
VModule:
LogBacktraceAt:
ReconcileThreadsNumber: 10
IncludeIntoPropagationAnnotations (0):
ExcludeFromPropagationAnnotations (0):
IncludeIntoPropagationLabels (0):
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
terminationGracePeriod: 30
I0104 07:16:33.842986       1 announcer.go:218] Log option 'logtostderr' change value from 'true' to 'true'
I0104 07:16:33.842994       1 announcer.go:218] Log option 'alsologtostderr' change value from 'false' to 'false'
I0104 07:16:33.842997       1 announcer.go:218] Log option 'v' change value from '1' to '1'
I0104 07:16:33.843002       1 announcer.go:220] Additional log options applied
I0104 07:16:33.843015       1 announcer.go:220] clickhouse_operator.go:102:Run():Log options parsed
I0104 07:16:33.843046       1 announcer.go:226] ConfigFilePath: /etc/clickhouse-operator/config.yaml
ConfigFolderPath: /etc/clickhouse-operator
Namespace: kube-system
WatchNamespaces (0):
CHCommonConfigsPath: /etc/clickhouse-operator/config.d
CHHostConfigsPath: /etc/clickhouse-operator/conf.d
CHUsersConfigsPath: /etc/clickhouse-operator/users.d
CHCommonConfigs (4):
  - [01-clickhouse-01-listen.xml]=<yandex>
    <!-- Listen wildcard address to allow accepting connections from other containers and host network. -->
    <listen_host>::</listen_host>
    <listen_host>0.0.0.0</listen_host>
    <listen_try>1</listen_try>
</yandex>

  - [01-clickhouse-02-logger.xml]=<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>error</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>1</console>
    </logger>
</yandex>

  - [01-clickhouse-03-query_log.xml]=<yandex>
    <query_log replace="1">
        <database>system</database>
        <table>query_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>
    <query_thread_log remove="1"/>
</yandex>

  - [01-clickhouse-04-part_log.xml]=<yandex>
    <part_log replace="1">
        <database>system</database>
        <table>part_log</table>
        <engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + interval 30 day</engine>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </part_log>
</yandex>

CHHostConfigs (0):
CHUsersConfigs (3):
  - [01-clickhouse-user.xml]=<yandex>
    <users>
        <clickhouse_operator>
            <networks>
                <ip>127.0.0.1</ip>
                <ip>0.0.0.0/0</ip>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>2d4cc92a0f88766f27f3d496c59a23dd7796c0dd0f23820758948a8ca25346da</password_sha256_hex>
            <profile>clickhouse_operator</profile>
            <quota>default</quota>
        </clickhouse_operator>
    </users>
    <profiles>
        <clickhouse_operator>
            <log_queries>0</log_queries>
            <skip_unavailable_shards>1</skip_unavailable_shards>
            <http_connection_timeout>10</http_connection_timeout>
        </clickhouse_operator>
    </profiles>
</yandex>

  - [02-clickhouse-default-profile.xml]=<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

  - [03-database-ordinary.xml]=<!--  Remove it for ClickHouse versions before 20.4 -->
<yandex>
    <profiles>
        <default>
            <default_database_engine>Ordinary</default_database_engine>
        </default>
    </profiles>
</yandex>

CHITemplatesPath: /etc/clickhouse-operator/templates.d
CHITemplateFiles (0):
StatefulSetUpdateTimeout: 300
StatefulSetUpdatePollPeriod: 5
OnStatefulSetCreateFailureAction: ignore
OnStatefulSetUpdateFailureAction: rollback
CHConfigUserDefaultProfile: default
CHConfigUserDefaultQuota: default
CHConfigUserDefaultNetworksIP (2):
  - ::1
  - 127.0.0.1
CHConfigUserDefaultPassword: ***
CHConfigNetworksHostRegexpTemplate: (chi-{chi}-[^.]+\d+-\d+|clickhouse\-server)\.{namespace}\.svc\.cluster\.local$
CHUsername: ***
CHPassword: ***
CHCredentialsSecretNamespace: kube-system
CHCredentialsSecretName: altinity-clickhouse-operator-secret
CHCredentialsSecretUsername: ***
CHCredentialsSecretPassword: ***
CHPort: 8123
LogToStderr: true
AlsoLogToStderr: false
V: 1
StderrThreshold:
VModule:
LogBacktraceAt:
ReconcileThreadsNumber: 10
IncludeIntoPropagationAnnotations (0):
ExcludeFromPropagationAnnotations (0):
IncludeIntoPropagationLabels (0):
ExcludeFromPropagationLabels (0):
appendScopeLabels: no (false)
terminationGracePeriod: 30
I0104 07:16:33.927506       1 announcer.go:220] clickhouse_operator.go:145:Run():Starting CHI controller
I0104 07:16:33.927583       1 announcer.go:220] Starting ClickHouseInstallation controller
I0104 07:16:33.927598       1 announcer.go:218] waitForCacheSync():Syncing caches for ClickHouseInstallation controller
I0104 07:16:34.234962       1 announcer.go:218] ENQUEUE new ReconcileCHI cmd=add for clickhouse/clickhouse
I0104 07:16:34.828423       1 announcer.go:218] waitForCacheSync():Caches are synced for ClickHouseInstallation controller
E0104 07:16:34.934900       1 announcer.go:265] labeler.go:211:labelDeployment():kube-system/altinity-clickhouse-operator:ERROR put label on Deployment kube-system/altinity-clickhouse-operator Operation cannot be fulfilled on deployments.apps "altinity-clickhouse-operator": the object has been modified; please apply your changes to the latest version and try again
E0104 07:16:34.934928       1 announcer.go:265] controller.go:431:Run():ERROR label objects, will retry. Err: Operation cannot be fulfilled on deployments.apps "altinity-clickhouse-operator": the object has been modified; please apply your changes to the latest version and try again
I0104 07:16:39.963601       1 announcer.go:218] controller.go:440:Run():ClickHouseInstallation controller: starting workers number: 11
I0104 07:16:39.963636       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 1 out of 11
I0104 07:16:39.963658       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 2 out of 11
I0104 07:16:39.963670       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 3 out of 11
I0104 07:16:39.963681       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 4 out of 11
I0104 07:16:39.963700       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 5 out of 11
I0104 07:16:39.963716       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 6 out of 11
I0104 07:16:39.963737       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 7 out of 11
I0104 07:16:39.963753       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 8 out of 11
I0104 07:16:39.963770       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 9 out of 11
I0104 07:16:39.963785       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 10 out of 11
I0104 07:16:39.963797       1 announcer.go:218] controller.go:442:Run():ClickHouseInstallation controller: starting worker 11 out of 11
I0104 07:16:39.963822       1 announcer.go:220] controller.go:452:Run():ClickHouseInstallation controller: workers started
I0104 07:16:49.985945       1 announcer.go:220] markReconcileStart():clickhouse/clickhouse:reconcile started
I0104 07:16:50.005575       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:huis|sha:0|rep:0|host:0-0
I0104 07:16:50.005607       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:huis|sha:0|rep:1|host:0-1
I0104 07:16:50.005617       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:huis|sha:1|rep:0|host:1-0
I0104 07:16:50.005629       1 announcer.go:224] clickhouse/clickhouse:ADD host: ns:clickhouse|chi:clickhouse|clu:huis|sha:1|rep:1|host:1-1
W0104 07:16:50.016689       1 announcer.go:245] unable to create PDB poddisruptionbudgets.policy "clickhouse" already exists
I0104 07:16:50.016766       1 announcer.go:218] CreateServiceCHI():clickhouse/clickhouse:clickhouse/clickhouse-server
I0104 07:16:50.021093       1 announcer.go:218] updateService():clickhouse/clickhouse:Update Service clickhouse/clickhouse-server
I0104 07:16:50.046889       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I0104 07:16:50.141668       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I0104 07:16:50.157530       1 announcer.go:218] CreateServiceCluster():clickhouse/clickhouse:clickhouse/cluster-clickhouse-huis
I0104 07:16:50.157599       1 announcer.go:218] CreateServiceShard():clickhouse/clickhouse:clickhouse/shard-clickhouse-huis-0
I0104 07:16:50.157660       1 announcer.go:218] reconcileHost():clickhouse/clickhouse:Reconcile Host 0-0 started
I0104 07:16:50.377182       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-huis-0-0 use custom template clickhouse
I0104 07:16:50.377424       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-huis-0-0 container:clickhouse-pod mounted clickhouse-data on /var/lib/clickhouse
I0104 07:16:50.378692       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE EQUAL based on labels no reconcile is actually needed clickhouse/chi-clickhouse-huis-0-0
I0104 07:16:50.378728       1 announcer.go:218] excludeHost():clickhouse/clickhouse:Exclude from cluster host 0 shard 0 cluster huis
I0104 07:16:50.390707       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I0104 07:16:50.783479       1 announcer.go:220] IsHostInCluster():clickhouse/clickhouse:Host inside the cluster
I0104 07:16:55.789110       1 announcer.go:220] IsHostInCluster():clickhouse/clickhouse:Host outside of the cluster
I0104 07:16:55.789136       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/0-0-OK
I0104 07:16:55.789190       1 announcer.go:218] Run query on: chi-clickhouse-huis-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-huis-0-0.clickhouse.svc.cluster.local]
I0104 07:16:55.790770       1 announcer.go:218] pollHostContext():clickhouse/clickhouse:clickhouse/0-0-OK
I0104 07:16:55.797626       1 announcer.go:218] updateConfigMap():clickhouse/clickhouse:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-huis-0-0
I0104 07:16:55.812646       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-huis-0-0 use custom template clickhouse
I0104 07:16:55.812897       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-huis-0-0 container:clickhouse-pod mounted clickhouse-data on /var/lib/clickhouse
I0104 07:16:55.814128       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-huis-0-0
I0104 07:16:55.814194       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-huis-0-0) - started
I0104 07:16:55.863961       1 announcer.go:218] waitConfigMapPropagation():clickhouse/clickhouse:Wait for ConfigMap propagation for 59.948595925s 51.404075ms/1m0s
I0104 07:17:55.824367       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 4=>5
I0104 07:17:55.834961       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-huis-0-0:OK  :ObservedGeneration:5 Replicas:1 ReadyReplicas:1 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-huis-0-0-6fcb7449d9 UpdateRevision:chi-clickhouse-huis-0-0-6fcb7449d9
I0104 07:18:00.843120       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-huis-0-0:OK  :ObservedGeneration:5 Replicas:0 ReadyReplicas:0 CurrentReplicas:0 UpdatedReplicas:0 CurrentRevision:chi-clickhouse-huis-0-0-6fcb7449d9 UpdateRevision:chi-clickhouse-huis-0-0-6fcb7449d9
I0104 07:18:00.861247       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-huis-0-0) - completed
I0104 07:18:00.877502       1 announcer.go:218] getPodTemplate():clickhouse/clickhouse:statefulSet chi-clickhouse-huis-0-0 use custom template clickhouse
I0104 07:18:00.877725       1 announcer.go:218] setupStatefulSetApplyVolumeMount():clickhouse/clickhouse:StatefulSet:chi-clickhouse-huis-0-0 container:clickhouse-pod mounted clickhouse-data on /var/lib/clickhouse
I0104 07:18:00.878922       1 announcer.go:224] getStatefulSetStatus():clickhouse/clickhouse:INFO StatefulSet ARE DIFFERENT based on labels. Reconcile is required for clickhouse/chi-clickhouse-huis-0-0
I0104 07:18:00.878997       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:Update StatefulSet(clickhouse/chi-clickhouse-huis-0-0) - started
I0104 07:18:00.893139       1 announcer.go:218] waitConfigMapPropagation():clickhouse/clickhouse:No need to wait for ConfigMap propagation - already elapsed. 1m5.080581567s/1m0s
I0104 07:18:00.897257       1 announcer.go:218] updateStatefulSet():clickhouse/clickhouse:generation change 5=>6
I0104 07:18:00.903342       1 announcer.go:218] deleteLabelReady():clickhouse/clickhouse:FAIL get pod for host clickhouse/0-0 err:pods "chi-clickhouse-huis-0-0-0" not found
I0104 07:18:05.912402       1 announcer.go:218] pollStatefulSet():clickhouse/chi-clickhouse-huis-0-0:OK  :ObservedGeneration:6 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-clickhouse-huis-0-0-6fcb7449d9 UpdateRevision:chi-clickhouse-huis-0-0-6fcb7449d9

@sunsingerus
Copy link
Collaborator

It would be helpful to see CHI manifest.

@czhfe
Copy link
Author

czhfe commented Jan 10, 2022

It would be helpful to see CHI manifest.

apiVersion: "clickhouse.altinity.com/v1"
kind: ClickHouseInstallation
metadata:
  name: clickhouse
spec:
  restart: "RollingUpdate"
  defaults:
    templates: 
      dataVolumeClaimTemplate: clickhouse-data
      podTemplate: clickhouse
      serviceTemplate: clickhouse-default
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper-0.zookeeper-headless
          port: 2181
        - host: zookeeper-1.zookeeper-headless
          port: 2181
        - host: zookeeper-2.zookeeper-headless
          port: 2181
    clusters:
      - name: huis
        layout:
          shardsCount: 2
          replicasCount: 2
    profiles:
      default/allow_experimental_map_type: 1
    settings:
      disable_internal_dns_cache: 1
      prometheus/asynchronous_metrics: "true"
      prometheus/endpoint: /metrics
      prometheus/events: "true"
      prometheus/metrics: "true"
      prometheus/port: "8001"
      prometheus/status_info: "true"
    users:
      clickhouse_admin/networks/ip: "::/0"
      clickhouse_admin/password: "xxx"
      clickhouse_admin/profile: default
      clickhouse_admin/access_management: 1
  templates:
    podTemplates:
      - name: clickhouse
        podDistribution:
          - type: ShardAntiAffinity
            scope: Shard
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:21.6.5.37
              ports:
              - name: metrics
                containerPort: 8001
              resources:
                requests:
                  memory: "512Mi"
                  cpu: "100m"
                limits:
                  memory: "2048Mi"
                  cpu: "1"
    volumeClaimTemplates:
      - name: clickhouse-data
        reclaimPolicy: Retain
        spec:
          storageClassName: rook-ceph-block-delete
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
    serviceTemplates:
      - name: clickhouse-default
        generateName: clickhouse-server
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: ClusterIP

@sozenh
Copy link

sozenh commented Jan 18, 2022

@czhfe Since you specified spec.restart, this is what caused the problem.

see #853 for more information.

@sunsingerus
Copy link
Collaborator

@czhfe
@su-houzhen
Version 0.18.0 of the operator is about to be released, which has this issue fixed. It is stable enough, so feel free to test.

@sunsingerus sunsingerus mentioned this issue Jan 18, 2022
Closed
@alex-zaitsev
Copy link
Member

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/0.18.0

@czhfe czhfe mentioned this issue Feb 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
in testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@czhfe @sunsingerus @alex-zaitsev @sozenh