Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add id to cluster manager's log #503

Merged
merged 1 commit into from
Feb 14, 2023
Merged

Add id to cluster manager's log #503

merged 1 commit into from
Feb 14, 2023

Conversation

masa213f
Copy link
Contributor

@masa213f masa213f commented Feb 3, 2023

This PR fixes the cluster-manager's log.

  • Generate an unique string for each cluster-manager's operation. And add it to the log events as operationId filed.
  • Output log events on start and end for cluster-manager's operation.
  • Record the cause of cluster-manager's operation as origin field.

After

{"level":"info","ts":"2023-02-08T10:55:39Z","msg":"detected mysql pod deletion","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"moco-single-0","namespace":"foo"},"namespace":"foo","name":"moco-single-0","reconcileID":"dd11da41-a28f-4b3f-9bde-a98bf9cebbbd","name":"moco-single-0"}
{"level":"info","ts":"2023-02-08T10:55:39Z","logger":"cluster-manager.foo/single","msg":"start operation","operationId":"op-6tklf","origin":"dd11da41-a28f-4b3f-9bde-a98bf9cebbbd"}
{"level":"error","ts":"2023-02-08T10:55:44Z","logger":"cluster-manager.foo/single","msg":"failed to get mysqld status","operationId":"op-6tklf","error":"failed to get global variables: pod=moco-single-0, namespace=foo: failed to get mysql global variables: dial tcp 10.244.3.31:33062: i/o timeout","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
{"level":"error","ts":"2023-02-08T10:55:52Z","logger":"cluster-manager.foo/single","msg":"failed to get mysqld status","operationId":"op-6tklf","error":"failed to get global variables: pod=moco-single-0, namespace=foo: failed to get mysql global variables: dial tcp 10.244.3.31:33062: i/o timeout","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
{"level":"error","ts":"2023-02-08T10:56:00Z","logger":"cluster-manager.foo/single","msg":"failed to get mysqld status","operationId":"op-6tklf","error":"failed to get global variables: pod=moco-single-0, namespace=foo: failed to get mysql global variables: dial tcp 10.244.3.31:33062: i/o timeout","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"update the status information","operationId":"op-6tklf"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"cluster state is Lost","operationId":"op-6tklf"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"finish","operationId":"op-6tklf","duration":24.065808786}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"start operation","operationId":"op-bq5jl","origin":"interval"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"update the status information","operationId":"op-bq5jl"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"cluster state is Incomplete","operationId":"op-bq5jl"}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"set read_only=0","operationId":"op-bq5jl","instance":0}
{"level":"info","ts":"2023-02-08T10:56:03Z","logger":"cluster-manager.foo/single","msg":"finish","operationId":"op-bq5jl","duration":0.163805937}

Before

{"level":"info","ts":"2023-02-08T06:27:59Z","msg":"detected mysql pod deletion","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"moco-single-0","namespace":"foo"},"namespace":"foo","name":"moco-single-0","reconcileID":"18a1933d-b42f-4ec5-82be-d4a1513851dc","name":"moco-single-0"}
{"level":"error","ts":"2023-02-08T06:28:05Z","logger":"cluster-manager.foo/single","msg":"failed to get mysqld status","error":"failed to get global variables: pod=moco-single-0, namespace=foo: failed to get mysql global variables: dial tcp 10.244.2.7:33062: i/o timeout","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
{"level":"error","ts":"2023-02-08T06:28:13Z","logger":"cluster-manager.foo/single","msg":"failed to get mysqld status","error":"failed to get global variables: pod=moco-single-0, namespace=foo: failed to get mysql global variables: dial tcp 10.244.2.7:33062: i/o timeout","stacktrace":"github.com/cybozu-go/moco/clustering.(*managerProcess).GatherStatus.func2\n\t/work/clustering/status.go:206"}
{"level":"info","ts":"2023-02-08T06:28:16Z","logger":"cluster-manager.foo/single","msg":"update the status information"}
{"level":"info","ts":"2023-02-08T06:28:16Z","logger":"cluster-manager.foo/single","msg":"cluster state is Lost"}
{"level":"info","ts":"2023-02-08T06:28:16Z","logger":"cluster-manager.foo/single","msg":"update the status information"}
{"level":"info","ts":"2023-02-08T06:28:16Z","logger":"cluster-manager.foo/single","msg":"cluster state is Incomplete"}
{"level":"info","ts":"2023-02-08T06:28:16Z","logger":"cluster-manager.foo/single","msg":"set read_only=0","instance":0}

Signed-off-by: Masayuki Ishii masa213f@gmail.com

@masa213f masa213f self-assigned this Feb 3, 2023
@masa213f masa213f changed the title Log WIP Log Feb 3, 2023
@masa213f masa213f changed the title WIP Log WIP: Debug log Feb 3, 2023
Signed-off-by: Masayuki Ishii <masa213f@gmail.com>
@masa213f masa213f changed the title WIP: Debug log Add id to cluster manager's log Feb 8, 2023
@masa213f masa213f marked this pull request as ready for review February 9, 2023 07:15
@masa213f masa213f requested a review from kmdkuk February 9, 2023 07:15
Copy link
Contributor

@kmdkuk kmdkuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@masa213f masa213f merged commit ccdcc45 into main Feb 14, 2023
@masa213f masa213f deleted the debug-log branch February 14, 2023 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants