Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd v3.4.13 compact failed after defrag #13841

Closed
nomorer opened this issue Mar 26, 2022 · 3 comments
Closed

etcd v3.4.13 compact failed after defrag #13841

nomorer opened this issue Mar 26, 2022 · 3 comments
Labels

Comments

@nomorer
Copy link

nomorer commented Mar 26, 2022

What happened?

I am running etcd V3.4.13 as cluster used by a large K8s Cluster, which db size over 4GB.
Etcd compaction is triggered by the api-server every 5min by default.

Why sometimes "finished scheduled compaction" is not showing up in etcd's log , after then Etcd request latencies increases (over 1s) when compacting
image

What did you expect to happen?

the compaction can be finished expectedly

How can we reproduce it (as minimally and precisely as possible)?

run a defrag for a endpoint.
it is unexpected behavior

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.4.13
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.16.6
Go OS/Arch: linux/amd64

Etcd configuration (command line flags or environment variables)

No response

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

No response

@nomorer
Copy link
Author

nomorer commented Mar 27, 2022

@tangcong

@lavacat
Copy link

lavacat commented Mar 30, 2022

run a defrag for a endpoint.

Are you running with --experimental-backend-bbolt-freelist-type=hashmap?
If yes, there is a bug that causes reset of bolt db options after defrag

#13713

@nomorer nomorer changed the title etcd v3.4.13 compaction fail etcd v3.4.13 compaction fail after defrag Mar 30, 2022
@nomorer nomorer changed the title etcd v3.4.13 compaction fail after defrag etcd v3.4.13 compact failed after defrag Mar 30, 2022
@nomorer nomorer closed this as completed Mar 31, 2022
@nomorer
Copy link
Author

nomorer commented Mar 31, 2022

yes.
compare to before, latency slow down significantly during compactions after the PR merged.
but etcd compaction is still abnormal, it doesn't print "finished scheduled compaction"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

2 participants