[bug] index-bboltdb: Nil Pointer at /go.etcd.io/bbolt/freelist.go:265 #7445
Comments
|
another boltdb panic issue: this issue just delete the problem file, so this bug：Is this due to |
|
Immediately after panic log, I see the logs related to cleaning up. It seems that the boltdb file that cannot be opened will be automatically deleted. But why the file won't open is still a mystery. |
// Init downloads all the db files for the table from object storage.
func (t *indexSet) Init(forQuerying bool) (err error) {
// Using background context to avoid cancellation of download when request times out.
// We would anyways need the files for serving next requests.
ctx, cancelFunc := context.WithTimeout(context.Background(), downloadTimeout)
t.cancelFunc = cancelFunc
logger := spanlogger.FromContextWithFallback(ctx, t.logger)
defer func() {
if err != nil {
level.Error(t.logger).Log("msg", fmt.Sprintf("failed to initialize table %s, cleaning it up", t.tableName), "err", err)
t.err = err
// cleaning up files due to error to avoid returning invalid results.
for fileName := range t.index {
if err := t.cleanupDB(fileName); err != nil {
level.Error(t.logger).Log("msg", "failed to cleanup partially downloaded file", "filename", fileName, "err", err)
}
}
}
t.cancelFunc()
t.indexMtx.markReady()
}()
dirEntries, err := os.ReadDir(t.cacheLocation)
if err != nil {
return err
}
// open all the locally present files first to avoid downloading them again during sync operation below.
for _, entry := range dirEntries {
if entry.IsDir() {
continue
}
fullPath := filepath.Join(t.cacheLocation, entry.Name())
// if we fail to open an index file, lets skip it and let sync operation re-download the file from storage.
idx, err := t.openIndexFileFunc(fullPath)
if err != nil {
level.Error(t.logger).Log("msg", fmt.Sprintf("failed to open existing index file %s, removing the file and continuing without it to let the sync operation catch up", fullPath), "err", err)
// Sometimes files get corrupted when the process gets killed in the middle of a download operation which can cause problems in reading the file.
// Implementation of openIndexFileFunc should take care of gracefully handling corrupted files.
// Let us just remove the file and let the sync operation re-download it.
if err := os.Remove(fullPath); err != nil {
level.Error(t.logger).Log("msg", fmt.Sprintf("failed to remove index file %s which failed to open", fullPath))
}
continue
}
t.index[entry.Name()] = idx
}
level.Debug(logger).Log("msg", fmt.Sprintf("opened %d local files, now starting sync operation", len(t.index)))
// sync the table to get new files and remove the deleted ones from storage.
err = t.syncWithRetry(ctx, false, forQuerying)
if err != nil {
return
}
level.Debug(logger).Log("msg", "finished syncing files")
return
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A clear and concise description of what the bug is.
from slack loki channel @xzyl
https://grafana.slack.com/archives/CEPJRLQNL/p1666062333990409
Hello, when I query from Grafana explore, loki-read pod got some errors:
I deploy loki with helm ( chart loki-simple-scalable-1.8.11, loki 2.6.1), and a S3-like storage.
How should I fix this? Thx in advance.
nil pointer here.
p:= db.page(db.meta().freelist)
p==nil
p.flags panic
To Reproduce
Steps to reproduce the behavior:
{} termExpected behavior
Environment:
Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.
The text was updated successfully, but these errors were encountered: