panic when retstart the 0.7.0 client

[BabySid]

The 0.7.0 client paniced when I restarted it after synchronizing to the lastest block.

The log shows as below:

[2023-06-25T03:17:23Z INFO  ord::options] Connecting to Bitcoin Core at http://127.0.0.1:8332
[2023-06-25T03:17:23Z INFO  ord::options] Using credentials from cookie file at `../conf/cookie`
thread 'main' panicked at 'internal error: entered unreachable code', /user/.cargo/registry/src/github.com-1ecc6299db9ec823/redb-0.13.0/src/multimap_table.rs:110:18
stack backtrace:
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:114:5
   3: redb::multimap_table::parse_subtree_roots
   4: redb::db::Database::new
   5: redb::db::Builder::open_mmapped
   6: ord::index::Index::open
   7: ord::subcommand::Subcommand::run
   8: ord::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

How could I resolve the client?
It seems that it cannot be restarted after the client starts...

[Apetree100122]

The log shows as below:
[2023-06-25T03:17:23Z
INFO ord
options:
]
Connecting to Bitcoin Core
at
http://127.0.0.1:8332
[
2023-06-25
T
03:17:23
Z INFO
ord
options:
]
Using credentials from cookie
file at
*
/
conf:
/
cookie
thread
main
panicked at
internal error
entered unreachable
code
/user/.cargo/registry/src/github.com/1ecc6299db9ec823/0.2/src/multimap-table.rs/10:8
stack:
backtrace:
0:rust-begin-unwind
at
/
<rustc
/
84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc
/library/std/src/panicking.rs:579:51:core:panicki:panic-fmt>
at /
<rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:142:core:panicking:panic>
at </rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:114:5>
3:
'redb
multimap
table
parse
subtree
roots
4
redb
db:
Database:
new:
5
db:
Builder:
open
M:
mapped:
6
ord:
index:
Indexopen:
7
ordsubcommand:
Subcommandrun:
8
ordmain:
note:
Some details are omitted
run with
RUST
BACKTRACE
full
for a verbose backtrace
It seems that it can
be restarted after the
client starts
<Ctrl+Shift+>
<Ctrl+e>

[omarkilani]

FWIW I see the same issue.

Panic is here: https://github.com/cberner/redb/blob/v0.13.0/src/multimap_table.rs#L110

The data seems corrupt in the on-disk redb. I definitely shut down the ord server cleanly before restarting it, so I'm not sure what's going on --something about multi maps is unsafe in v0.13.0 of redb.

[omarkilani]

Okay, so the issue only occurs if the redb thinks it needs repair.

I'm not sure multimaps are crash safe or Control-C safe.

[jack-linden]

Also encountered this several times under the following circumstances

• ord index run terminated with ctrl+c
• ord index run terminated because ssh session logout
• Gracefully stopping the ord webserver ord server (probably the most concerning)

[so7ow]

FWIW I see the same issue.

Panic is here: https://github.com/cberner/redb/blob/v0.13.0/src/multimap_table.rs#L110

The data seems corrupt in the on-disk redb. I definitely shut down the ord server cleanly before restarting it, so I'm not sure what's going on --something about multi maps is unsafe in v0.13.0 of redb.

In my experience ord server consistently leaves the redb in a bad state. I don’t know if “corrupt” is the right word, but I reported this behavior months ago: #1861

[omarkilani]

FWIW I see the same issue.
Panic is here: https://github.com/cberner/redb/blob/v0.13.0/src/multimap_table.rs#L110
The data seems corrupt in the on-disk redb. I definitely shut down the ord server cleanly before restarting it, so I'm not sure what's going on --something about multi maps is unsafe in v0.13.0 of redb.

In my experience ord server consistently leaves the redb in a bad state. I don’t know if “corrupt” is the right word, but I reported this behavior months ago: #1861

In this case, if you log value on that line, it's just random data -- I think the multimap repair code path wasn't really tested until the index used multimaps for the 0.7.0 fixes.

In my case it was just gracefully stopping ord server.

[omarkilani]

Okay, tracked this down to this redb commit:

cberner/redb@23d1a70

If I make the same modification on a local redb 0.13.0 checkout, the recovery works.

Might be time to bump redb to 1.0.1 @raphjaph ?

[lnky79]

Okay, tracked this down to this redb commit:

cberner/redb@23d1a70

If I make the same modification on a local redb 0.13.0 checkout, the recovery works.

Might be time to bump redb to 1.0.1 @raphjaph ?

Raph has been working on a refactor PR for redb 0.20.0: #2141, so 1.0.1 seems far.

I just checkout a new disk space to reorg for 0.7.0 and encountered the issue too, not sure if it will be an issue for most people or there's an official solution on that @raphjaph

[satsquant]

also encountered the same issue under exactly the circumstances described by @jack-linden.

ord index run stopped by

• ssh logout
• ctrl-c

had to delete the index db and re-index again. A bit frustrating

[bitfrog009]

also encountered the same issue under exactly the circumstances described by @jack-linden.

ord index run stopped by

• ssh logout
• ctrl-c

had to delete the index db and re-index again. A bit frustrating

how do you delete the index db and re-index again, do you have a specific command for that ?

[satsquant]

also encountered the same issue under exactly the circumstances described by @jack-linden.
ord index run stopped by

• ssh logout
• ctrl-c

had to delete the index db and re-index again. A bit frustrating

how do you delete the index db and re-index again, do you have a specific command for that ?

just look for the index.redb file and delete it, then ord index run again. I actually ended up rolling back ord to 0.6.2 until the problem is solved because my index kept crashing

[bitfrog009]

same here, crushing again and again.

[bodily11]

Okay, tracked this down to this redb commit:

cberner/redb@23d1a70

If I make the same modification on a local redb 0.13.0 checkout, the recovery works.

Might be time to bump redb to 1.0.1 @raphjaph ?

I can confirm that when I pull in these changes to a local version of redb 0.13.0 it fixes the issue and I can reindex 0.7.0 fine even after ctrl-c or stopping the server.

[veryordinally]

Discussed with @raphjaph - plan is to rebase our 0.20x redb update PR to master and verify it also fixes the issue. Alternative might be a manually patched redb 0.13.x but we'd prefer the proper redb update.

[SatoKentaNayoro]

same here

[raphjaph]

#2141

[veryordinally]

Waiting for redb release, issue seems fixed in cberner/redb#627

[raphjaph]

Fixed by updating redb #2141

[nick07002]

Even with the fix, I am still getting error

thread 'main' panicked at 'internal error: entered unreachable code', /Users/xx/.cargo/registry/src/index.crates.io-6f17d22bba15001f/redb-1.0.2/src/tree_store/btree.rs:546:18
stack backtrace:
0: rust_begin_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
1: core::panicking::panic_fmt
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
2: core::panicking::panic
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
3: redb::tree_store::btree::RawBtree::verify_checksum_helper
4: redb::tree_store::table_tree::TableTree::verify_checksums
5: redb::db::Database::verify_primary_checksums
6: redb::db::Database::do_repair
7: redb::db::Database::new
8: redb::db::Builder::open
9: ord::index::Index::open
10: ord::subcommand::Subcommand::run
11: ord::main