Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic while composing an ostree container native image in container mode #4646

Closed
travier opened this issue Oct 9, 2023 · 6 comments · Fixed by ostreedev/ostree-rs-ext#555
Closed

Panic while composing an ostree container native image in container mode #4646

travier opened this issue Oct 9, 2023 · 6 comments · Fixed by ostreedev/ostree-rs-ext#555

Comments

@travier
Copy link
Member

travier commented Oct 9, 2023

Manifest:

releasever: 38

container: true
# selinux: false

recommends: false

packages:
  # We are creating a container
  - fedora-release-container
  # Needed for rpmdb
  - rpm
  # Application to effectively install
  - nginx

# exclude-packages:
#   - dbus-broker
#   - systemd

repos:
  - fedora
  - fedora-updates

postprocess:
  - |
    #!/usr/bin/env bash
    set -xeuo pipefail

    # Remove all systemd configs that we will not be using
    rm -rf /usr/lib/systemd/ /etc/systemd/

# Remove all docs
remove-files:
  - usr/share/info
  - usr/share/man
  - usr/share/doc

Exact error:

thread 'tokio-runtime-worker' panicked at 'slice index starts at 36 but ends at 27', /builddir/build/BUILD/rustc-1.72.1-src/library/alloc/src/vec/mod.rs:1993:36

Full backtrace below.

Host system details

Provide the output of rpm-ostree status.

Fedora 38 container, rpm-ostree 2023.8-2

Expected vs actual behavior

No panic while composing a container image.

Steps to reproduce it

See: https://github.com/travier/ostree-containers and job: https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1 from travier/ostree-containers#1.

Full backtrace:

thread 'tokio-runtime-worker' panicked at 'slice index starts at 36 but ends at 27', /builddir/build/BUILD/rustc-1.72.1-src/library/alloc/src/vec/mod.rs:1993:36
stack backtrace:
   0:     0x564bd74a1611 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h35073a9e56bccd9e
   1:     0x564bd74d4baf - core::fmt::write::h5f7e0780d5d3bf57
   2:     0x564bd749b8e9 - std::io::Write::write_fmt::h3b815ddcd433bacb
   3:     0x564bd74a1425 - std::sys_common::backtrace::print::he80699aa34d7dd8e
   4:     0x564bd74a2c43 - std::panicking::default_hook::{{closure}}::hb9c74eb24233d20e
   5:     0x564bd74a29d4 - std::panicking::default_hook::h4db4f15e70760494
   6:     0x564bd74a31c9 - std::panicking::rust_panic_with_hook::h05b3acb3dfba5d03
   7:     0x564bd74a30c7 - std::panicking::begin_panic_handler::{{closure}}::hd4ae69032d8549b8
   8:     0x564bd74a1a76 - std::sys_common::backtrace::__rust_end_short_backtrace::h702380e0acea7e95
   9:     0x564bd74a2e12 - rust_begin_unwind
  10:     0x564bd6a48413 - core::panicking::panic_fmt::h8cf040903fbd6815
  11:     0x564bd6a48a32 - core::slice::index::slice_index_order_fail::h513297e3b05b6361
  12:     0x564bd72feae5 - core::slice::index::range::hd218bb08f12459e2
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/core/src/slice/index.rs:720:9
  13:     0x564bd72feae5 - alloc::vec::Vec<T,A>::drain::h1f74039689770bc8
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/alloc/src/vec/mod.rs:1993:36
  14:     0x564bd72feae5 - ostree_ext::chunking::get_partitions_with_threshold::hde1972078ac47f66
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:465:38
  15:     0x564bd72feae5 - ostree_ext::chunking::basic_packing::hd0d336e8e174f152
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:687:26
  16:     0x564bd72faba7 - ostree_ext::chunking::Chunking::process_mapping::hd3890a5f21ad1e64
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:324:23
  17:     0x564bd7368f23 - ostree_ext::chunking::Chunking::from_mapping::h7a7fcceccccf2133
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/chunking.rs:277:9
  18:     0x564bd7368f23 - ostree_ext::container::encapsulate::build_oci::{{closure}}::{{closure}}::ha7650087a68c8b16
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:236:13
  19:     0x564bd7368f23 - core::option::Option<T>::map::h385098f247a4028d
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/core/src/option.rs:1075:29
  20:     0x564bd7368f23 - ostree_ext::container::encapsulate::build_oci::{{closure}}::hf082cc4fffd1fbc4
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:235:10
  21:     0x564bd7368f23 - ostree_ext::container::encapsulate::build_oci::h8d583fa5289addae
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:182:1
  22:     0x564bd6e2d031 - ostree_ext::container::encapsulate::build_impl::{{closure}}::{{closure}}::h35748af3dfea0bab
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:345:23
  23:     0x564bd6e2d031 - ostree_ext::container::encapsulate::build_impl::{{closure}}::h43ccbc6a1b5b279a
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:320:1
  24:     0x564bd6e4243f - ostree_ext::container::encapsulate::encapsulate::{{closure}}::h127e63cc8d7e3083
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/ostree-ext/src/container/encapsulate.rs:412:63
  25:     0x564bd6e4243f - rpmostree_rust::container::container_encapsulate::{{closure}}::{{closure}}::hd0d6ca23ffa3b7c1
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/container.rs:456:18
  26:     0x564bd6e4243f - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h98ae8a0785467a77
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/park.rs:282:63
  27:     0x564bd6e4243f - tokio::runtime::coop::with_budget::h7b479efdcd79c72f
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/coop.rs:107:5
  28:     0x564bd6e4243f - tokio::runtime::coop::budget::h785512ff84853072
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/coop.rs:73:5
  29:     0x564bd6e4243f - tokio::runtime::park::CachedParkThread::block_on::h0f3da393884e7183
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/park.rs:282:31
  30:     0x564bd6fccb9c - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::hfc14a57d4dcef743
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/context/blocking.rs:66:9
  31:     0x564bd6fccb9c - tokio::runtime::handle::Handle::block_on::{{closure}}::hcc84bf8fc7c33818
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/handle.rs:310:22
  32:     0x564bd6fccb9c - tokio::runtime::context::runtime::enter_runtime::hc289e9840fcb6d3e
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/context/runtime.rs:65:16
  33:     0x564bd704c72c - tokio::runtime::handle::Handle::block_on::hdb3a01ac46b19a97
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/handle.rs:309:9
  34:     0x564bd704c72c - rpmostree_rust::container::container_encapsulate::{{closure}}::haff05f7ffa75603c
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/container.rs:454:16
  35:     0x564bd704c72c - rpmostree_rust::progress::progress_task::h1581555f0dc0942c
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/progress.rs:14:5
  36:     0x564bd6dee1d4 - rpmostree_rust::container::container_encapsulate::h0a105babff13f85a
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/container.rs:453:18
  37:     0x564bd6fb95ad - rpmostree_rust::ffi::_::__container_encapsulate::__container_encapsulate::h1dc669a567641fb5
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/lib.rs:[176](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:177):12
  38:     0x564bd6fb95ad - rpmostree_rust::ffi::_::__container_encapsulate::{{closure}}::hdebf9db5156ab6ff
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/lib.rs:47:1
  39:     0x564bd6fb95ad - cxx::unwind::prevent_unwind::hdb39e390d3e85cd3
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/cxx/src/unwind.rs:23:15
  40:     0x564bd6fb95ad - rpmostreecxx$cxxbridge1$container_encapsulate
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/lib.rs:176:66
  41:     0x564bd6c3766d - _ZN12rpmostreecxx21container_encapsulateEN4rust10cxxbridge13VecINS1_6StringEEE
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rpmostree-cxxrs.cxx:3662:79
  42:     0x564bd6c3766d - rpmostree_compose_builtin_container_encapsulate
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/src/app/rpmostree-compose-builtin-tree.cxx:[177](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:178)3:3
  43:     0x564bd6bd8d98 - rpmostree_handle_subcommand
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/src/app/libmain.cxx:442:33
  44:     0x564bd6bdd91e - _ZN12rpmostreecxx14rpmostree_mainEN4rust10cxxbridge15SliceIKNS1_3StrEEE
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/src/app/libmain.cxx:548:20
  45:     0x564bd6bd6944 - operator()
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rpmostree-cxxrs.cxx:2980:56
  46:     0x564bd6bd6944 - trycatch<rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>, int32_t*)::<lambda()>, rust::cxxbridge1::detail::Fail>
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rpmostree-cxxrs.cxx:1280:10
  47:     0x564bd6bd6944 - rpmostreecxx$cxxbridge1$rpmostree_main
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rpmostree-cxxrs.cxx:2978:32
  48:     0x564bd6fb872c - rpmostree_rust::ffi::rpmostree_main::hb2da0d5324a44ce5
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/lib.rs:824:12
  49:     0x564bd6afccb4 - rpm_ostree::inner_async_main::{{closure}}::{{closure}}::ha5f64927165f8f22
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/rust/src/main.rs:43:25
  50:     0x564bd6afccb4 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hf36b406d90719669
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/blocking/task.rs:42:21
  51:     0x564bd6b78583 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hbbf226bcdd[190](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:191)373
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/core.rs:334:17
  52:     0x564bd6b78583 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h0a2936510cd20f35
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/loom/std/unsafe_cell.rs:16:9
  53:     0x564bd6b78583 - tokio::runtime::task::core::Core<T,S>::poll::hbe6601aff2d3d58e
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/core.rs:323:30
  54:     0x564bd6b4b65b - tokio::runtime::task::harness::poll_future::{{closure}}::h904beea9d34cd69f
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:485:19
  55:     0x564bd6b4b65b - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8314e13d90ed91b4
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/core/src/panic/unwind_safe.rs:271:9
  56:     0x564bd6b4b65b - std::panicking::try::do_call::h7c5f326b2dcc6635
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panicking.rs:500:40
  57:     0x564bd6b4b65b - std::panicking::try::hcdd613bcbdb6cbb3
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panicking.rs:464:19
  58:     0x564bd6b4b65b - std::panic::catch_unwind::h7deb0dcabdb2a464
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panic.rs:142:14
  59:     0x564bd6b4b65b - tokio::runtime::task::harness::poll_future::h38f4427fd37c7bfa
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:473:18
  60:     0x564bd6b4b65b - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h558978900b62231c
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:208:27
  61:     0x564bd6b4b65b - tokio::runtime::task::harness::Harness<T,S>::poll::h9c8615c05a3a41fd
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/harness.rs:153:15
  62:     0x564bd74570ad - tokio::runtime::task::raw::RawTask::poll::h175bbe491b4743ed
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/raw.rs:[200](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:201):18
  63:     0x564bd74570ad - tokio::runtime::task::UnownedTask<S>::run::h64bcc460e510caa7
                               at /usr/src/debug/rpm-ostree-[202](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:203)3.8-2.fc38.x86_64/vendor/tokio/src/runtime/task/mod.rs:437:9
  64:     0x564bd74570ad - tokio::runtime::blocking::pool::Task::run::h1639872ed7e816b3
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:159:9
  65:     0x564bd74570ad - tokio::runtime::blocking::pool::Inner::run::h474b64f49f0fe58b
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:513:17
  66:     0x564bd74481ca - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hff5f26abb59d0aca
                               at /usr/src/debug/rpm-ostree-2023.8-2.fc38.x86_64/vendor/tokio/src/runtime/blocking/pool.rs:471:13
  67:     0x564bd74481ca - std::sys_common::backtrace::__rust_begin_short_backtrace::h75f3036ab051eedd
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/sys_common/backtrace.rs:135:18
  68:     0x564bd74488d9 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hb6fea9448081aa29
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/thread/mod.rs:529:17
  69:     0x564bd74488d9 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h984aad30de781aef
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/core/src/panic/unwind_safe.rs:271:9
  70:     0x564bd74488d9 - std::panicking::try::do_call::h08b9ec626b517ff5
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panicking.rs:500:40
  71:     0x564bd74488d9 - std::panicking::try::habd904054c9104bf
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panicking.rs:464:19
  72:     0x564bd74488d9 - std::panic::catch_unwind::h9e4da63ebcb79fd1
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/panic.rs:142:14
  73:     0x564bd74488d9 - std::thread::Builder::spawn_unchecked_::{{closure}}::h075bbfd023ea7fb0
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/std/src/thread/mod.rs:528:30
  74:     0x564bd74488d9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h62f07726093de85b
                               at /builddir/build/BUILD/rustc-1.72.1-src/library/core/src/ops/function.rs:250:5
  75:     0x564bd74aa1c5 - std::sys::unix::thread::Thread::new::thread_start::hbb8f49fe2903a6dd
  76:     0x7fbc15[229](https://github.com/travier/ostree-containers/actions/runs/6459112989/job/17534243713?pr=1#step:5:230)947 - start_thread
  77:     0x7fbc152af860 - __clone3
  78:                0x0 - <unknown>

Would you like to work on the issue?

If nobody gets to it before me :).
@travier
Copy link
Member Author

travier commented Oct 9, 2023

Might be an ostree-ext bug from the backtrace.

@travier
Copy link
Member Author

travier commented Oct 9, 2023

From https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.drain:

Panics
Panics if the starting point is greater than the end point or if the end point is greater than the length of the vector.

Which appears to be the case here:

panicked at 'slice index starts at 36 but ends at 27', /builddir/build/BUILD/rustc-1.72.1-src/library/alloc/src/vec/mod.rs:1993:36

So it looks like https://github.com/ostreedev/ostree-rs-ext/blob/main/lib/src/chunking.rs#L465 does not handle this case correctly.

travier added a commit to travier/ostree-rs-ext that referenced this issue Oct 9, 2023
@travier
chunking: Handle low amount of high-size packages
c598440 
If we have a low amount of high-size packages then we will drain with an
index higher than the size of the Vec which triggers a panic.

Fixes: coreos/rpm-ostree#4646
@travier travier mentioned this issue Oct 9, 2023
Merged
@travier
Copy link
Member Author

travier commented Oct 9, 2023

Tentative fix in ostreedev/ostree-rs-ext#555 as I've not tested it. (I need to learn how to update the vendored code locally).

travier added a commit to travier/ostree-rs-ext that referenced this issue Oct 9, 2023
@travier
chunking: Handle low amount of high-size packages
5724349 
If we have a low amount of high-size packages then we will drain with an
index higher than the size of the Vec which triggers a panic.

Fixes: coreos/rpm-ostree#4646
travier added a commit to travier/ostree-rs-ext that referenced this issue Oct 9, 2023
@travier
chunking: Handle low amount of high-size packages
70f918f 
If we have a low amount of high-size packages then we will drain with an
index higher than the size of the Vec which triggers a panic.

Fixes: coreos/rpm-ostree#4646
@cgwalters
Copy link
Member

(I need to learn how to update the vendored code locally).

See https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html#the-patch-section

travier added a commit to travier/ostree-rs-ext that referenced this issue Oct 10, 2023
@travier
chunking: Handle low amount of high-size packages
c6e5aa7 
If we have a low amount of high-size packages then we will drain with an
index higher than the size of the Vec which triggers a panic.

Fixes: coreos/rpm-ostree#4646
@cgwalters cgwalters reopened this Oct 10, 2023
@travier
Copy link
Member Author

travier commented Oct 11, 2023

Example patch for reference:

diff --git a/Cargo.toml b/Cargo.toml
index b70db64f..2ce931f7 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -72,7 +72,7 @@ nix = { version = "0.27.1", features = ["fs", "mount", "signal", "user"] }
 openssl = "0.10.57"
 once_cell = "1.18.0"
 os-release = "0.1.0"
-ostree-ext = "0.12"
+ostree-ext = { path = '../ostree-rs-ext/lib/' }
 paste = "1.0"
 phf = { version = "0.11", features = ["macros"] }
 rand = "0.8.5"

@cgwalters
Copy link
Member

Should be fixed by #4665

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chunking: Handle low amount of high-size packages travier/ostree-rs-ext
2 participants
@cgwalters @travier