Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'attempt to add with overflow' during deflate. #74

Closed
itzurabhi opened this issue Mar 24, 2020 · 5 comments
Closed

'attempt to add with overflow' during deflate. #74

itzurabhi opened this issue Mar 24, 2020 · 5 comments

Comments

@itzurabhi
Copy link

itzurabhi commented Mar 24, 2020 •
edited
Loading

Found as part of the Rust compiler build.
miniz version - miniz_oxide-0.3.5
Rust version tag : release v1.41.1
Channel : nightly
config.toml

[llvm]
ninja = true
link-jobs = 1
version-suffix = "-rust"
release-debuginfo = true
use-libcxx = true
use-linker = "lld"

[rust]
optimize = true
debug = true
debuginfo-level = 2
debuginfo-level-rustc = 2
debuginfo-level-std = 2
debuginfo-level-tools = 2
channel = "nightly"

[build]
extended = true
verbose = 1

during x.py install

Dist rustc stage2 (x86_64-unknown-linux-gnu)
running: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
thread '<unnamed>' panicked at 'attempt to add with overflow', /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:1669:31
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1028
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:188
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:205
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:464
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:373
  12: rust_begin_unwind
             at src/libstd/panicking.rs:302
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:139
  14: core::panicking::panic
             at src/libcore/panicking.rs:70
  15: miniz_oxide::deflate::core::compress_inner
           at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:0
  16: miniz_oxide::deflate::core::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/core.rs:2070
  17: miniz_oxide::deflate::stream::deflate
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.3.5/src/deflate/stream.rs:52
  18: <flate2::ffi::rust::Deflate as flate2::ffi::DeflateBackend>::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/ffi/rust.rs:149
  19: flate2::mem::Compress::compress
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:297
  20: flate2::mem::Compress::compress_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/mem.rs:322
  21: <flate2::mem::Compress as flate2::zio::Ops>::run_vec
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:54
  22: flate2::zio::Writer<W,D>::write_with_status
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:217
  23: <flate2::zio::Writer<W,D> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/zio.rs:256
  24: <flate2::gz::write::GzEncoder<W> as std::io::Write>::write
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.12/src/gz/write.rs:149
  25: std::io::Write::write_all
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/io/mod.rs:1341
  26: <installer::tarballer::RayonTee<A,B> as std::io::Write>::write_all::{{closure}}
             at src/tools/rust-installer/src/tarballer.rs:175
  27: rayon_core::join::join::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:103
  28: rayon_core::join::join_context::call_b::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/join/mod.rs:130
  29: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::call::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:113
  30: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
  31: std::panicking::try::do_call
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
  32: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  33: std::panicking::try
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
  34: std::panic::catch_unwind
             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
  35: rayon_core::unwind::halt_unwinding
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/unwind.rs:17
  36: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:119
  37: rayon_core::job::JobRef::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/job.rs:59
  38: rayon_core::registry::WorkerThread::execute
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:735
  39: rayon_core::registry::WorkerThread::wait_until_cold
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:719
  40: rayon_core::registry::WorkerThread::wait_until
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:693
  41: rayon_core::registry::main_loop
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:813
  42: rayon_core::registry::ThreadBuilder::run
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:58
  43: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
             at /volume/compiler-images/release/build/rust-build/v1411/cargo_home/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.6.0/src/registry.rs:103


command did not execute successfully: "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/x86_64-unknown-linux-gnu/stage0-tools-bin/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=Rust-is-ready-to-roll." "--image-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-image" "--work-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist" "--output-dir" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/dist" "--non-installed-overlay" "/volume/compiler-images/release/build/rust-build/v1411/build/20200310/Linux/Ubuntu-12.04/x86_64/build/tmp/dist/rustc-nightly-x86_64-unknown-linux-gnu-overlay" "--package-name=rustc-nightly-x86_64-unknown-linux-gnu" "--component-name=rustc" "--legacy-manifest-dirs=rustlib,cargo"
expected success, got: signal: 4

solana-labs/solana#5411 looks similar.
@oyvindln
Copy link
Collaborator

oyvindln commented Mar 24, 2020 •
edited
Loading

Yeah seems similar. The backtrace points to line 0 in core.rs for some reason, so I will try to reproduce to find which variable overflows.

oyvindln added a commit that referenced this issue Apr 23, 2020
@oyvindln
fix(deflate): overflow panic with very large input buffer
f0b0e8f 
Attempt to fix overflow panic when calling compress with a >4GB large slice. (#74)
@oyvindln
Copy link
Collaborator

Sorry for taking a while to try to solve this. Are you able to test if it works on the latest commit?

I've changed the buffers to use usize for indexing, and it fixed the overflow when testing with a huge file for me, so hopefully it solves this too.

@itzurabhi
Copy link
Author

Thanks for fixing it.

I didn't test the library standalone,I got the error while building rustc itself (package step) .

I'll try and update here.

@oyvindln oyvindln mentioned this issue Apr 28, 2020
Closed
@dylanmckay
Copy link

dylanmckay commented May 20, 2020 •
edited
Loading

I'm also hitting this whilst compiling rustc - is it possible to publish the 0.3.7 release to crates.io?

I'd like to submit a PR to flate2, then subsequently to rustc, fixing the overflow errors during rustc x.py install.

@oyvindln
Copy link
Collaborator

Ah, thought I had put it up, seems I put up the c api part instead.

tblah added a commit to tblah/rust that referenced this issue Jun 10, 2020
@tblah
Bump miniz oxide
844a2c7 
cargo update -p miniz_oxide

0.3.7 includes a fix for Frommi/miniz_oxide#74
which solves a panic while generating rust distribution tarballs.
@tblah tblah mentioned this issue Jun 10, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@itzurabhi @dylanmckay @oyvindln