Debug assertion when printing nested dataframe containing timestamps

[jhorstmann]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

    let lazy_frame = LazyFrame::scan_parquet("eventlog.parquet", Default::default()).unwrap();
    let mut frame = lazy_frame
        .select([col("events")])
        .limit(1)
        .collect().unwrap();

    dbg!(&frame.schema());
    dbg!(&frame);

Log output

[src/main.rs:10] &frame.schema() = Schema:
name: events, data type: List(Struct([Field { name: "event_name", dtype: Utf8 }, Field { name: "event_time", dtype: Datetime(Milliseconds, Some("+00:00")) }]))

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `Timestamp(Millisecond, Some("+00:00"))`,
 right: `Int64`', /home/i526205/Source/github/pola-rs/polars/crates/polars-core/src/chunked_array/from.rs:206:17

stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:228:5
   4: polars_core::chunked_array::from::<impl polars_core::chunked_array::ChunkedArray<T>>::from_chunks_and_dtype
             at /github/pola-rs/polars/crates/polars-core/src/chunked_array/from.rs:206:17
   5: polars_core::chunked_array::from::<impl polars_core::chunked_array::ChunkedArray<T>>::from_chunks
             at /github/pola-rs/polars/crates/polars-core/src/chunked_array/from.rs:174:9
   6: polars_core::series::from::<impl polars_core::series::Series>::_try_from_arrow_unchecked
             at /github/pola-rs/polars/crates/polars-core/src/series/from.rs:182:40
   7: polars_core::series::from::<impl polars_core::series::Series>::_try_from_arrow_unchecked::{{closure}}
             at /github/pola-rs/polars/crates/polars-core/src/series/from.rs:390:25
   8: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:91:28
   9: core::iter::traits::iterator::Iterator::try_fold
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2304:21
  10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:117:9
  11: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/mod.rs:195:9
  12: core::iter::traits::iterator::Iterator::try_for_each
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2366:9
  13: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/mod.rs:178:14
  14: alloc::vec::Vec<T,A>::extend_desugared
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/mod.rs:2812:35
  15: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_extend.rs:17:9
  16: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_from_iter_nested.rs:43:9
  17: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_from_iter.rs:33:9
  18: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/mod.rs:2712:9
  19: core::iter::traits::iterator::Iterator::collect
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:1896:9
  20: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:1969:51
  21: core::iter::adapters::try_process
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/mod.rs:164:17
  22: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:1969:9
  23: core::iter::traits::iterator::Iterator::collect
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:1896:9
  24: polars_core::series::from::<impl polars_core::series::Series>::_try_from_arrow_unchecked
             at /github/pola-rs/polars/crates/polars-core/src/series/from.rs:385:30
  25: polars_core::series::from::<impl polars_core::series::Series>::from_chunks_and_dtype_unchecked
             at /github/pola-rs/polars/crates/polars-core/src/series/from.rs:102:17
  26: polars_core::chunked_array::ops::any_value::arr_to_any_value
             at /github/pola-rs/polars/crates/polars-core/src/chunked_array/ops/any_value.rs:55:25
  27: polars_core::chunked_array::ops::any_value::<impl polars_core::chunked_array::ops::ChunkAnyValue for polars_core::chunked_array::ChunkedArray<polars_core::datatypes::ListType>>::get_any_value
             at /github/pola-rs/polars/crates/polars-core/src/chunked_array/ops/any_value.rs:255:9
  28: polars_core::series::implementations::list::<impl polars_core::series::series_trait::SeriesTrait for polars_core::series::implementations::SeriesWrap<polars_core::chunked_array::ChunkedArray<polars_core::datatypes::ListType>>>::get
             at /github/pola-rs/polars/crates/polars-core/src/series/implementations/list.rs:163:9
  29: polars_core::series::Series::str_value
             at /github/pola-rs/polars/crates/polars-core/src/series/mod.rs:863:25
  30: polars_core::fmt::<impl core::fmt::Display for polars_core::frame::DataFrame>::fmt::{{closure}}
             at /github/pola-rs/polars/crates/polars-core/src/fmt.rs:583:42
  31: core::iter::adapters::map::map_fold::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:84:28
  32: core::iter::traits::iterator::Iterator::fold
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2482:21
  33: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:124:9
  34: core::iter::traits::iterator::Iterator::for_each
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:857:9
  35: alloc::vec::Vec<T,A>::extend_trusted
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/mod.rs:2844:17
  36: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_extend.rs:26:9
  37: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_from_iter_nested.rs:62:9
  38: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/spec_from_iter.rs:33:9
  39: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/mod.rs:2712:9
  40: core::iter::traits::iterator::Iterator::collect
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:1896:9
  41: polars_core::fmt::<impl core::fmt::Display for polars_core::frame::DataFrame>::fmt
             at /github/pola-rs/polars/crates/polars-core/src/fmt.rs:580:39
  42: polars_core::fmt::<impl core::fmt::Debug for polars_core::frame::DataFrame>::fmt
             at /github/pola-rs/polars/crates/polars-core/src/fmt.rs:334:9
  43: <&T as core::fmt::Debug>::fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:2418:62
  44: <&T as core::fmt::Debug>::fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:2418:62
  45: core::fmt::run
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1302:5
  46: core::fmt::write
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1270:26
  47: std::io::Write::write_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
  48: <&std::io::stdio::Stderr as std::io::Write>::write_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/stdio.rs:945:9
  49: <std::io::stdio::Stderr as std::io::Write>::write_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/stdio.rs:919:9
  50: std::io::stdio::print_to
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/stdio.rs:1018:21
  51: std::io::stdio::_eprint
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/stdio.rs:1106:5
  52: polars_test::main
             at ./src/main.rs:11:5
  53: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Issue description

When printing a dataframe read from a nested parquet file (List -> Record -> Timestamp) the logical data type of the timestamp seems to get lost temporarily, triggering a debug assertion in

polars/crates/polars-core/src/chunked_array/ops/any_value.rs

Line 54 in 05d9eb2

let s = Series::from_chunks_and_dtype_unchecked("", vec![v], &dt.to_physical())

I think the issue is the to_physical call in

polars/crates/polars-core/src/chunked_array/from.rs

Line 206 in 05d9eb2

assert_eq!(chunks[0].data_type(), &dtype.to_physical().to_arrow())

Expected behavior

No panic. Running the sample code in release mode actually passes.

Installed versions

Current main branch 05d9eb2

[dependencies]
polars = { version = "0.33.2", features = ["dtype-full", "lazy", "parquet", "timezones"] }

[patch.crates-io]
polars = { path = "../../../pola-rs/polars/crates/polars"}

[ritchie46]

Thanks for the issue report.

The nested types should be cast to their physical types, that doesn't seem to happen, indicating a bug in our invariants.

[jhorstmann]

The parquet file to reproduce the issue: eventlog.zip

[ritchie46]

Fix coming up.