Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serialization error: batch is not valid: Invalid: Buffer #1 too small #11729

Closed
fexolm opened this issue Nov 19, 2024 · 4 comments
Closed

Serialization error: batch is not valid: Invalid: Buffer #1 too small #11729

fexolm opened this issue Nov 19, 2024 · 4 comments
Assignees
@fexolm

Comments

@fexolm
Copy link
Contributor

fexolm commented Nov 19, 2024 •
edited by zverevgeny
Loading

version: stable-24-3-13
database: olap-click-perf

Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: VERIFY failed (2024-11-19T02:16:28.641775+0300): tablet_id=72075186767817822;parent_id=[50009:7438762186202536004:445];column=_yql_write_id: uint64;id=4294967042;verification=status.ok();fline=validation.cpp:7;problem=Serialization error: batch is not valid: Invalid: Buffer #1 too small in array of type uint64 and length 13358: expected at least 106864 byte(s), got 1670;
Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]:   contrib/ydb/library/actors/core/log.cpp:745
Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]:   ~TVerifyFormattedRecordWriter(): requirement false failed
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 0. /-S/util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0x55FD628F132B
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 1. /-S/contrib/ydb/library/actors/core/log.cpp:745: NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() @ 0x55FD62E9E167
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 2. /-S/contrib/ydb/library/formats/arrow/validation/validation.cpp:7: NKikimr::NArrow::TStatusValidator::Validate(arrow::Status const&) @ 0x55FD66BB3C88
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 3. /-S/contrib/ydb/library/formats/arrow/validation/validation.h:21: std::__y1::shared_ptr<arrow::RecordBatch> NKikimr::NArrow::TStatusValidator::GetValid<std::__y1::shared_ptr<arrow::RecordBatch>>(arrow::Result<std::__y1::shared_ptr<arrow::RecordBatch>>&&) @ 0x55FD66CA41BB
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 4. /-S/contrib/ydb/core/formats/arrow/save_load/loader.cpp:55: NKikimr::NArrow::NAccessor::TColumnLoader::ApplyVerified(TBasicString<char, std::__y1::char_traits<char>> const&, unsigned int) const @ 0x55FD66CA41BB
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 5. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:728: NKikimr::NOlap::TPortionInfo::TAssembleBlobInfo::BuildRecordBatch(NKikimr::NArrow::NAccessor::TColumnLoader const&) const @ 0x55FD6E3AEC2D
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 6. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:684: NKikimr::NOlap::TPortionInfo::TPreparedColumn::AssembleAccessor() const @ 0x55FD6E3AE4EA
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 7. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:741: NKikimr::NOlap::TPortionInfo::TPreparedBatchData::AssembleToGeneralContainer(std::__y1::set<unsigned int, std::__y1::less<unsigned int>, std::__y1::allocator<unsigned int>> const&) const @ 0x55FD6E3B02F0
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 8. /-S/contrib/ydb/core/tx/columnshard/engines/portions/read_with_blobs.cpp:23: NKikimr::NOlap::TReadPortionInfoWithBlobs::RestoreBatch(NKikimr::NOlap::ISnapshotSchema const&, NKikimr::NOlap::ISnapshotSchema const&, std::__y1::set<unsigned int, std::__y1::less<unsigned int>, std::__y1::allocator<unsigned int>> const&) const @ 0x55FD6E3FF3A3
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 9. /-S/contrib/ydb/core/tx/columnshard/engines/changes/general_compaction.cpp:132: NKikimr::NOlap::NCompaction::TGeneralCompactColumnEngineChanges::BuildAppendedPortionsByChunks(NKikimr::NOlap::TConstructionContext&, std::__y1::vector<NKikimr::NOlap::TReadPortionInfoWithBlobs, std::__y1::allocator<NKikimr::NOlap::TReadPortionInfoWithBlobs>>&&) @ 0x55FD6E3F6A3F
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 10. /-S/contrib/ydb/core/tx/columnshard/engines/changes/general_compaction.cpp:173: NKikimr::NOlap::NCompaction::TGeneralCompactColumnEngineChanges::DoConstructBlobs(NKikimr::NOlap::TConstructionContext&) @ 0x55FD6E3F8459
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 11. /-S/contrib/ydb/core/tx/columnshard/engines/changes/abstract/abstract.cpp:23: NKikimr::NOlap::TColumnEngineChanges::ConstructBlobs(NKikimr::NOlap::TConstructionContext&) @ 0x55FD6E3E96C1
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 12. /-S/contrib/ydb/core/tx/columnshard/columnshard_impl.cpp:534: NKikimr::NColumnShard::TChangesTask::DoExecute(std::__y1::shared_ptr<NKikimr::NConveyor::ITask> const&) @ 0x55FD6E704C18
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 13. /-S/contrib/ydb/core/tx/conveyor/usage/abstract.cpp:13: NKikimr::NConveyor::ITask::Execute(std::__y1::shared_ptr<NKikimr::NConveyor::TTaskSignals>, std::__y1::shared_ptr<NKikimr::NConveyor::ITask> const&) @ 0x55FD6E37805E
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 14. /-S/contrib/ydb/core/tx/conveyor/service/worker.cpp:10: NKikimr::NConveyor::TWorker::ExecuteTask(NKikimr::NConveyor::TWorkerTask const&) @ 0x55FD75B178F5
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 15. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:251: NActors::TGenericExecutorThread::TProcessingResult NActors::TGenericExecutorThread::Execute<NActors::TMailboxTable::THTSwapMailbox>(NActors::TMailboxTable::THTSwapMailbox*, unsigned int, bool) @ 0x55FD62E7CAF0
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 16. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:440: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*)::$_0::operator()(unsigned int, bool) const @ 0x55FD62E7254A
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 17. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:493: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*) @ 0x55FD62E71D9F
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 18. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:524: NActors::TExecutorThread::ThreadProc() @ 0x55FD62E73184
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 19. /-S/util/system/thread.cpp:244: (anonymous namespace)::TPosixThread::ThreadProxy(void*) @ 0x55FD628FF509
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 20. ??:0: ?? @ 0x7F882CA3F608
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 21. ??:0: ?? @ 0x7F882C95F132
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Key file not found!
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Determined node ID: 0
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Trying to register dynamic node to ydb-ru-prestable-klg-0028.search.yandex.net:2135
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Success. Registered via discovery service as 50009
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Node name:
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Trying to get configs from ydb-ru-prestable-vla-0009.search.yandex.net:2135
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Success.
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: configured
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Starting Kikimr r14302450 built by sandbox
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: UDFsDir is not specified, no dynamic UDFs will be loaded.
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Unknown grpc service "experimental" was not enabled!
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Unknown grpc service "experimental" was not enabled!
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: GRpc memory quota was set but disabled due to issues with grpc quoter, to enable it use EnableGRpcMemoryQuota option
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: GRpc memory quota was set but disabled due to issues with grpc quoter, to enable it use EnableGRpcMemoryQuota option
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: VERIFY failed (2024-11-19T02:16:50.672990+0300): tablet_id=72075186767817840;parent_id=[50009:7438762278765279122:390];column=_yql_write_id: uint64;id=4294967042;verification=status.ok();fline=validation.cpp:7;problem=Serialization error: batch is not valid: Invalid: Buffer #1 too small in array of type uint64 and length 13405: expected at least 107240 byte(s), got 1676;
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]:   contrib/ydb/library/actors/core/log.cpp:745
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]:   ~TVerifyFormattedRecordWriter(): requirement false failed
@fexolm fexolm self-assigned this Nov 19, 2024
@zverevgeny
Copy link
Collaborator

Вот тут собраны все рестарты
https://paste.yandex-team.ru/727adf3a-3597-49c3-a14c-39b620908aa7/text
Но потом они внезапно прекратились до того, как мы смогли снять нужную диагностику
Закрываю до воспроизведения

@zverevgeny zverevgeny reopened this Nov 19, 2024
@zverevgeny zverevgeny closed this as not planned Won't fix, can't repro, duplicate, stale Nov 19, 2024
@zverevgeny zverevgeny reopened this Nov 21, 2024
@zverevgeny
Copy link
Collaborator

Repro on updating https://nda.ya.ru/t/IstUuDv679omE5 from 24-3-12? to 24-3-13:
https://paste.yandex-team.ru/991650c9-f68f-46b4-89be-d7b9601a72d2

@zverevgeny
Copy link
Collaborator

Недоимплмент модифицируюищих транзакций. Сейчас эта функциональность отключена фиче-флагом по-умолчанию. Воспроизведение есть только на базах, где эта фича была включена

@fexolm
Copy link
Contributor Author

fexolm commented Dec 6, 2024

Фиксится нормалайзером #9679

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fexolm fexolm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fexolm @zverevgeny