ARROW-105: Fix BaseAllocator.java NPE when assertions are disabled #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When verifying memory using verifyAllocator() method, BaseAllocator throws NPE if assertions are disabled. Fixing this issue by checking first if assertion are disabled
cb343a9 to
6d2f61c
Compare
wesm
added a commit
to wesm/arrow
that referenced
this pull request
Sep 2, 2018
I incorporated quite a bit of code from Impala for this patch, but did a bunch of work to get everything working. In particular, I wasn't happy with the hash table implementation in `dict-encoder.h` and so have written a simple new one that we can benchmark and tune as necessary. The simplest way to pull in the DictEncoder (PARQUET-493) was to also bring in the `MemPool` implementation, suitably trimmed down. We can continue to refactor this as needed for parquet-cpp. I also did some light refactoring using `TYPED_TEST` in `plain-encoding-test` (now `encoding-test`). Author: Wes McKinney <wesm@apache.org> Closes apache#64 from wesm/PARQUET-494 and squashes the following commits: c634abe [Wes McKinney] Refactor to create TestEncoderBase a3a563a [Wes McKinney] Consolidate dictionary encoding code 2cc4ffe [Wes McKinney] Retrieve type_length() only once in PlainDecoder ctor 20ccd9e [Wes McKinney] Remove DictionaryEncoder shim layer for now dcfc0aa [Wes McKinney] Remove redundant Int96 comparison d98a2c0 [Wes McKinney] Dictionary encoding for booleans throws exception 05414f0 [Wes McKinney] Test dictionary encoding more types 9a5b1a4 [Wes McKinney] Enable include_order linting per PARQUET-539 f3f0efc [Wes McKinney] IWYU cleaning d4191c6 [Wes McKinney] Add header installs, fix clang warning 1347b13 [Wes McKinney] Rename plain-encoding-test to encoding-test 09bf0fa [Wes McKinney] Fix bugs and add dictionary repeats 2e6af48 [Wes McKinney] Fix some bugs. FixedLenByteArray remains to get working. 69b5b69 [Wes McKinney] Refactor test fixtures to be less coupled to state. process on getting dict encoding working 6b23716 [Wes McKinney] Create reusable DataType structs for test fixtures and other compile-time type resolution matters 67883fd [Wes McKinney] Bunch of combined work for dict encoding support:
wesm
added a commit
to wesm/arrow
that referenced
this pull request
Sep 4, 2018
I incorporated quite a bit of code from Impala for this patch, but did a bunch of work to get everything working. In particular, I wasn't happy with the hash table implementation in `dict-encoder.h` and so have written a simple new one that we can benchmark and tune as necessary. The simplest way to pull in the DictEncoder (PARQUET-493) was to also bring in the `MemPool` implementation, suitably trimmed down. We can continue to refactor this as needed for parquet-cpp. I also did some light refactoring using `TYPED_TEST` in `plain-encoding-test` (now `encoding-test`). Author: Wes McKinney <wesm@apache.org> Closes apache#64 from wesm/PARQUET-494 and squashes the following commits: c634abe [Wes McKinney] Refactor to create TestEncoderBase a3a563a [Wes McKinney] Consolidate dictionary encoding code 2cc4ffe [Wes McKinney] Retrieve type_length() only once in PlainDecoder ctor 20ccd9e [Wes McKinney] Remove DictionaryEncoder shim layer for now dcfc0aa [Wes McKinney] Remove redundant Int96 comparison d98a2c0 [Wes McKinney] Dictionary encoding for booleans throws exception 05414f0 [Wes McKinney] Test dictionary encoding more types 9a5b1a4 [Wes McKinney] Enable include_order linting per PARQUET-539 f3f0efc [Wes McKinney] IWYU cleaning d4191c6 [Wes McKinney] Add header installs, fix clang warning 1347b13 [Wes McKinney] Rename plain-encoding-test to encoding-test 09bf0fa [Wes McKinney] Fix bugs and add dictionary repeats 2e6af48 [Wes McKinney] Fix some bugs. FixedLenByteArray remains to get working. 69b5b69 [Wes McKinney] Refactor test fixtures to be less coupled to state. process on getting dict encoding working 6b23716 [Wes McKinney] Create reusable DataType structs for test fixtures and other compile-time type resolution matters 67883fd [Wes McKinney] Bunch of combined work for dict encoding support: Change-Id: I0fe7d47373b9da106e700381bee6538199af8a69
wesm
added a commit
to wesm/arrow
that referenced
this pull request
Sep 6, 2018
I incorporated quite a bit of code from Impala for this patch, but did a bunch of work to get everything working. In particular, I wasn't happy with the hash table implementation in `dict-encoder.h` and so have written a simple new one that we can benchmark and tune as necessary. The simplest way to pull in the DictEncoder (PARQUET-493) was to also bring in the `MemPool` implementation, suitably trimmed down. We can continue to refactor this as needed for parquet-cpp. I also did some light refactoring using `TYPED_TEST` in `plain-encoding-test` (now `encoding-test`). Author: Wes McKinney <wesm@apache.org> Closes apache#64 from wesm/PARQUET-494 and squashes the following commits: c634abe [Wes McKinney] Refactor to create TestEncoderBase a3a563a [Wes McKinney] Consolidate dictionary encoding code 2cc4ffe [Wes McKinney] Retrieve type_length() only once in PlainDecoder ctor 20ccd9e [Wes McKinney] Remove DictionaryEncoder shim layer for now dcfc0aa [Wes McKinney] Remove redundant Int96 comparison d98a2c0 [Wes McKinney] Dictionary encoding for booleans throws exception 05414f0 [Wes McKinney] Test dictionary encoding more types 9a5b1a4 [Wes McKinney] Enable include_order linting per PARQUET-539 f3f0efc [Wes McKinney] IWYU cleaning d4191c6 [Wes McKinney] Add header installs, fix clang warning 1347b13 [Wes McKinney] Rename plain-encoding-test to encoding-test 09bf0fa [Wes McKinney] Fix bugs and add dictionary repeats 2e6af48 [Wes McKinney] Fix some bugs. FixedLenByteArray remains to get working. 69b5b69 [Wes McKinney] Refactor test fixtures to be less coupled to state. process on getting dict encoding working 6b23716 [Wes McKinney] Create reusable DataType structs for test fixtures and other compile-time type resolution matters 67883fd [Wes McKinney] Bunch of combined work for dict encoding support: Change-Id: I0fe7d47373b9da106e700381bee6538199af8a69
wesm
added a commit
to wesm/arrow
that referenced
this pull request
Sep 7, 2018
I incorporated quite a bit of code from Impala for this patch, but did a bunch of work to get everything working. In particular, I wasn't happy with the hash table implementation in `dict-encoder.h` and so have written a simple new one that we can benchmark and tune as necessary. The simplest way to pull in the DictEncoder (PARQUET-493) was to also bring in the `MemPool` implementation, suitably trimmed down. We can continue to refactor this as needed for parquet-cpp. I also did some light refactoring using `TYPED_TEST` in `plain-encoding-test` (now `encoding-test`). Author: Wes McKinney <wesm@apache.org> Closes apache#64 from wesm/PARQUET-494 and squashes the following commits: c634abe [Wes McKinney] Refactor to create TestEncoderBase a3a563a [Wes McKinney] Consolidate dictionary encoding code 2cc4ffe [Wes McKinney] Retrieve type_length() only once in PlainDecoder ctor 20ccd9e [Wes McKinney] Remove DictionaryEncoder shim layer for now dcfc0aa [Wes McKinney] Remove redundant Int96 comparison d98a2c0 [Wes McKinney] Dictionary encoding for booleans throws exception 05414f0 [Wes McKinney] Test dictionary encoding more types 9a5b1a4 [Wes McKinney] Enable include_order linting per PARQUET-539 f3f0efc [Wes McKinney] IWYU cleaning d4191c6 [Wes McKinney] Add header installs, fix clang warning 1347b13 [Wes McKinney] Rename plain-encoding-test to encoding-test 09bf0fa [Wes McKinney] Fix bugs and add dictionary repeats 2e6af48 [Wes McKinney] Fix some bugs. FixedLenByteArray remains to get working. 69b5b69 [Wes McKinney] Refactor test fixtures to be less coupled to state. process on getting dict encoding working 6b23716 [Wes McKinney] Create reusable DataType structs for test fixtures and other compile-time type resolution matters 67883fd [Wes McKinney] Bunch of combined work for dict encoding support: Change-Id: I0fe7d47373b9da106e700381bee6538199af8a69
wesm
added a commit
to wesm/arrow
that referenced
this pull request
Sep 8, 2018
I incorporated quite a bit of code from Impala for this patch, but did a bunch of work to get everything working. In particular, I wasn't happy with the hash table implementation in `dict-encoder.h` and so have written a simple new one that we can benchmark and tune as necessary. The simplest way to pull in the DictEncoder (PARQUET-493) was to also bring in the `MemPool` implementation, suitably trimmed down. We can continue to refactor this as needed for parquet-cpp. I also did some light refactoring using `TYPED_TEST` in `plain-encoding-test` (now `encoding-test`). Author: Wes McKinney <wesm@apache.org> Closes apache#64 from wesm/PARQUET-494 and squashes the following commits: c634abe [Wes McKinney] Refactor to create TestEncoderBase a3a563a [Wes McKinney] Consolidate dictionary encoding code 2cc4ffe [Wes McKinney] Retrieve type_length() only once in PlainDecoder ctor 20ccd9e [Wes McKinney] Remove DictionaryEncoder shim layer for now dcfc0aa [Wes McKinney] Remove redundant Int96 comparison d98a2c0 [Wes McKinney] Dictionary encoding for booleans throws exception 05414f0 [Wes McKinney] Test dictionary encoding more types 9a5b1a4 [Wes McKinney] Enable include_order linting per PARQUET-539 f3f0efc [Wes McKinney] IWYU cleaning d4191c6 [Wes McKinney] Add header installs, fix clang warning 1347b13 [Wes McKinney] Rename plain-encoding-test to encoding-test 09bf0fa [Wes McKinney] Fix bugs and add dictionary repeats 2e6af48 [Wes McKinney] Fix some bugs. FixedLenByteArray remains to get working. 69b5b69 [Wes McKinney] Refactor test fixtures to be less coupled to state. process on getting dict encoding working 6b23716 [Wes McKinney] Create reusable DataType structs for test fixtures and other compile-time type resolution matters 67883fd [Wes McKinney] Bunch of combined work for dict encoding support: Change-Id: I0fe7d47373b9da106e700381bee6538199af8a69
xuechendi
pushed a commit
to xuechendi/arrow
that referenced
this pull request
Aug 4, 2020
…round Remove compression workaround due to performance impact
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
This closes apache#64 When verifying memory using verifyAllocator() method, BaseAllocator throws NPE if assertions are disabled. Fixing this issue by checking first if assertion are disabled
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When verifying memory using verifyAllocator() method, BaseAllocator throws NPE
if assertions are disabled.
Fixing this issue by checking first if assertion are disabled