feat: support _rowid meta column for spark connector in java #3194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def8623 to
f62523a
Compare
wjones127
requested changes
Dec 4, 2024
java/spark/src/main/java/org/apache/spark/sql/vectorized/LanceArrowColumnVector.java
Show resolved
Hide resolved
java/spark/src/test/java/com/lancedb/lance/spark/TestUtils.java
Outdated
Show resolved
Hide resolved
java/spark/src/test/scala/org/apache/spark/sql/vectorized/LanceArrowColumnVectorSuite.scala
Show resolved
Hide resolved
b329f52 to
d324d62
Compare
|
@wjones127 I had added the license file in lancedb. Plz review it |
wjones127
requested changes
Dec 5, 2024
LICENSE
Outdated
Comment on lines
438
to
437
| ------------------------------------------------------------------------------------ | ||
| This product bundles various third-party components under other open source licenses. | ||
| This section summarizes those components and their licenses. See licenses/ | ||
| for text of these licenses. | ||
|
|
||
|
|
||
| Apache Software Foundation License 2.0 | ||
| -------------------------------------- | ||
|
|
||
| common/network-common/src/main/java/org/apache/spark/network/util/LimitedInputStream.java | ||
| core/src/main/java/org/apache/spark/util/collection/TimSort.java | ||
| core/src/main/resources/org/apache/spark/ui/static/bootstrap* | ||
| core/src/main/resources/org/apache/spark/ui/static/vis* | ||
| docs/js/vendor/bootstrap.js | ||
| connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java | ||
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css | ||
|
|
||
| Python Software Foundation License | ||
| ---------------------------------- | ||
|
|
||
| python/pyspark/loose_version.py | ||
|
|
||
| BSD 3-Clause | ||
| ------------ | ||
|
|
||
| python/lib/py4j-*-src.zip | ||
| python/pyspark/cloudpickle/*.py | ||
| python/pyspark/join.py | ||
|
|
||
| The CSS style for the navigation sidebar of the documentation was originally | ||
| submitted by Óscar Nájera for the scikit-learn project. The scikit-learn project | ||
| is distributed under the 3-Clause BSD license. | ||
|
|
||
|
|
||
| MIT License | ||
| ----------- | ||
|
|
||
| core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/*dataTables* | ||
| core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/jquery* | ||
| core/src/main/resources/org/apache/spark/ui/static/sorttable.js | ||
| docs/js/vendor/anchor.min.js | ||
| docs/js/vendor/jquery* | ||
| docs/js/vendor/modernizer* | ||
|
|
||
| ISC License | ||
| ----------- | ||
|
|
||
| core/src/main/resources/org/apache/spark/ui/static/d3.min.js | ||
|
|
||
|
|
||
| Creative Commons CC0 1.0 Universal Public Domain Dedication | ||
| ----------------------------------------------------------- | ||
| (see LICENSE-CC0.txt) | ||
|
|
||
| data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg | ||
| data/mllib/images/kittens/54893.jpg | ||
| data/mllib/images/kittens/DP153539.jpg | ||
| data/mllib/images/kittens/DP802813.jpg | ||
| data/mllib/images/multi-channel/chr30.4.184.jpg | ||
|
|
||
| https://github.com/apache/spark/blob/master/LICENSE |
There was a problem hiding this comment.
I think you can remove all of this, since none of this applies to the files you brought over.
Suggested change
| ------------------------------------------------------------------------------------ | |
| This product bundles various third-party components under other open source licenses. | |
| This section summarizes those components and their licenses. See licenses/ | |
| for text of these licenses. | |
| Apache Software Foundation License 2.0 | |
| -------------------------------------- | |
| common/network-common/src/main/java/org/apache/spark/network/util/LimitedInputStream.java | |
| core/src/main/java/org/apache/spark/util/collection/TimSort.java | |
| core/src/main/resources/org/apache/spark/ui/static/bootstrap* | |
| core/src/main/resources/org/apache/spark/ui/static/vis* | |
| docs/js/vendor/bootstrap.js | |
| connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java | |
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css | |
| Python Software Foundation License | |
| ---------------------------------- | |
| python/pyspark/loose_version.py | |
| BSD 3-Clause | |
| ------------ | |
| python/lib/py4j-*-src.zip | |
| python/pyspark/cloudpickle/*.py | |
| python/pyspark/join.py | |
| The CSS style for the navigation sidebar of the documentation was originally | |
| submitted by Óscar Nájera for the scikit-learn project. The scikit-learn project | |
| is distributed under the 3-Clause BSD license. | |
| MIT License | |
| ----------- | |
| core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/*dataTables* | |
| core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/jquery* | |
| core/src/main/resources/org/apache/spark/ui/static/sorttable.js | |
| docs/js/vendor/anchor.min.js | |
| docs/js/vendor/jquery* | |
| docs/js/vendor/modernizer* | |
| ISC License | |
| ----------- | |
| core/src/main/resources/org/apache/spark/ui/static/d3.min.js | |
| Creative Commons CC0 1.0 Universal Public Domain Dedication | |
| ----------------------------------------------------------- | |
| (see LICENSE-CC0.txt) | |
| data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg | |
| data/mllib/images/kittens/54893.jpg | |
| data/mllib/images/kittens/DP153539.jpg | |
| data/mllib/images/kittens/DP802813.jpg | |
| data/mllib/images/multi-channel/chr30.4.184.jpg | |
| https://github.com/apache/spark/blob/master/LICENSE |
There was a problem hiding this comment.
OK, I had removed it
Comment on lines
25
to
36
| /** | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| SPDX-FileCopyrightText: Copyright The Lance Authors | ||
|
|
||
| The following code is originally from https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/util/ArrowUtilsSuite.scala | ||
| and is licensed under the Apache license: | ||
|
|
||
| License: Apache License 2.0, Copyright 2014 and onwards The Apache Software Foundation. | ||
| https://github.com/apache/spark/blob/master/LICENSE | ||
|
|
||
| It has been modified by the Lance developers to fit the needs of the Lance project. | ||
| */ |
There was a problem hiding this comment.
Could you move this up? It's confusing with the other license above.
There was a problem hiding this comment.
I just put it in the apache license header.
Comment on lines
25
to
36
| /** | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| SPDX-FileCopyrightText: Copyright The Lance Authors | ||
|
|
||
| The following code is originally from https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala | ||
| and is licensed under the Apache license: | ||
|
|
||
| License: Apache License 2.0, Copyright 2014 and onwards The Apache Software Foundation. | ||
| https://github.com/apache/spark/blob/master/LICENSE | ||
|
|
||
| It has been modified by the Lance developers to fit the needs of the Lance project. | ||
| */ |
df3ca8e to
bbf165f
Compare
wjones127
approved these changes
Dec 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussion in PR, I had implement the _rowid meta column just in java package.