Merge branch 'main' into AddFileWriterOptions

Author: SaintBacchus

Cargo.lock

@@ -3,13 +3,13 @@
 
 <img width="257" alt="Lance Logo" src="https://user-images.githubusercontent.com/917119/199353423-d3e202f7-0269-411d-8ff2-e747e419e492.png">
 
-**Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, a vector index, data versioning, and more.<br/>**
-**Compatible with pandas, DuckDB, Polars, and pyarrow with more integrations on the way.**
+**Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, zero-cost schema evolution, rich secondary indices, versioning, and more.<br/>**
+**Compatible with Pandas, DuckDB, Polars, Pyarrow, and Ray with more integrations on the way.**
 
 <a href="https://lancedb.github.io/lance/">Documentation</a> •
 <a href="https://blog.lancedb.com/">Blog</a> •
 <a href="https://discord.gg/zMM32dvNtd">Discord</a> •
-<a href="https://twitter.com/lancedb">Twitter</a>
+<a href="https://x.com/lancedb">X</a>
 
 [CI]: https://github.com/lancedb/lance/actions/workflows/rust.yml
 [CI Badge]: https://github.com/lancedb/lance/actions/workflows/rust.yml/badge.svg
@@ -44,7 +44,7 @@ The key features of Lance include:
 
 * **Zero-copy, automatic versioning:** manage versions of your data without needing extra infrastructure.
 
-* **Ecosystem integrations:** Apache Arrow, Pandas, Polars, DuckDB and more on the way.
+* **Ecosystem integrations:** Apache Arrow, Pandas, Polars, DuckDB, Ray, Spark and more on the way.
 
 > [!TIP]
 > Lance is in active development and we welcome contributions. Please see our [contributing guide](docs/contributing.rst) for more information.
@@ -66,7 +66,7 @@ pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ pylance
 > [!TIP]
 > Preview releases are released more often than full releases and contain the
 > latest features and bug fixes. They receive the same level of testing as full releases.
-> We guarantee they will remain published and available for download for at 
+> We guarantee they will remain published and available for download for at
 > least 6 months. When you want to pin to a specific version, prefer a stable release.
 
 **Converting to Lance**
@@ -186,8 +186,8 @@ Support both CPUs (``x86_64`` and ``arm``) and GPU (``Nvidia (cuda)`` and ``Appl
 
 **Fast updates** (ROADMAP): Updates will be supported via write-ahead logs.
 
-**Rich secondary indices** (ROADMAP):
-  - Inverted index for fuzzy search over many label / annotation fields.
+**Rich secondary indices**: Support `BTree`, `Bitmap`, `Full text search`, `Label list`,
+`NGrams`, and more.
 
 ## Benchmarks
 
@@ -253,11 +253,16 @@ A comparison of different data formats in each stage of ML development cycle.
 
 Lance is currently used in production by:
 * [LanceDB](https://github.com/lancedb/lancedb), a serverless, low-latency vector database for ML applications
+* [LanceDB Enterprise](https://docs.lancedb.com/enterprise/introduction), hyperscale LanceDB with enterprise SLA.
+* Leading multimodal Gen AI companies for training over petabyte-scale multimodal data.
 * Self-driving car company for large-scale storage, retrieval and processing of multi-modal data.
 * E-commerce company for billion-scale+ vector personalized search.
 * and more.
 
-## Presentations and Talks
+## Presentations, Blogs and Talks
 
+* [Designing a Table Format for ML Workloads](https://blog.lancedb.com/designing-a-table-format-for-ml-workloads/), Feb 2025.
+* [Transforming Multimodal Data Management with LanceDB, Ray Summit](https://www.youtube.com/watch?v=xmTFEzAh8ho), Oct 2024.
+* [Lance v2: A columnar container format for modern data](https://blog.lancedb.com/lance-v2/), Apr 2024.
 * [Lance Deep Dive](https://drive.google.com/file/d/1Orh9rK0Mpj9zN_gnQF1eJJFpAc6lStGm/view?usp=drive_link). July 2023.
 * [Lance: A New Columnar Data Format](https://docs.google.com/presentation/d/1a4nAiQAkPDBtOfXFpPg7lbeDAxcNDVKgoUkw3cUs2rE/edit#slide=id.p), [Scipy 2022, Austin, TX](https://www.scipy2022.scipy.org/posters). July, 2022.

README.md

@@ -83,6 +83,7 @@ ignore = [
     { id = "RUSTSEC-2021-0153", reason = "`encoding` is used by lindera" },
     { id = "RUSTSEC-2024-0384", reason = "`instant` is used by tantivy" },
     { id = "RUSTSEC-2024-0436", reason = "`paste` is used by datafusion" },
+    { id = "RUSTSEC-2025-0014", reason = "`humantime` is used by object_store" },
 ]
 # If this is true, then cargo deny will use the git executable to fetch advisory database.
 # If this is false, then it uses a built-in git library.

deny.toml

@@ -2,6 +2,7 @@ APIs
 ----
 
 .. toctree::
+   :maxdepth: 1
 
-  Rust <https://docs.rs/crate/lance/latest>
-  Python <./python.rst>
+   Rust <https://docs.rs/crate/lance/latest>
+   Python <./python.rst>

docs/api/api.rst

@@ -1,18 +1,5 @@
 # Configuration file for the Sphinx documentation builder.
 
-import shutil
-
-
-def run_apidoc(_):
-    from sphinx.ext.apidoc import main
-
-    shutil.rmtree("api/python", ignore_errors=True)
-    main(["-f", "-o", "api/python", "../python/python/lance"])
-
-
-def setup(app):
-    app.connect("builder-inited", run_apidoc)
-
 
 # -- Project information -----------------------------------------------------
 
@@ -29,6 +16,7 @@ def setup(app):
 extensions = [
     "breathe",
     "sphinx_immaterial",
+    "sphinx_immaterial.apidoc.python.apigen",
     "sphinx.ext.autodoc",
     "sphinx.ext.doctest",
     "sphinx.ext.githubpages",
@@ -55,8 +43,22 @@ def setup(app):
     "numpy": ("https://numpy.org/doc/stable/", None),
     "pyarrow": ("https://arrow.apache.org/docs/", None),
     "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+    "ray": ("https://docs.ray.io/en/latest/", None),
 }
 
+python_apigen_modules = {
+    "lance": "api/python/",
+}
+object_description_options = [
+    (
+        "py:.*",
+        dict(
+            include_object_type_in_xref_tooltip=False,
+            include_in_toc=False,
+            include_fields_in_toc=False,
+        ),
+    ),
+]
 
 # -- Options for HTML output -------------------------------------------------
 
@@ -95,7 +97,7 @@ def setup(app):
         },
     ],
 }
-include_in_toc = False
+
 
 # -- doctest configuration ---------------------------------------------------
 

docs/conf.py

@@ -43,14 +43,16 @@ Preview releases receive the same level of testing as regular releases.
    :maxdepth: 2
 
    Quickstart <./notebooks/quickstart>
-   ./read_and_write
+   ./introduction/read_and_write
+   ./introduction/schema_evolution
 
 .. toctree::
    :caption: Advanced Usage
    :maxdepth: 1
 
    Lance Format Spec <./format>
    Blob API <./blob>
+   Object Store Configuration <./object_store>
    Performance Guide <./performance>
    Tokenizer <./tokenizer>
    Extension Arrays <./arrays>

docs/index.rst

@@ -1,27 +1,35 @@
 Lance ❤️ Ray
 --------------------
 
-Ray effortlessly scale up ML workload to large distributed compute environment.
+`Ray <https://www.anyscale.com/product/open-source/ray>`_ effortlessly scale up ML workload to large distributed
+compute environment.
 
-`Ray Data <https://docs.ray.io/en/latest/data/data.html>`_ can be directly written in Lance format by using the
-:class:`lance.ray.sink.LanceDatasink` class. For example:
+Lance format is one of the official `Ray data sources <https://docs.ray.io/en/latest/data/api/input_output.html#lance>`_:
 
-.. code-block:: bash
+* Lance Data Source :py:meth:`ray.data.read_lance`
+* Lance Data Sink :py:meth:`ray.data.Dataste.write_lance`
 
-    pip install pylance[ray]
+.. testsetup::
 
+    shutil.rmtree("./alice_bob_and_charlie.lance", ignore_errors=True)
 
-``Ray Data Dataset`` can be written to Lance format using the following code:
-
-.. code-block:: python
+.. testcode::
 
     import ray
-    from lance.ray.sink import LanceDatasink
 
     ray.init()
 
-    sink = LanceDatasink("s3://bucket/to/data.lance")
-    ray.data.range(10).map(
-        lambda x: {"id": x["id"], "str": f"str-{x['id']}"}
-    ).write_datasink(sink)
+    data = [
+        {"id": 1, "name": "alice"},
+        {"id": 2, "name": "bob"},
+        {"id": 3, "name": "charlie"}
+    ]
+    ray.data.from_items(data).write_lance("./alice_bob_and_charlie.lance")
+
+    # It can be read via lance directly
+    tbl = lance.dataset("./alice_bob_and_charlie.lance").to_table()
+    assert tbl == pa.Table.from_pylist(data)
 
+    # Or via Ray.data.read_lance
+    pd_df = ray.data.read_lance("./alice_bob_and_charlie.lance").to_pandas()
+    assert tbl == pa.Table.from_pandas(pd_df)