Merge branch 'main' of https://github.com/lancedb/lance into add-avg-loss

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Author: BubbleCal

Cargo.lock

@@ -3,13 +3,13 @@
 
 <img width="257" alt="Lance Logo" src="https://user-images.githubusercontent.com/917119/199353423-d3e202f7-0269-411d-8ff2-e747e419e492.png">
 
-**Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, a vector index, data versioning, and more.<br/>**
-**Compatible with pandas, DuckDB, Polars, and pyarrow with more integrations on the way.**
+**Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, zero-cost schema evolution, rich secondary indices, versioning, and more.<br/>**
+**Compatible with Pandas, DuckDB, Polars, Pyarrow, and Ray with more integrations on the way.**
 
 <a href="https://lancedb.github.io/lance/">Documentation</a> •
 <a href="https://blog.lancedb.com/">Blog</a> •
 <a href="https://discord.gg/zMM32dvNtd">Discord</a> •
-<a href="https://twitter.com/lancedb">Twitter</a>
+<a href="https://x.com/lancedb">X</a>
 
 [CI]: https://github.com/lancedb/lance/actions/workflows/rust.yml
 [CI Badge]: https://github.com/lancedb/lance/actions/workflows/rust.yml/badge.svg
@@ -44,7 +44,7 @@ The key features of Lance include:
 
 * **Zero-copy, automatic versioning:** manage versions of your data without needing extra infrastructure.
 
-* **Ecosystem integrations:** Apache Arrow, Pandas, Polars, DuckDB and more on the way.
+* **Ecosystem integrations:** Apache Arrow, Pandas, Polars, DuckDB, Ray, Spark and more on the way.
 
 > [!TIP]
 > Lance is in active development and we welcome contributions. Please see our [contributing guide](docs/contributing.rst) for more information.
@@ -66,7 +66,7 @@ pip install --pre --extra-index-url https://pypi.fury.io/lancedb/ pylance
 > [!TIP]
 > Preview releases are released more often than full releases and contain the
 > latest features and bug fixes. They receive the same level of testing as full releases.
-> We guarantee they will remain published and available for download for at 
+> We guarantee they will remain published and available for download for at
 > least 6 months. When you want to pin to a specific version, prefer a stable release.
 
 **Converting to Lance**
@@ -186,8 +186,8 @@ Support both CPUs (``x86_64`` and ``arm``) and GPU (``Nvidia (cuda)`` and ``Appl
 
 **Fast updates** (ROADMAP): Updates will be supported via write-ahead logs.
 
-**Rich secondary indices** (ROADMAP):
-  - Inverted index for fuzzy search over many label / annotation fields.
+**Rich secondary indices**: Support `BTree`, `Bitmap`, `Full text search`, `Label list`,
+`NGrams`, and more.
 
 ## Benchmarks
 
@@ -253,11 +253,16 @@ A comparison of different data formats in each stage of ML development cycle.
 
 Lance is currently used in production by:
 * [LanceDB](https://github.com/lancedb/lancedb), a serverless, low-latency vector database for ML applications
+* [LanceDB Enterprise](https://docs.lancedb.com/enterprise/introduction), hyperscale LanceDB with enterprise SLA.
+* Leading multimodal Gen AI companies for training over petabyte-scale multimodal data.
 * Self-driving car company for large-scale storage, retrieval and processing of multi-modal data.
 * E-commerce company for billion-scale+ vector personalized search.
 * and more.
 
-## Presentations and Talks
+## Presentations, Blogs and Talks
 
+* [Designing a Table Format for ML Workloads](https://blog.lancedb.com/designing-a-table-format-for-ml-workloads/), Feb 2025.
+* [Transforming Multimodal Data Management with LanceDB, Ray Summit](https://www.youtube.com/watch?v=xmTFEzAh8ho), Oct 2024.
+* [Lance v2: A columnar container format for modern data](https://blog.lancedb.com/lance-v2/), Apr 2024.
 * [Lance Deep Dive](https://drive.google.com/file/d/1Orh9rK0Mpj9zN_gnQF1eJJFpAc6lStGm/view?usp=drive_link). July 2023.
 * [Lance: A New Columnar Data Format](https://docs.google.com/presentation/d/1a4nAiQAkPDBtOfXFpPg7lbeDAxcNDVKgoUkw3cUs2rE/edit#slide=id.p), [Scipy 2022, Austin, TX](https://www.scipy2022.scipy.org/posters). July, 2022.

README.md

@@ -83,6 +83,7 @@ ignore = [
     { id = "RUSTSEC-2021-0153", reason = "`encoding` is used by lindera" },
     { id = "RUSTSEC-2024-0384", reason = "`instant` is used by tantivy" },
     { id = "RUSTSEC-2024-0436", reason = "`paste` is used by datafusion" },
+    { id = "RUSTSEC-2025-0014", reason = "`humantime` is used by object_store" },
 ]
 # If this is true, then cargo deny will use the git executable to fetch advisory database.
 # If this is false, then it uses a built-in git library.

deny.toml

@@ -2,6 +2,7 @@ APIs
 ----
 
 .. toctree::
+   :maxdepth: 1
 
-  Rust <https://docs.rs/crate/lance/latest>
-  Python <./python.rst>
+   Rust <https://docs.rs/crate/lance/latest>
+   Python <./python.rst>

docs/api/api.rst

@@ -14,7 +14,7 @@ a 32-bit float: ~1e-38 to ~1e38. By comparison, a 16-bit float has a range of
 ~5.96e-8 to 65504.
 
 Lance provides an Arrow extension array (:class:`lance.arrow.BFloat16Array`)
-and a Pandas extension array (:class:`lance.pandas.BFloat16Dtype`) for BFloat16.
+and a Pandas extension array (:class:`~lance._arrow.PandasBFloat16Type`) for BFloat16.
 These are compatible with the `ml_dtypes <https://github.com/jax-ml/ml_dtypes>`_
 bfloat16 NumPy extension array.
 
@@ -31,7 +31,7 @@ the array:
     2      3.40625
     dtype: lance.bfloat16
 
-To create an an arrow array, use the :func:`lance.arrow.bfloat16_array` function:
+To create an Arrow array, use the :func:`lance.arrow.bfloat16_array` function:
 
 .. code-block:: python
 

docs/arrays.rst

@@ -10,11 +10,6 @@ Lance provides a high-level API to store and retrieve large binary objects (blob
 Lance serves large binary data using :py:class:`lance.BlobFile`, which
 is a file-like object that lazily reads large binary objects.
 
-.. autoclass:: lance.BlobFile
-    :members:
-    :show-inheritance:
-    :noindex:
-
 To fetch blobs from a Lance dataset, you can use :py:meth:`lance.dataset.LanceDataset.take_blobs`.
 
 For example, it's easy to use `BlobFile` to extract frames from a video file without

docs/blob.rst

@@ -1,24 +1,10 @@
 # Configuration file for the Sphinx documentation builder.
 
-import shutil
-from datetime import datetime
-
-
-def run_apidoc(_):
-    from sphinx.ext.apidoc import main
-
-    shutil.rmtree("api/python", ignore_errors=True)
-    main(["-f", "-o", "api/python", "../python/python/lance"])
-
-
-def setup(app):
-    app.connect("builder-inited", run_apidoc)
-
 
 # -- Project information -----------------------------------------------------
 
 project = "Lance"
-copyright = f"{datetime.today().year}, Lance Developer"
+copyright = "%Y, Lance Developer"
 author = "Lance Developer"
 
 
@@ -29,7 +15,8 @@ def setup(app):
 # ones.
 extensions = [
     "breathe",
-    "sphinx_copybutton",
+    "sphinx_immaterial",
+    "sphinx_immaterial.apidoc.python.apigen",
     "sphinx.ext.autodoc",
     "sphinx.ext.doctest",
     "sphinx.ext.githubpages",
@@ -56,25 +43,61 @@ def setup(app):
     "numpy": ("https://numpy.org/doc/stable/", None),
     "pyarrow": ("https://arrow.apache.org/docs/", None),
     "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None),
+    "ray": ("https://docs.ray.io/en/latest/", None),
 }
 
+python_apigen_modules = {
+    "lance": "api/python/",
+}
+object_description_options = [
+    (
+        "py:.*",
+        dict(
+            include_object_type_in_xref_tooltip=False,
+            include_in_toc=False,
+            include_fields_in_toc=False,
+        ),
+    ),
+]
 
 # -- Options for HTML output -------------------------------------------------
 
-html_theme = "piccolo_theme"
+html_theme = "sphinx_immaterial"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
 html_favicon = "_static/favicon_64x64.png"
-# html_logo = "_static/high-res-icon.png"
+html_logo = "_static/high-res-icon.png"
 html_theme_options = {
-    "source_url": "https://github.com/lancedb/lance",
-    "source_icon": "github",
+    "icon": {
+        "repo": "fontawesome/brands/github",
+        "edit": "material/file-edit-outline",
+    },
+    "site_url": "https://github.com/lancedb/lance",
+    "repo_url": "https://github.com/lancedb/lance",
+    "repo_name": "Lance",
+    "features": [
+        "navigation.expand",
+        # "navigation.tabs",
+        "content.tabs.link",
+        "content.code.copy",
+    ],
+    "social": [
+        {
+            "icon": "fontawesome/brands/github",
+            "link": "https://github.com/jbms/sphinx-immaterial",
+            "name": "Source on github.com",
+        },
+        {
+            "icon": "fontawesome/brands/python",
+            "link": "https://pypi.org/project/pylance/",
+        },
+    ],
 }
-html_css_files = ["custom.css"]
+
 
 # -- doctest configuration ---------------------------------------------------
 

docs/conf.py

@@ -39,15 +39,35 @@ Preview releases receive the same level of testing as regular releases.
 
 
 .. toctree::
-   :maxdepth: 1
+   :caption: Introduction
+   :maxdepth: 2
 
    Quickstart <./notebooks/quickstart>
-   ./read_and_write
-   Lance Formats <./format>
-   Arrays <./arrays>
+   ./introduction/read_and_write
+   ./introduction/schema_evolution
+
+.. toctree::
+   :caption: Advanced Usage
+   :maxdepth: 1
+
+   Lance Format Spec <./format>
    Blob API <./blob>
-   Integrations <./integrations/integrations>
+   Object Store Configuration <./object_store>
    Performance Guide <./performance>
+   Tokenizer <./tokenizer>
+   Extension Arrays <./arrays>
+
+.. toctree::
+   :caption: Integrations
+
+   Huggingface <./integrations/huggingface>
+   Tensorflow <./integrations/tensorflow>
+   PyTorch <./integrations/pytorch>
+   Ray <./integrations/ray>
+
+.. toctree::
+   :maxdepth: 1
+
    API References <./api/api>
    Contributor Guide <./contributing>
    Examples <./examples/examples>