Create directory specified by prefix for LocalStore if it does not exist

[maxrjones]

It would be useful to provide a LocalStore __init__ parameter that creates the prefix directory if it does not exist. This is particularly important for using obstore with Zarr, since users will often want to create a new store. The current behavior returns a generic error, e.g.,

import os
from obstore.store import LocalStore
cwd = os.getcwd()
m = LocalStore(prefix=f"{cwd}/data/example.zarr")

GenericError: Generic {
    store: "LocalFileSystem",
    source: UnableToCanonicalize {
        path: "/Users/max/Documents/Code/maxrjones/zarr-obstore-performance/data/example.zarr",
        source: Os {
            code: 2,
            kind: NotFound,
            message: "No such file or directory",
        },
    },
}

[kylebarron]

I'm curious whether a put operation will automatically create intermediate directories? I.e.

store = LocalStore("/root/dir")
obs.put(store, path="/dir1/dir2/item.txt")

Hopefully that will automatically create /root/dir/dir1/dir2/.

It would be nice to create the directory specified by prefix, but we need to ensure that we're able to parse the path input in exactly the same way by Python and Rust. Perhaps that means we should coerce to a pathlib.Path, canonicalize it, call Path.mkdir(exist_ok=True, parents=True), and then pass that pre-canonicalized path to object_store::LocalStore.

[kylebarron]

I'm curious whether a put operation will automatically create intermediate directories?

From apache/arrow-rs#7030

• LocalFileSystem automatically creates them on-demand (and optionally deletes them Automatically cleanup empty dirs in LocalFileSystem #5978)