Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more fsspec filesystems #307

Closed
2 of 3 tasks
hombit opened this issue Jul 19, 2024 · 1 comment
Closed
2 of 3 tasks

Support more fsspec filesystems #307

hombit opened this issue Jul 19, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@hombit
Copy link
Contributor

hombit commented Jul 19, 2024

Bug report

We currently support only few filesystems, while community have very different setups for network filesystems, which may include HTTP, FTP, Google Cloud Block Storage, SSHFS, etc. The interface we have do not support any user-defined fsspec filesystems, I propose here to support them.

This is related and complimentary to #301

Before submitting
Please check the following:

  • I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
  • I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
  • If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
@hombit hombit added the bug Something isn't working label Jul 19, 2024
@hombit
Copy link
Contributor Author

hombit commented Jul 22, 2024

This is how you can use google drive filesystem

Install the package, it is not on PyPI (there is a package with the same name there, but it is not that one)

pip install git+https://github.com/fsspec/gdrivefs

Create a fs and read some files

import gdrivefs
import pyarrow

# Use token='browser' first time to create a token and store it locally, then use token='cache' 
gdfs = gdrivefs.GoogleDriveFileSystem(token='cache', root_file_id='1mocyakfy_8OgFGOIQ813S7POqwdDtfX_')
with gdfs.open('Norder=6/Dir=20000/Npix=28672.parquet') as f:
    metadata = pq.read_metadata(f)
print(metadata)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants