gcsfuse stalls?

[mrocklin]

The following causes both terminals to hang in such a way that they can not be interrupted

$ pip install gcsfs --upgrade
$ mkdir gcs
$ gcsfuse pangeo-data gcs
Mounting bucket pangeo-data to directory gcs

$ ls gcs

[martindurant]

Until #57 is ready, ls on the bucket is fetching the full listing of all keys and their details (98k files), which might take longer than some FUSE timeout.
@asford , are you planning on working on file listing?

[martindurant]

I am adding more logging in #67 and any further improvements. In this case, I don't think it would tell you more than that the directly listing had started.

[asford]

#57 includes optimization of the directory listing codepath, so the ls time should be proportional to the number of files in the directory.

@mrocklin I've rebased that pull against the release revision. Would it be possible for you to quickly test git+https://github.com/asford/gcsfs@per_dir_cache in that install to see if it resolves your issue?

[mrocklin]

Sure. Here is an attempt from that branch

mrocklin@carbon:~/workspace/gcsfs$ git checkout per_dir_cache 
Branch per_dir_cache set up to track remote branch per_dir_cache from asford.
Switched to a new branch 'per_dir_cache'

mrocklin@carbon:~/workspace/gcsfs$ python gcsfs/cli/gcsfuse.py pangeo-data gcs
Mounting bucket pangeo-data to directory gcs
Traceback (most recent call last):
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 693, in readdir
    fip.contents.fh):
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/home/mrocklin/workspace/gcsfs/gcsfs/gcsfuse.py", line 57, in readdir
    files = [f.rstrip('/').rsplit('/', 1)[1] for f in files]
  File "/home/mrocklin/workspace/gcsfs/gcsfs/gcsfuse.py", line 57, in <listcomp>
    files = [f.rstrip('/').rsplit('/', 1)[1] for f in files]
IndexError: list index out of range
Traceback (most recent call last):
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 693, in readdir
    fip.contents.fh):
  File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/home/mrocklin/workspace/gcsfs/gcsfs/gcsfuse.py", line 57, in readdir
    files = [f.rstrip('/').rsplit('/', 1)[1] for f in files]
  File "/home/mrocklin/workspace/gcsfs/gcsfs/gcsfuse.py", line 57, in <listcomp>
    files = [f.rstrip('/').rsplit('/', 1)[1] for f in files]
IndexError: list index out of range

Also, fwiw that bucket is publicly readable. You should be able to do the same thing locally if you're interested.

[asford]

Great, thanks. This is likely due to a mismatch in how objects & prefixes are matched to files & directories in that patch.

Is there any intention to lift the gcsfuse implementation into a generic interface support s3fs?

[martindurant]

@asford , we certainly could implement the same model of fuse for s3fs, and this idea has been mentioned; but there are already several projects around that do that for s3, which may be good enough.

[martindurant]

This should now work better

[martindurant]

Likely fixed in #67 , #57