tar.py uses "--force-local" option on MacOSX, which doesn't exist

[bshaffer]

Environment details

OS: MacOSX
Language runtime version:
Package version: 0.6.2

Steps to reproduce

1. Run docuploader upload on MacOSX

$  docuploader upload out --staging-bucket docs-staging-v2-dev --destination-prefix docfx- --metadata-file /path/to/docs.metadata

Stack trace

  tar: Option --force-local is not supported                                                                                                                                                                                            
  Usage:                                                                                                                                                                                                                                
    List:    tar -tf <archive-filename>                                                                                                                                                                                                 
    Extract: tar -xf <archive-filename>                                                                                                                                                                                                 
    Create:  tar -cf <archive-filename> [filenames...]                                                                                                                                                                                  
    Help:    tar --help                                                                                                                                                                                                                 
                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                        
  Error Output:                                                                                                                                                                                                                         
  ================                                                                                                                                                                                                                      
  docuploader > Let's upload some docs!                                                                                                                                                                                   
  docuploader > Loading up your metadata.                                                                                                                                                                                 
  docuploader > Looks like we're uploading vision version 2.2.3 for php.                                                                                                                                                  
  docuploader > Sit tight, I'm tarring up your docs in out.                                                                                                                                                               
  docuploader > Failed executing tar --create --directory=out --file=/var/folders/l2/600rtyfd7bjb5k5ypp_g6xl8009z5m/T/tmpnyh50hxv --force-local --gzip --verbose .:                                                           
                                                                                                                                                                                                                                        
  None                                                                                                                                                                                                                              
  Traceback (most recent call last):                                                                                                                                                                                                    
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1130, in __call__                                                                                                                         
      return self.main(*args, **kwargs)                                                                                                                                                                                                 
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1055, in main                                                                                                                             
      rv = self.invoke(ctx)                                                                                                                                                                                                             
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1657, in invoke                                                                                                                           
      return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                           
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1404, in invoke                                                                                                                           
      return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                    
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/click/core.py", line 760, in invoke                                                                                                                            
      return __callback(*args, **kwargs)                                                                                                                                                                                                
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/__main__.py", line 126, in upload                                                                                                                  
      docuploader.tar.compress(documentation_path, tar_filename)                                                                                                                                                                        
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/tar.py", line 27, in compress                                                                                                                      
      return shell.run(                                                                                                                                                                                                                 
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/shell.py", line 48, in run                                                                                                                         
      raise exc                                                                                                                                                                                                                         
    File "/Users/betterbrent/Library/Python/3.9/lib/python/site-packages/docuploader/shell.py", line 35, in run                                                                                                                         
      return subprocess.run(                                                                                                                                                                                                            
    File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run                                                                                                
      raise CalledProcessError(retcode, process.args,                                                                                                                                                                                   
  subprocess.CalledProcessError: Command '['tar', '--create', '--directory=out', '--file=/var/folders/l2/600rtyfd7bjb5k5ypp_g6xl8009z5m/T/tmpnyh50hxv', '--force-local', '--gzip', '--verbose', '.']' returned non-zero exit status 1.

Solution

The error is "tar: Option --force-local is not supported"

I was able to solve this by opening up tar.py and commenting out the option --force-local.

[tbpg]

The --force-local option says:

Archive file is local even if it has a colon.

We don't have any archives with a colon in the name:

$ gsutil ls gs://docs-staging-v2 | sed 's|gs://||' | grep :`
# Empty
$ gsutil ls gs://docs-staging | sed 's|gs://||' | grep :
# Empty

So, I believe we can safely remove the --force-local flag. I'll send a PR.

[tbpg]

Ah, it was added in #9 to handle Windows filenames. @jskeet, I assume this is still an issue? Any opinions on how to handle this?

[tbpg]

I sent #139 to add a fallback so we can support both modes. An alternative would be to check if we're running on Mac, but I'd prefer to try for the --force-local functionality rather than test indirectly.

[jskeet]

@tbpg: I'm unsure - it doesn't look like we actually use that anywhere. I suspect it was actually a matter of when I was running it locally.

To be honest, with other bits of Python eventually going away, what I'd really welcome is a specification in this repo of what the file needs to look like, so I could build my own .NET equivalent that would avoid me having to upgrade Python dependencies that I don't really know the details of...

[tbpg]

@jskeet that's fair. Mind filing a feature request? I'd prefer not to "export" the API of docuploader and the specific format docuploader uploads in...

I'd also prefer not to rewrite this in some other language where we can ship a static binary, so you wouldn't have to worry about installation/dependencies/Python/etc.

Not sure how to achieve both.

[jskeet]

Created #141, but I do take your point, so feel free to close.

I think it partly depends on how "done" we think docuploader is. My understanding is that's really pretty simple:

• A metadata file
• A tar file
• Cloud Storage upload

(To the extent that if the metadata file is documented, the rest can probabyl be done via tar + gsutil.)

If the metadata format is only going to change very rarely, it's quite possibly worth doing. If we see a future with more features being added or a larger overall design than the three bullets above, it's probably not worth doing.