Ability to Upload / Download with Passphrase

[unl1k3ly]

Hi,

Firstly, thank you very much for the cli. It's very useful for many, including me!!

Would be possible to add as a feature the ability to download files and upload files with a passphrase protection ?

Cheers

[ehuggett]

Working on it! The password support will come after upload/download support for this version.

Send now makes use of HKDF via crypt.subtle.deriveKey, but as yet i have not yet been able to get the same output in python (using HKDF from Cryptodome.Protocol.KDF).

There are now more keys in use, all of which are derived from the url-secret (secretKey) using HKDF (previously the url-secret WAS the encryption key, but now the encryption key is derived from this secret).

• secretKey - a random 128bit value, all other keys are derived from this value.
• encryptKey - used to encrypt the file
• authKey - Submitted to the Send server during upload in a HTTP header (WWW-Authenticate, base64 encoded and prefixed with "send-v1 "), I need to check further but i suspect the downloader is expected to prove possession of the secretKey by deriving authKey from it (and using it somehow, I've not looked at downloading properly yet)
• metaKey - used the encrypt the files metadata (filename, file type and the IV used when encrypting the file).

[ehuggett]

No idea what i was doing wrong with HKDF, it seems quite simple now. I can generate all of the required keys

import Cryptodome.Hash.SHA256
from Cryptodome.Protocol.KDF import HKDF
from Cryptodome.Random import get_random_bytes

def generateSecretKey():
   return get_random_bytes(16)

def generateEncryptKey(secretKey):
   master = secretKey
   key_len = 16
   salt = b''
   hashmod = Cryptodome.Hash.SHA256
   num_keys = 1
   context = b'encryption'
   return HKDF(master, key_len, salt, hashmod, num_keys=num_keys, context=context)

def generateAuthKey(secretKey):
   master = secretKey
   key_len = 64
   salt = b''
   hashmod = Cryptodome.Hash.SHA256
   num_keys = 1
   context = b'authentication'
   return HKDF(master, key_len, salt, hashmod, num_keys=num_keys, context=context)

def generateMetaKey(secretKey):
   master = secretKey
   key_len = 16
   salt = b''
   hashmod = Cryptodome.Hash.SHA256
   num_keys = 1
   context = b'metadata'
   return HKDF(master, key_len, salt, hashmod, num_keys=num_keys, context=context)

def generateSubKeys(secretKey=generateSecretKey() ):
   return {
      'secretKey' : secretKey,
      'encryptKey' : generateEncryptKey(secretKey),
      'authKey' : generateAuthKey(secretKey),
      'metaKey' : generateMetaKey(secretKey),
   }

keys = generateSubKeys()
for i in keys:
   print(str(i) + ':' + str(keys[i]))

[unl1k3ly]

Thanks man !

[ehuggett]

I have file upload and setting a password working in a rather ugly local branch, but i haven't touched downloading yet.

I've copied everything but setting a password into https://github.com/ehuggett/send-cli/tree/next-version (based the branch on travis-pytest instead of the current master, making this somewhat dependant upon #2 )

edit: next-version can now optionally set a password

# get the source
git clone https://github.com/ehuggett/send-cli.git
cd send-cli
git checkout next-version
# I wouldn't install this outside of a virtual environment, but technically optional... 
mkdir venv
virtualenv ./venv
source venv/bin/activate
# Install in editable mode, 
pip install -e .
# Now use it, you should find that send-cli is in your $PATH
send-cli --service "https://send.dev.mozaws.net/api/upload" file-to-upload

#when done, exit the venv
deactivate

edit2: the local branch can now download! (without a password)

[ehuggett]

To know if a password is required for download, the page body/html must be parsed.

I already knew how to do this with BeautifulSoup, but its an additional dependency that should be avoidable

BeautifulSoup(response.text, 'html.parser').find(id='dl-file')['data-requires-password']

[ehuggett]

@unl1k3ly the next-version branch can now be used to upload and download files, with or without a password

Remember to specify --service "your-server" or --service-dev and you may need to use --ignore-version

[unl1k3ly]

Thanks heaps mate!! I'm deploying a private SEND instance and will test this out... btw, did you set up send in a private AWS or you relaying on Mozilla infra ? Cheers

[ehuggett]

Nearly all testing is done against a local send (npm start with some options changed to bind to 0.0.0.0 as its running in a vm) but other than that don't have anything running myself.

head's up: the --service url no longer ends with api/upload, but its needs to end with a forward slash at the moment

You can now set the environment variable SEND_SERVER instead of using --service ie

SEND_SERVER="https://your-server.tld/" send-cli path/to/file

will upload "file" to a send server running at https://your-server.tld/

[ehuggett]

Closing as these changes are now in master.