Encryption at rest support #2424

ewoutp · 2017-06-08T07:18:11Z

What

This PR adds support for encrypting data stored by RocksDB when written to disk.

How

It adds an EncryptedEnv override of the Env class with matching overrides for sequential&random access files.
The encryption itself is done through a configurable EncryptionProvider. This class creates is asked to create BlockAccessCipherStream for a file. This is where the actual encryption/decryption is being done.
Currently there is a Counter mode implementation of BlockAccessCipherStream with a ROT13 block cipher (NOTE the ROT13 is for demo purposes only!!).

The Counter operation mode uses an initial counter & random initialization vector (IV).
Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The EncryptedEnv implementation is such that clients of the Env class do not see this prefix (nor data, nor in filesize).
The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there.

Testing

To test the encryption, the DBTestBase class has been extended to consider a new environment variable called ENCRYPTED_ENV. If set, the test will setup a encrypted instance of the Env class to use for all tests.
Typically you would run it like this:

ENCRYPTED_ENV=1 make check_some

There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With ENCRYPTED_ENV active it must not find plain text strings, with ENCRYPTED_ENV unset, it must find the plain text strings.

Fetch updates from facebook/rocksdb

Encryption

facebook-github-bot · 2017-06-08T07:18:19Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

db/db_encryption_test.cc

env/env_encryption.cc

Makefile

sagar0 · 2017-06-12T22:04:28Z

Thanks for your contribution!
Seems like Travis tests failed to kickoff, and there's also a merge conflict; so could you please rebase and submit again?

db/db_encryption_test.cc

facebook-github-bot · 2017-06-13T13:28:04Z

@ewoutp updated the pull request - view changes

Import changes from facebook-master

facebook-github-bot · 2017-06-13T14:18:19Z

@ewoutp updated the pull request - view changes

facebook-github-bot · 2017-06-13T14:23:04Z

@ewoutp updated the pull request - view changes

facebook-github-bot · 2017-06-13T14:25:29Z

@ewoutp updated the pull request - view changes

facebook-github-bot · 2017-06-13T14:27:19Z

@ewoutp updated the pull request - view changes

sagar0 · 2017-06-14T06:15:15Z

Travis lite_builds are failing on both linux and mac. Can you please take a look?
Also, I just restarted linux platform_dependent test, as it failed with 'No space left on device' error. (rate_limiter test in Mac platform_dependent test suite has been consistently failing due to unrelated issues. So we can disregard it for now).

ewoutp · 2017-06-14T06:21:57Z

@sagar0 Shall I just wrap the entire file in #ifndef ROCKSDB_LITE?

facebook-github-bot · 2017-06-14T17:30:01Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

facebook-github-bot · 2017-06-15T06:26:37Z

@ewoutp updated the pull request - view changes

facebook-github-bot · 2017-06-15T07:37:55Z

@ewoutp updated the pull request - view changes

ewoutp · 2017-06-15T08:48:46Z

Only java_test is failing. Have no clue how that is related.
Any ideas?

sagar0 · 2017-06-15T18:43:37Z

Thanks for fixing the lite build. Java test is a little flaky ... so we don't need to worry about it most of the time (we are working on making it less flaky though). I'll take a look.

commit 19ee74c Author: Ewout Prangsma <ewout@prangsma.net> Date: Thu Jun 15 09:37:40 2017 +0200 Excluded headers on ROCKSDB_LITE commit c688186 Author: Ewout Prangsma <ewout@prangsma.net> Date: Thu Jun 15 08:21:34 2017 +0200 Exclude encryption implementation on ROCKSDB_LITE commit 28d8074 Author: Ewout Prangsma <ewout@prangsma.net> Date: Tue Jun 13 16:27:08 2017 +0200 Commented `std::cout` in test (see review facebook#2424 (review)) commit d677055 Author: Ewout Prangsma <ewout@prangsma.net> Date: Tue Jun 13 16:25:18 2017 +0200 db_encryption_test.cc to TARGETS file (see review facebook#2424 (review)) commit 47c909b Author: Ewout Prangsma <ewout@prangsma.net> Date: Tue Jun 13 16:22:50 2017 +0200 Removed unwanted likes (see review facebook#2424 (review)) commit 64fccc8 Merge: b452426 66a5ef2 Author: Ewout Prangsma <ewoutp@users.noreply.github.com> Date: Tue Jun 13 16:18:08 2017 +0200 Merge pull request facebook#5 from arangodb-helper/facebook-master Import changes from facebook-master commit 66a5ef2 Merge: b452426 5d5a28a Author: Ewout Prangsma <ewout@prangsma.net> Date: Tue Jun 13 15:39:25 2017 +0200 Imported changed from facebook-master commit b452426 Author: Ewout Prangsma <ewout@prangsma.net> Date: Tue Jun 13 15:27:51 2017 +0200 Added GenerateUniqueId override commit 9d446a6 Author: Ewout Prangsma <ewoutp@users.noreply.github.com> Date: Thu Jun 8 08:55:29 2017 +0200 Encryption (wip) (facebook#2) Encryption commit 8d846d8 Merge: fe18356 c2be434 Author: Ewout Prangsma <ewoutp@users.noreply.github.com> Date: Fri May 12 15:33:34 2017 +0200 Merge pull request facebook#1 from facebook/master Fetch updates from facebook/rocksdb

sagar0

Thanks for the contribution, and making the prior requested changes. This change looks good to me, and is ready to be accepted once the below few nits are fixed.

Could you run a "make format" once to adhere to the formatting rules.

TARGETS

src.mk

env/env_encryption.cc

sagar0 · 2017-06-22T23:02:40Z

Would it also be possible to contribute a small example to show how this API can be used? It can be done in a separate pull request, when time permits.

facebook-github-bot · 2017-06-23T05:44:33Z

@ewoutp updated the pull request - view changes

ewoutp · 2017-06-23T05:50:50Z

Result of make format

$ make format
Makefile:105: Warning: Compiling in debug mode. Don't use the resulting binary in production
build_tools/format-diff.sh
Nothing needs to be reformatted!

ewoutp · 2017-06-23T06:13:54Z

As for sample code.

Typically you would implement a class derived from BlockCipher (see ROT13Cipher) and use it as follows:

ROT13BlockCipher rot13Cipher(16);
// ...
DBOptions options;
options.env = NewEncryptedEnv(Env::Default(), new CTREncryptionProvider(rot13Cipher));
// use options for opening/creating databases

facebook-github-bot · 2017-06-23T19:33:58Z

@ewoutp updated the pull request - view changes

sagar0 · 2017-06-23T22:04:42Z

I am trying to import this PR on to Facebook's internal phabricator to run some tests, but unfortunately the import is failing. I am trying to figure out a way to move forward.

facebook-github-bot · 2017-06-26T18:34:58Z

@ewoutp updated the pull request - view changes

sagar0 · 2017-06-26T18:41:06Z

@ewoutp Can you please rebase your changes on master?
We are encountering an issue when trying to import this PR onto our internal phabricator for running additional tests, and our workflow does not let us to merge the changes without that step.
I appreciate your patience while we are working through these kinks.

facebook-github-bot · 2017-06-26T18:54:25Z

@sdwilsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ewoutp · 2017-06-27T05:43:22Z

@sagar0 looks like it's already merged. If anything is still needed, let me know.

sagar0 · 2017-06-27T17:12:19Z

@ewoutp yeah, I merged it yesterday. @sdwilsh fixed an issue on our side in the import script, and helped in the process.

vivekpm · 2021-05-12T23:10:45Z

Is this available through RocksDBJNI?

ewoutp added 2 commits May 12, 2017 15:33

Merge pull request #1 from facebook/master

8d846d8

Fetch updates from facebook/rocksdb

Encryption (wip) (#2)

9d446a6

Encryption

siying requested review from ajkr and lightmark June 12, 2017 13:23

sagar0 reviewed Jun 12, 2017

View reviewed changes

db/db_encryption_test.cc Outdated Show resolved Hide resolved

env/env_encryption.cc Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

sagar0 reviewed Jun 12, 2017

View reviewed changes

db/db_encryption_test.cc Outdated Show resolved Hide resolved

Added GenerateUniqueId override

b452426

ewoutp and others added 2 commits June 13, 2017 15:39

Imported changed from facebook-master

66a5ef2

Merge pull request #5 from arangodb-helper/facebook-master

64fccc8

Import changes from facebook-master

Removed unwanted likes (see review #2424 (review))

47c909b

db_encryption_test.cc to TARGETS file (see review #2424 (review))

d677055

Commented std::cout in test (see review #2424 (review))

28d8074

facebook-github-bot added the CLA Signed label Jun 14, 2017

Exclude encryption implementation on ROCKSDB_LITE

c688186

Excluded headers on ROCKSDB_LITE

19ee74c

sagar0 reviewed Jun 22, 2017

View reviewed changes

TARGETS Show resolved Hide resolved

src.mk Show resolved Hide resolved

env/env_encryption.cc Outdated Show resolved Hide resolved

Processed sagar0's review comments

49e5f81

Merge branch 'master' into master

ae3ed39

Merge branch 'master' into master

80920f5

facebook-github-bot closed this in 5177861 Jun 26, 2017

manishrjain mentioned this pull request Jan 18, 2019

Support encryption at rest dgraph-io/badger#443

Closed

maysamyabandeh mentioned this pull request Jun 12, 2019

Assertion failed in BlockAccessCipherStream #5417

Open

SomberNight mentioned this pull request Aug 8, 2019

replace json on disk, with leveldb/sql/...? wallet file does not scale. partial writes spesmilo/electrum#4823

Open

CMCDragonkai mentioned this pull request Sep 29, 2021

Integrate Automatic Indexing into DB MatrixAI/js-db#1

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encryption at rest support #2424

Encryption at rest support #2424

ewoutp commented Jun 8, 2017

facebook-github-bot commented Jun 8, 2017

sagar0 commented Jun 12, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

sagar0 commented Jun 14, 2017 •

edited

Loading

ewoutp commented Jun 14, 2017

facebook-github-bot commented Jun 14, 2017

facebook-github-bot commented Jun 15, 2017

facebook-github-bot commented Jun 15, 2017

ewoutp commented Jun 15, 2017

sagar0 commented Jun 15, 2017 •

edited

Loading

sagar0 left a comment •

edited

Loading

sagar0 commented Jun 22, 2017

facebook-github-bot commented Jun 23, 2017

ewoutp commented Jun 23, 2017 •

edited

Loading

ewoutp commented Jun 23, 2017 •

edited

Loading

facebook-github-bot commented Jun 23, 2017

sagar0 commented Jun 23, 2017

facebook-github-bot commented Jun 26, 2017

sagar0 commented Jun 26, 2017

facebook-github-bot commented Jun 26, 2017

ewoutp commented Jun 27, 2017

sagar0 commented Jun 27, 2017

vivekpm commented May 12, 2021

Encryption at rest support #2424

Encryption at rest support #2424

Conversation

ewoutp commented Jun 8, 2017

What

How

Testing

facebook-github-bot commented Jun 8, 2017

sagar0 commented Jun 12, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

facebook-github-bot commented Jun 13, 2017

sagar0 commented Jun 14, 2017 • edited Loading

ewoutp commented Jun 14, 2017

facebook-github-bot commented Jun 14, 2017

facebook-github-bot commented Jun 15, 2017

facebook-github-bot commented Jun 15, 2017

ewoutp commented Jun 15, 2017

sagar0 commented Jun 15, 2017 • edited Loading

sagar0 left a comment • edited Loading

Choose a reason for hiding this comment

sagar0 commented Jun 22, 2017

facebook-github-bot commented Jun 23, 2017

ewoutp commented Jun 23, 2017 • edited Loading

ewoutp commented Jun 23, 2017 • edited Loading

facebook-github-bot commented Jun 23, 2017

sagar0 commented Jun 23, 2017

facebook-github-bot commented Jun 26, 2017

sagar0 commented Jun 26, 2017

facebook-github-bot commented Jun 26, 2017

ewoutp commented Jun 27, 2017

sagar0 commented Jun 27, 2017

vivekpm commented May 12, 2021

sagar0 commented Jun 14, 2017 •

edited

Loading

sagar0 commented Jun 15, 2017 •

edited

Loading

sagar0 left a comment •

edited

Loading

ewoutp commented Jun 23, 2017 •

edited

Loading

ewoutp commented Jun 23, 2017 •

edited

Loading