Add TUF type #383

asraa · 2021-07-26T15:50:37Z

Makes #265 up to date. It is functional and tested now!

Some notes:

Root is passed in as the public-key. It is verified when a new public key is created (trusted the keys).
An artifact is a TUF manifest of any type. Since it includes signatures, you don't need to provide those out of band.
You can search by the root.json file, or the manifest hash, or the type/version.

asraa@asraa-glaptop:~/git/rekor$ ./rekor upload --artifact snapshot.json --public-key 1.root.json --type tuf --rekor_server http://localhost:3000
Created entry at index 0, available at: http://localhost:3000/api/v1/log/entries/6ed8fa5e9f0aa31b6cdfd2cc6877692f5afba52edd7ff5774eebfb22228e8847
asraa@asraa-glaptop:~/git/rekor$ ./rekor search --public-key 1.root.json --pki-format tuf --rekor_server http://localhost:3000
Found matching entries (listed by UUID):
6ed8fa5e9f0aa31b6cdfd2cc6877692f5afba52edd7ff5774eebfb22228e8847

@SantiagoTorres @trishankatdatadog @mnm678

pkg/types/tuf/tuf.go

trishankatdatadog · 2021-07-26T21:57:42Z

Looking forward to hacking on this this week!

Co-authored-by: Santiago Torres <santiagotorres@purdue.edu> Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com> Co-authored-by: Marina Moore <mnm678@gmail.com> Signed-off-by: Asra Ali <asraa@google.com>

dlorenc

The code LGTM, but I'll let it stick around for @bobcallaway to have a look.

One nit: could you add some docs here: https://github.com/sigstore/rekor/blob/main/types.md

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2021-08-13T19:52:04Z

One nit: could you add some docs here: https://github.com/sigstore/rekor/blob/main/types.md

Done!

mnm678

Great work @asraa! Sometime in the future it'd be great if we could generate the test data so that it doesn't clog up the repo, but otherwise LGTM

bobcallaway

overall looking good, a few questions and nits

pkg/types/tuf/v0.0.1/entry.go

pkg/types/tuf/v0.0.1/tuf_v0_0_1_schema.json

pkg/types/tuf/v0.0.1/tuf_manifest_v0_0_1_schema.json

pkg/types/tuf/tuf_schema.json

pkg/types/tuf/tuf_test.go

pkg/types/tuf/v0.0.1/tuf_key_v0_0_1_schema.json

pkg/types/tuf/v0.0.1/tuf_manifest_v0_0_1_schema.json

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2021-08-16T20:04:22Z

Comments addressed! had a minor snafu around decoding strfmt.DateTime -- all good now

dlorenc · 2021-08-16T20:10:58Z

Nice! I'll let @bobcallaway take another pass.

trishankatdatadog

Some initial comments. Great job, Asra!

pkg/generated/models/tuf_root_v001_schema.go

pkg/generated/models/tuf_snapshot_v001_schema.go

trishankatdatadog · 2021-08-17T01:58:59Z

pkg/pki/tuf/testdata/1.root.json

+				"threshold": 3
+			}
+		},
+		"spec_version": "1.0",


Hmm, this should fail, no? Don't we expect the spec version to be 1.0.0?

Two things: just fixed the TODO that actually grabs the spec version.
Second: go-tuf doesn't actually do major.minor.patch.... https://github.com/theupdateframework/go-tuf/blob/aee6270feb5596036edde4b6d7564fa17db811cb/client/testdata/go-tuf/consistent-snapshot-true/0/repository/root.json#L84

To close this out I made this a general string since go-tuf doesn't write in that format anyway.

trishankatdatadog · 2021-08-17T01:59:24Z

pkg/pki/tuf/testdata/timestamp.json

+				"version": 1
+			}
+		},
+		"spec_version": "1.0",


Same: should not work bcos should be 1.0.0, no?

pkg/types/tuf/tuf_schema.json

pkg/types/tuf/v0.0.1/tuf_v0_0_1_schema.json

trishankatdatadog · 2021-08-17T02:14:16Z

types.md

+IntegratedTime: 2021-08-13T19:17:33Z
+UUID: 6ed8fa5e9f0aa31b6cdfd2cc6877692f5afba52edd7ff5774eebfb22228e8847
+Body: {
+  "TufObj": {


Hmm, no version property here, is that intended?

I'm unsure why this is, I think on get it doesn't write the version, but when I unmarshal the actual content of the stored entry I can view it.

asraa@asraa-glaptop:~/git/rekor$ curl -fs0 http://localhost:3000/api/v1/log/entries/68163ebd7d7ba41ece736e2910915944a85985c61892f947841729476f81b951 > logEntry.json asraa@asraa-glaptop:~/git/rekor$ jq '."68163ebd7d7ba41ece736e2910915944a85985c61892f947841729476f81b951".body' logEntry.json | tr -d \" | base64 -d > body.json asraa@asraa-glaptop:~/git/rekor$ cat body.json | jq { "apiVersion": "0.0.1", "spec": { "metadata": { "content": "..." }, "root": { "content": "..." }, "spec_version": "1.0" }, "kind": "tuf" }

trishankatdatadog · 2021-08-17T02:23:10Z

pkg/types/tuf/v0.0.1/entry.go

+		result = append(result, strings.ToLower(hex.EncodeToString(rootHash[:])))
+	}
+
+	// TODO: Index individual key IDs?


Aha, I was looking for this :)

Yes, I think it would be super useful to be able to query Rekor for TUF metadata based on keyids. Could we pretty please add this? Happy to help with this effort also.

Yes!! Let's do this in a follow up. Should be pretty easy to do by adding a method for KeyIDs in the PublicKey type for TUF

pkg/types/tuf/v0.0.1/entry.go

pkg/types/tuf/v0.0.1/tuf_v0_0_1_schema.json

pkg/types/tuf/v0.0.1/tuf_manifest_v0_0_1_schema.json

Signed-off-by: Asra Ali <asraa@google.com>

asraa · 2021-08-18T14:32:02Z

OK TUF metadata files are now stored as JSON. Unmarshalling the content is just the whole JSON of the metadata file rather than the inline content + the fields we cared about (type, expires, etc)

asraa@asraa-glaptop:~/git/rekor$ curl -fs0  http://localhost:3000/api/v1/log/entries/68163ebd7d7ba41ece736e2910915944a85985c61892f947841729476f81b951 > logEntry.json
asraa@asraa-glaptop:~/git/rekor$ jq '."68163ebd7d7ba41ece736e2910915944a85985c61892f947841729476f81b951".body' logEntry.json | tr -d \" | base64 -d > body.json
asraa@asraa-glaptop:~/git/rekor$ cat body.json | jq
{
  "apiVersion": "0.0.1",
  "spec": {
    "metadata": {
      "content": "..."
    },
    "root": {
      "content": "..."
    },
    "spec_version": "1.0"
  },
  "kind": "tuf"
}
asraa@asraa-glaptop:~/git/rekor$ 
asraa@asraa-glaptop:~/git/rekor$ jq '."68163ebd7d7ba41ece736e2910915944a85985c61892f947841729476f81b951".body' logEntry.json | tr -d \" | base64 -d | jq '.spec.metadata.content' | tr -d \" | base64 -d  | jq
{
  "signatures": [
    {
      "keyid": "2f64fb5eac0cf94dd39bb45308b98920055e9a0d8e012a7220787834c60aef97",
      "sig": "3045022100a7aa1299c325e9d42442ab3d2d25332692d598d5128bc44849fe7108738104ba02201fd3551848bbe94d5c74b9c305920c2d885746134241f54af705380ca4326789"
    },
    {
      "keyid": "bdde902f5ec668179ff5ca0dabf7657109287d690bf97e230c21d65f99155c62",
      "sig": "304502202d3eff55465a12fb9f35ce189872f1d5487428e2b3dab32a57cdc02bf652ebc5022100a735d4399ccd6f78dfa3b708b9fd4ce8148d3e9cb9e6bdb3b4a9f997d3e6f88b"
    },
    {
      "keyid": "eaf22372f417dd618a46f6c627dbc276e9fd30a004fc94f9be946e73f8bd090b",
      "sig": "3046022100a0eefb45472bef9803dfdd16e657603cf2c96561c587057f7ad92dc7851dcfcc02210095a78985adb2a5a1c239ea43019eaed8e4cffe58a224a32f31ffb9c45bd45928"
    },
    {
      "keyid": "f40f32044071a9365505da3d1e3be6561f6f22d0e60cf51df783999f6c3429cb",
      "sig": "304502201cb475203a2726ba21db3cceeff39739f4e1c4b05a8af5e4d1155b60b88a107702210085d841e5420cf403ef65cd94ca92cb75c90b558ebf82de9b93370d7ea0612b56"
    },
    {
      "keyid": "f505595165a177a41750a8e864ed1719b1edfccd5a426fd2c0ffda33ce7ff209",
      "sig": "3044022044b6fa3e918110508c21999974f0a7716a9dd4a0a68234e7330c889e2e336cab0220271a14fa0ea822ed539746f13546139414ae2e8a961db6732f18c3aba08da892"
    }
  ],
  "signed": {
    "_type": "snapshot",
    "expires": "2021-12-18T13:28:12.99008-06:00",
    "meta": {
      "root.json": {
        "hashes": {
          "sha512": "87d41965ac08a2e03a33aa2db1dbbf5a7d6616e49f0ca3b20afe1c46ae78e52e841e21cca347f7b77ec929b6a3687d86d320ee04ba5f75a1880974310958b9c2"
        },
        "length": 5096,
        "version": 1
      },
      "targets.json": {
        "hashes": {
          "sha512": "14018a7442402d31ea7090cf9d7f1e0f56b1811884d843f33e1754b4d50521a83cf3425b93b49ad8a7eaedf5c993cec053e558888d1a97ec94d539abd97022a3"
        },
        "length": 2235,
        "version": 1
      }
    },
    "spec_version": "1.0",
    "version": 1
  }
}

If it's okay I'd like to address the size limits #414 in a separate PR. It's a little complicated and I want to make it general to each type (so that there are configurable limits on the bytes per type if you send content with a file or URL)

dlorenc · 2021-08-18T15:22:31Z

If it's okay I'd like to address the size limits #414 in a separate PR. It's a little complicated and I want to make it general to each type (so that there are configurable limits on the bytes per type if you send content with a file or URL)

SGTM

trishankatdatadog

I'm happy merging this PR now, and think we can address any improvements/changes in future PRs. Great work, Asra! 💯 💯

dlorenc · 2021-08-18T17:12:26Z

Here we go!

bobcallaway reviewed Jul 26, 2021

View reviewed changes

pkg/types/tuf/tuf.go Outdated Show resolved Hide resolved

cpanato modified the milestones: 0.3.0, v0.3.0, v0.4.0 Jul 27, 2021

asraa changed the title ~~WIP Add TUF~~ Add TUF type Aug 13, 2021

Adds rekor TUF type

6b85b73

Co-authored-by: Santiago Torres <santiagotorres@purdue.edu> Co-authored-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com> Co-authored-by: Marina Moore <mnm678@gmail.com> Signed-off-by: Asra Ali <asraa@google.com>

asraa force-pushed the add-tuf branch from 7f855ca to 6b85b73 Compare August 13, 2021 19:35

SantiagoTorres approved these changes Aug 13, 2021

View reviewed changes

dlorenc approved these changes Aug 13, 2021

View reviewed changes

add type documentation

7082aa4

Signed-off-by: Asra Ali <asraa@google.com>

mnm678 approved these changes Aug 13, 2021

View reviewed changes

bobcallaway reviewed Aug 13, 2021

View reviewed changes

asraa added 2 commits August 16, 2021 10:29

Address bob comments

bbc11ed

Signed-off-by: Asra Ali <asraa@google.com>

run make

7516242

Signed-off-by: Asra Ali <asraa@google.com>

trishankatdatadog reviewed Aug 17, 2021

View reviewed changes

bobcallaway reviewed Aug 17, 2021

View reviewed changes

asraa added 2 commits August 17, 2021 13:52

wip

e2d1df2

Signed-off-by: Asra Ali <asraa@google.com>

Address comments

a81c165

Signed-off-by: Asra Ali <asraa@google.com>

trishankatdatadog approved these changes Aug 18, 2021

View reviewed changes

dlorenc merged commit 4fcdcaa into sigstore:main Aug 18, 2021

cpanato modified the milestones: v0.4.0, v1.0.0 Aug 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TUF type #383

Add TUF type #383

asraa commented Jul 26, 2021 •

edited

Loading

trishankatdatadog commented Jul 26, 2021

dlorenc left a comment

asraa commented Aug 13, 2021

mnm678 left a comment

bobcallaway left a comment

asraa commented Aug 16, 2021

dlorenc commented Aug 16, 2021

trishankatdatadog left a comment

trishankatdatadog Aug 17, 2021

asraa Aug 17, 2021

asraa Aug 18, 2021

asraa Aug 18, 2021

trishankatdatadog Aug 17, 2021

trishankatdatadog Aug 17, 2021

asraa Aug 18, 2021

trishankatdatadog Aug 17, 2021

asraa Aug 17, 2021

asraa commented Aug 18, 2021

dlorenc commented Aug 18, 2021

trishankatdatadog left a comment

dlorenc commented Aug 18, 2021

Add TUF type #383

Add TUF type #383

Conversation

asraa commented Jul 26, 2021 • edited Loading

trishankatdatadog commented Jul 26, 2021

dlorenc left a comment

Choose a reason for hiding this comment

asraa commented Aug 13, 2021

mnm678 left a comment

Choose a reason for hiding this comment

bobcallaway left a comment

Choose a reason for hiding this comment

asraa commented Aug 16, 2021

dlorenc commented Aug 16, 2021

trishankatdatadog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asraa commented Aug 18, 2021

dlorenc commented Aug 18, 2021

trishankatdatadog left a comment

Choose a reason for hiding this comment

dlorenc commented Aug 18, 2021

asraa commented Jul 26, 2021 •

edited

Loading