Move to TOML for manifest and lock #342

carolynvs · 2017-03-21T01:29:59Z

Fixes #119; fixes #168.

I'm doing this in stages so that we can identify when I'm off in the weeds or need to make more decisions about the config file format.

Vendor https://github.com/pelletier/go-toml.
Write manifest and lock in TOML format.
Read manifest and lock in TOML format (all tests but diff should pass)
Remove old JSON files and logic.
Fix output of dry-run to use TOML (diff test and all others should pass)
(Optional) Bike-shed the look of the generated TOML, e.g. inline tables, indenting, key sorting, etc.

carolynvs · 2017-03-21T01:35:58Z

@sdboyer @tro3 I didn't find any reflection-based marshal magic for converting between structs and TOML in this go-toml package. If we want to go with this lib, it may make sense to contribute that functionality back into the library. For the moment, I am explicitly performing the mapping in MarshalTOML.

https://github.com/golang/dep/pull/342/files#diff-8e0a04a8bff74b300db5882b4be50589R133

Do you think that makes sense or would you prefer that I try out another lib?

sdboyer · 2017-03-21T12:33:42Z

@carolynvs hmm...that puts me on the fence about it, as I'd generally rather have the (presumably cross-TOML implementation portable) struct tags as the canonical way of expressing this information, to minimize effort in the event that we switch TOML libs later. idk how much work that's really saving, though, so it seems like a soft requirement at worst.

Maybe some input from @pelletier here would be good - any plans to add reflection-based struct tag support to go-toml? Would you accept such a change, if offered?

sdboyer · 2017-03-21T12:41:28Z

Eh, that was too wishy-washy, sorry. If we can have struct tags and reflection-based parsing, great. If not, well, it seems like the one-off implementation here isn't too awful. Either way, not a reason to derail the current track.

Is there are reason you're keeping the JSON struct tags around?

pelletier · 2017-03-21T12:42:20Z

@sdboyer yes, reflection-based (un)marshaling is the next thing that I'd like to implement in go-toml. no eta on that though, but pull requests are welcome :)

tro3 · 2017-03-21T13:04:50Z

@pelletier - I have some time coming up the next few weeks. Mind if I take a crack at the reflection-based marshaling? If so, I'll open an issue at go-toml and get you a WIP PR in the short term to make sure you approve of the structure before I dive in too deep. Thoughts?

pelletier · 2017-03-21T13:06:09Z

@tro3 sounds great!

carolynvs · 2017-03-21T13:07:35Z

@sdboyer No reason, just hadn't gotten around to removing all the json stuff yet. 😃 I've added a task to the PR to make that more clear.

carolynvs · 2017-03-22T17:18:55Z

Changed manifest format. dependencies and overrides use an array of tables instead of maps now, which is easier to work with.

ignores = ["github.com/foo/bar"]

[[dependencies]]
  name = "github.com/sdboyer/gps"
  version = ">=0.12.0, <1.0.0"

[[dependencies]]
  name = "github.com/babble/brook"
  revision = "d05d5aca9f895d19e9265839bffeadd74a2d2ecb"

[[overrides]]
  branch = "master"
  name = "github.com/sdboyer/gps"
  source = "https://github.com/sdboyer/gps"

Update all the test files from JSON to TOML
Poor man's mapping from TOML to structs. Can't wait for Reflection-based marshaling / unmarshaling pelletier/go-toml#149! ❤️

Next up I'm fixing the lock diff format and looking sorting bug (tests pass then fail due to sorting).

omeid · 2017-03-23T00:59:55Z

Couldn't now overrides be just an attribute on the dependency itself?

carolynvs · 2017-03-23T15:14:22Z

@omeid Let's not make changes to the config file structure in this PR, beyond the initial scope of switching from JSON to TOML. 😀 It's already a pretty big change, and I want to limit the amount of regressions/confusion it may cause.

nathany · 2017-03-23T16:55:47Z

For generated TOML, I think it would be nice to use inline tables to get each dependency onto a single line.

I'm think that may work better for diffs/merges, though I'm not totally sure. Also it's nice for rearranging the dependencies by movings lines up/down in an editor (for super organized types).

I can imagine counter arguments though, as lines could get pretty long when there are a number of options specified.

carolynvs · 2017-03-23T17:31:42Z

@nathany I think that depends on how we feel about long lines in the file. Personally, I find it hard to read.

For example, here's what I think the lock would look like with inline tables. Note, I am cheating a bit by using "." for the package list, to make the shortest possible entry.

memo = ""
projects = [
 { name = "github.com/sdboyer/deptest", packages = ["."], revision = "ff2948a2ac8f538c4ecd55962e919d1e13e74baf", version = "v0.8.0" }
 { name = "github.com/sdboyer/deptestdos", packages = ["."], revision = "a0196baa11ea047dd65037287451d36b861b00ea", version = "1.0.0" }
]

Though I'm not 100% sure that's how an array of inline tables should look? Either way support for changing the formatter in go-toml would be needed, not sure about the parser.

nathany · 2017-03-23T18:45:41Z

Okay. Those are some pretty long lines.

It's probably best to just use it for a while to see how well diffs/merges work with the multi-line syntax. Thanks

carolynvs · 2017-03-29T03:31:31Z

I've updated the lock diff to output matching TOML. I've also tried out @tro3 's reflection based mapping, it's super close and I think it's worth waiting for it, removes a TON of code. 😃

pelletier · 2017-03-29T21:54:37Z

pelletier/go-toml#149 has been merged, bringing reflection-based marshaling goodness to go-toml. props to @tro3 :)

sdboyer · 2017-03-31T23:08:20Z

We need to roll the file name change into this, too, so that we minimize the amount of change that goes in at once. We're still pondering those, but...

Cursory look has me thinking this is pretty good. I'll look more soon, I hope later tonight :)

sdboyer

Mostly nits, a couple slightly bigger things.

Sorry for the slow action on this!

sdboyer · 2017-04-01T00:24:28Z

analyzer_test.go

@@ -18,7 +18,7 @@ func TestDeriveManifestAndLock(t *testing.T) {
 	defer h.Cleanup()

 	h.TempDir("dep")
-	golden := "analyzer/manifest.json"
+	golden := "analyzer/manifest.toml"


I realize you didn't introduce this, but now that I'm seeing it, I believe it should be filepath.Join("analyzer", ManifestName).

FWIW, being a PC guy, I've found Go does a startingly good job of using '/' in a platform-independent manner in path strings. But filepath.Join is still likely more robust.

sdboyer · 2017-04-01T00:27:24Z

cmd/dep/testdata/harness_tests/ensure/empty/case1/final/lock.toml

+memo = "63510efb9632ec69c1164ce396d7ebea4ad3884b4fa508373da17226d5a39739"
+
+[[projects]]
+  name = "github.com/sdboyer/deptest"


Hmm...yeah, I hate to bikeshed, but these indents are kinda making me twitch. What's your thinking behind indenting vs. not? Are there any typical standards out there for it? Do we know anything about, say, widespread editor defaults for indentation in TOML files?

The go-toml package will indent on -update. (It will read fine with or without indent.) Don't know if there is a setting to stop it - never looked.

Yeah, preferably go-toml would let us specify formatting options. I'll double-check that doesn't exist already and see if it's an easy thing to add.

Okay, it's not exposed but seems like it wouldn't be hard to add a flag to control the auto-indentation that it's currently doing. I'll submit a PR to go-toml for that.

sdboyer · 2017-04-01T00:28:59Z

cmd/dep/testdata/harness_tests/init/case1/final/lock.toml

@@ -0,0 +1,12 @@
+memo = ""


Any idea how this one ended up without a memo? Mostly just curious.

I am not sure why the memo is empty? 😄 It's been like that for a while; here's there current lock.json for that test:

https://github.com/golang/dep/blob/master/cmd/dep/testdata/harness_tests/init/case1/final/lock.json#L2

Think that's a bug?

sdboyer · 2017-04-01T00:37:06Z

fs.go

+// Marshaler is the interface implemented by types that
+// can marshal themselves into valid TOML.
+// TODO(carolynvs) Consider adding this to go-toml?
+type Marshaler interface {


Probably better not to export this as an interface, unless we have to. As your TODO notes, if it's going to be exported, that should be done by go-toml. If we can get away with exporting, marshaler, or better tomlMarshaler would be preferable.

Thanks for noticing this! I had intended to put it in go-toml but that slipped my mind... 😊

I ended up needing this to use reflection for the LockDiff. Here's the go-toml PR: pelletier/go-toml#151

sdboyer · 2017-04-01T00:46:36Z

manifest.go

-	for n, pp := range rm.Dependencies {
-		m.Dependencies[gps.ProjectRoot(n)], err = toProps(n, pp)
+	for i := 0; i < len(raw.Dependencies); i++ {
+		name, prj, err := toProject(raw.Dependencies[i])


The cost of moving to an array from a map is that it's now possible for users to express the same project more than once. This loop has to validate that each entry in the input is unique, and fail hard otherwise.

sdboyer · 2017-04-01T00:48:13Z

manifest.go


-	for n, pp := range m.Ovr {
-		raw.Overrides[string(n)] = toPossible(pp)
+// TODO(carolynvs) when gps is moved, we can use the unexported gps.sortedConstraints


It'll still be unexported within the gps package, so probably not 😄

sdboyer · 2017-04-01T00:49:57Z

test/test.go

@@ -584,7 +584,7 @@ func (h *Helper) Cleanup() {

 // ReadManifest returns the manifest in the current directory.
 func (h *Helper) ReadManifest() string {
-	m := filepath.Join(h.pwd(), "manifest.json")
+	m := filepath.Join(h.pwd(), "manifest.toml")


Use the constant

I don't believe I can use the constant here (in the test pkg) because that would create a circular dependency?

ahhh, totally, yes. my bad :)

let's create a local constant in this test package too, and use that. it's not as good as one constant, but it's still better than string literals all over.

sdboyer · 2017-04-01T00:50:03Z

test/test.go

@@ -594,7 +594,7 @@ func (h *Helper) ReadManifest() string {

 // ReadLock returns the lock in the current directory.
 func (h *Helper) ReadLock() string {
-	l := filepath.Join(h.pwd(), "lock.json")
+	l := filepath.Join(h.pwd(), "lock.toml")


Use the constant

sdboyer · 2017-04-01T00:50:52Z

testdata/txn_writer/expected_diff_output.txt

-    }
-]
+
+Add:


Hot diggity damn these look so much better 😄

sdboyer · 2017-04-01T00:52:09Z

txn_writer.go

-	Branch   *StringDiff     `json:"branch,omitempty"`
-	Revision *StringDiff     `json:"revision,omitempty"`
-	Packages []StringDiff    `json:"packages,omitempty"`
+	Name     gps.ProjectRoot


these don't need toml struct tags?

Sorry an emergency came up and I stopped mid review. I'll start back on this today. The current toml mapper doesn't yet support allowing a nested type to use a custom marshal function, so it was manual. I started adding that this weekend and hopefully can submit that to go-toml soon and use it.

carolynvs · 2017-04-05T02:08:17Z

@sdboyer You mentioned rolling up the filename changes into this PR to limit churn. Do we have final names yet?

github.com/pelletier/go-toml@fee7787d3f811af92276f5ff10107092e95b7a1d - Can't use latest release, it has a show-stopping bug in ToTomlString github.com/pelletier/go-buffruneio@v0.2.0

Using the project root as a key in TOML makes our life harder. This siplifies the manifest format and parsing.

Even when gps is moved into dep, sortedConstraints won’t be exported

carolynvs · 2017-04-05T04:40:43Z

@sdboyer @tro3 Right now go-toml doesn't preserve the ordering of the fields when marshaling, instead simply outputting them in alphabetical order:

[[projects]]
  branch = "master"
  name = "github.com/pelletier/go-buffruneio"
  packages = ["."]
  revision = "c37440a7cf42ac63b919c752ca73a85067e05992"

instead of this (which matches the ordering of the fields as defined on the struct which is what encoding/json does)

[[projects]]
  name = "github.com/pelletier/go-buffruneio"
  branch = "master"
  revision = "c37440a7cf42ac63b919c752ca73a85067e05992"
  packages = ["."]

Is this something that I need to address as part of this PR, in the name of not changing the file structure again later? It may be a big change to how the TomlTree is represented internally and could be a lot of work, so I wanted to get your thoughts (or maybe @tro3 you see a clean way of supporting that)?

carolynvs · 2017-04-05T04:46:36Z

Rebased, resolving the merge conflicts from updating gps
Added constants for the file names to the lock package
Updated LockDiff formatting to the toml struct tags

Looking into making the auto-indentation configurable next.

We can’t reference dep.ManifestName/LockName because it would create a circular reference

tro3 · 2017-04-05T12:10:47Z

Is it really needed? Don't know if he key order is important. In any case, I think it's an easy tweak to go-toml, as I recall an explicit sorting of the keys. If correct, this would just have to be bypassed with a configure option of some sort. I'm on vacation though, so if it needs to be me, it will have to wait a while.

carolynvs · 2017-04-06T02:41:15Z

I spoke with @sdboyer and we will leave off trying to make the toml file format "pretty", so I am not going to try to tackle the indenting or sorting in this PR. I believe I've hit all the rest of the review feedback so it's ready for another look.

sdboyer · 2017-04-06T09:23:33Z

OK for filenames, let's do Gopkg.toml and Gopkg.lock.

carolynvs · 2017-04-06T16:46:12Z

File renaming is in! 💥

bradleyfalzon · 2017-04-08T22:07:02Z

Is it possible, or was it already decided not, to read the old format and update the relevant files?

As the migration path, as I understand it, would be to remove the old files and generate new lock files - which would lock to different revisions?

Detecting existing files, reading them and rewriting to new files and new format would obviously be ideal from an early user's perspective. Perhaps a flag, or a separate application (depfix)?

I don't see this as a major requirement at all, just asking.

sdboyer · 2017-04-13T00:51:58Z

Sorry for delay on merge, here. I was trying to get things together with a blog for dep (re: #294), and I put this in the queue behind doing that. Looks like I should have that blog done tonight, and I plan to merge this once it's all set up 😄

sdboyer · 2017-04-13T05:24:29Z

And in, finally. Thanks! 🎉 🎉 🎉 🎉

golang/dep#342

Move to TOML for manifest and lock

future-architect#421) golang/dep#342

googlebot added the cla: yes label Mar 21, 2017

sdboyer added the before-merge-into-toolchain label Mar 21, 2017

carolynvs force-pushed the toml-allthethings branch from 8652142 to 5bba6bd Compare March 29, 2017 01:22

sdboyer self-requested a review March 29, 2017 22:06

carolynvs changed the title ~~[WIP] Move to TOML for manifest and lock~~ Move to TOML for manifest and lock Apr 1, 2017

sdboyer suggested changes Apr 1, 2017

View reviewed changes

carolynvs added 7 commits April 4, 2017 21:10

Vendor pelletier/go-toml TOM library and deps

49136f2

github.com/pelletier/go-toml@fee7787d3f811af92276f5ff10107092e95b7a1d - Can't use latest release, it has a show-stopping bug in ToTomlString github.com/pelletier/go-buffruneio@v0.2.0

Add MarshalTOML to manifest and lock

d3135b8

Write manifest and lock in TOML

6366f9e

Represent manifest dependencies as an array of tables, not a map

7b22cd8

Using the project root as a key in TOML makes our life harder. This siplifies the manifest format and parsing.

Change manifest and lock file names to manifest.toml and lock.toml

d2a557e

Update test files from JSON to TOML

12e713b

Update dep's own manifest and lock to TOML

d82a114

carolynvs added 2 commits April 4, 2017 23:31

Remove TODO comment

9b835ca

Even when gps is moved into dep, sortedConstraints won’t be exported

Use reflection-based TOML mapping for the LockDiff

b30a25c

carolynvs force-pushed the toml-allthethings branch from 079a526 to 920e5d4 Compare April 5, 2017 04:41

carolynvs added 2 commits April 4, 2017 23:56

Update go-toml for nested custom Marshaler support

0afa554

Use a constant for the manifest/lock file names

a068a3c

We can’t reference dep.ManifestName/LockName because it would create a circular reference

carolynvs force-pushed the toml-allthethings branch from 920e5d4 to a068a3c Compare April 5, 2017 04:57

sdboyer mentioned this pull request Apr 6, 2017

Discuss renaming lock.json and manifest.json #168

Closed

New filenames: Gopkg.toml and Gopkg.lock

e31a186

sdboyer approved these changes Apr 13, 2017

View reviewed changes

sdboyer merged commit b839831 into golang:master Apr 13, 2017

carolynvs mentioned this pull request Apr 13, 2017

hack/validate-vendor.bash: fix build dep & script without TRAVIS_BRANCH #369

Merged

elfgoh added a commit to elfgoh/vuls that referenced this pull request Apr 13, 2017

Fixing future-architect#420 where lock and manifest have moved to TOML

cbb2a81

golang/dep#342

elfgoh mentioned this pull request Apr 13, 2017

Fixing #420 where lock and manifest have moved to TOML future-architect/vuls#421

Merged

8 tasks

smira mentioned this pull request Apr 13, 2017

Update dep lock files aptly-dev/aptly#549

Closed

elfgoh mentioned this pull request Apr 13, 2017

Lock and manifest changed from json to toml vulsio/go-cve-dictionary#50

Closed

kotakanbe pushed a commit to future-architect/vuls that referenced this pull request Apr 14, 2017

Fixing #420 where lock and manifest have moved to TOML (#421)

f878e22

golang/dep#342

smira mentioned this pull request Apr 14, 2017

Rewrite dep files into new format aptly-dev/aptly#550

Merged

ibrasho pushed a commit to ibrasho-forks/dep that referenced this pull request May 10, 2017

Merge pull request golang#342 from carolynvs/toml-allthethings

e68b550

Move to TOML for manifest and lock

lapthorn pushed a commit to lapthorn/vuls that referenced this pull request May 11, 2017

Fixing future-architect#420 where lock and manifest have moved to TOML (

3dc7453

future-architect#421) golang/dep#342

carolynvs deleted the toml-allthethings branch June 11, 2017 15:26

marccarre mentioned this pull request Oct 6, 2017

Q: Migrate / upgrade project from manifest.json / lock.json to Gopkg.toml / Gopkg.lock? #1242

Closed

-                  }
-              ]
+              Add:

Move to TOML for manifest and lock #342

Move to TOML for manifest and lock #342

Conversation

carolynvs commented Mar 21, 2017 • edited by sdboyer Loading

carolynvs commented Mar 21, 2017 • edited Loading

sdboyer commented Mar 21, 2017 • edited Loading

sdboyer commented Mar 21, 2017

pelletier commented Mar 21, 2017 • edited Loading

tro3 commented Mar 21, 2017

pelletier commented Mar 21, 2017

carolynvs commented Mar 21, 2017 • edited Loading

carolynvs commented Mar 22, 2017 • edited Loading

omeid commented Mar 23, 2017

carolynvs commented Mar 23, 2017

nathany commented Mar 23, 2017 • edited Loading

carolynvs commented Mar 23, 2017

nathany commented Mar 23, 2017

carolynvs commented Mar 29, 2017

pelletier commented Mar 29, 2017

sdboyer commented Mar 31, 2017

sdboyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolynvs commented Apr 5, 2017

carolynvs commented Apr 5, 2017 • edited Loading

carolynvs commented Apr 5, 2017

tro3 commented Apr 5, 2017 via email

carolynvs commented Apr 6, 2017

sdboyer commented Apr 6, 2017

carolynvs commented Apr 6, 2017

bradleyfalzon commented Apr 8, 2017

sdboyer commented Apr 13, 2017

sdboyer commented Apr 13, 2017

carolynvs commented Mar 21, 2017 •

edited by sdboyer

Loading

carolynvs commented Mar 21, 2017 •

edited

Loading

sdboyer commented Mar 21, 2017 •

edited

Loading

pelletier commented Mar 21, 2017 •

edited

Loading

carolynvs commented Mar 21, 2017 •

edited

Loading

carolynvs commented Mar 22, 2017 •

edited

Loading

nathany commented Mar 23, 2017 •

edited

Loading

carolynvs commented Apr 5, 2017 •

edited

Loading