Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: db driven platform settings #1110

Merged
merged 87 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
9aad646
wip: settings started
plyr4 Mar 7, 2024
9038758
chore: revert makefile
plyr4 Mar 19, 2024
0a4bf6a
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Mar 20, 2024
e779f5d
wip: platform settings using new types pattern
plyr4 Mar 21, 2024
be2edac
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 1, 2024
4628b02
feat: boiler plate api code for settings endpoints
plyr4 Apr 1, 2024
4cf60eb
feat: wip settings object used for clone image
plyr4 Apr 2, 2024
0909e2f
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 2, 2024
7e24e65
feat: wip settings embedded into compiler
plyr4 Apr 2, 2024
336dcf5
feat: query params to control output format, and clone image control
plyr4 Apr 3, 2024
a200311
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 9, 2024
42726c8
wip: infra for queue routes and starlark exec limit
plyr4 Apr 10, 2024
04e43f0
wip: update queue and compiler using middleware
plyr4 Apr 11, 2024
648dd78
wip: debugging compile and publish panic
plyr4 Apr 11, 2024
3dba7ba
chore: merge with main
plyr4 Apr 16, 2024
c528210
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 Apr 16, 2024
ee9b90f
wip: structure for settings done
plyr4 Apr 17, 2024
51c82bc
wip: restructure using nested settings
plyr4 Apr 17, 2024
e052235
enhance: reorganize into subtypes using db scan
plyr4 Apr 17, 2024
1f36713
fix: linter
plyr4 Apr 17, 2024
82fabca
chore: remove debug files
plyr4 Apr 17, 2024
17d6e68
chore: remove debug line
plyr4 Apr 17, 2024
d9ac8db
chore: cleanup, fmt, remove repo allowlist
plyr4 Apr 17, 2024
68aff5e
golangci-lint run --fix
plyr4 Apr 17, 2024
05d0389
fix: linter
plyr4 Apr 17, 2024
73f67f8
Merge branch 'main' into feat/platform-settings
plyr4 Apr 17, 2024
763f107
feat: schedule allowlist and more linting
plyr4 Apr 17, 2024
a612efa
fix: comma missing
plyr4 Apr 17, 2024
fd7f354
fix: compiler middleware test
plyr4 Apr 17, 2024
a6ae525
feat: delete settings
plyr4 Apr 17, 2024
50b8493
fix: uncomment delete code
plyr4 Apr 17, 2024
033ff65
Merge branch 'main' into feat/platform-settings
plyr4 Apr 17, 2024
47368ab
fix: create nested api types
plyr4 Apr 17, 2024
453f645
fix: check for nils
plyr4 Apr 17, 2024
dac5053
fix: remove sanitize but still crop fields
plyr4 Apr 19, 2024
6ba421a
Merge branch 'fix/owner-token' of github.com:go-vela/server into feat…
plyr4 Apr 22, 2024
8377d66
Merge branch 'main' into feat/platform-settings
plyr4 Apr 22, 2024
9e1af91
Merge branch 'main' into feat/platform-settings
plyr4 Apr 24, 2024
dfd5e6b
chore: merge with main
plyr4 Apr 24, 2024
ce6c60e
chore: merge with main
plyr4 Apr 24, 2024
be0d672
feat: types tests and general cleanup
plyr4 Apr 29, 2024
aaf0038
feat: db integration tests
plyr4 Apr 29, 2024
dfdcec5
chore: remove unneeded api endpoints
plyr4 Apr 29, 2024
1e0b4da
feat: db tests and general cleanup
plyr4 Apr 29, 2024
b0d9bbc
chore: lint
plyr4 Apr 29, 2024
af4acdd
feat: flag for resetting settings record on startup
plyr4 Apr 29, 2024
d322449
fix: get template depth in log, not pointer
plyr4 Apr 30, 2024
392cb5b
Merge branch 'main' of github.com:go-vela/server into feat/platform-s…
plyr4 May 1, 2024
be90441
enhance: timestamps for created and updated
plyr4 May 1, 2024
8c37621
chore: lint
plyr4 May 1, 2024
9e4a327
enhance: timestamps for created and updated
plyr4 May 1, 2024
e765740
enhance: validations for timestamps
plyr4 May 1, 2024
83215da
fix: test
plyr4 May 1, 2024
0d38a59
Merge branch 'main' into feat/platform-settings
plyr4 May 3, 2024
b12c907
Merge branch 'main' into feat/platform-settings
plyr4 May 3, 2024
d7ff645
fix: return updated settings
plyr4 May 3, 2024
cd3c3f9
Merge branch 'feat/platform-settings' of github.com:go-vela/server in…
plyr4 May 3, 2024
7ab84af
fix: return updated settings
plyr4 May 3, 2024
069da63
chore: lint
plyr4 May 3, 2024
9b47082
Merge branch 'main' into feat/platform-settings
plyr4 May 7, 2024
f5effff
enhance: move to db types, fix tests
plyr4 May 7, 2024
20f5691
fix: add jitter around initialization
plyr4 May 7, 2024
7bd45ca
fix: import order
plyr4 May 7, 2024
83f9380
fix: comment and jitter time
plyr4 May 7, 2024
8a25c07
Merge branch 'main' into feat/platform-settings
plyr4 May 7, 2024
7bcf015
chore: merge with main
plyr4 May 8, 2024
5a7fda9
fix: use refresh interval to avoid db spam
plyr4 May 8, 2024
b358b72
enhance: background go func that refreshes settings on an interval, c…
plyr4 May 8, 2024
297014f
chore: linter
plyr4 May 8, 2024
dee4488
Merge branch 'main' into feat/platform-settings
plyr4 May 8, 2024
c53ddbe
chore: typo
plyr4 May 8, 2024
4110ac2
chore: typos and missing test
plyr4 May 8, 2024
409a267
fix: typos, swagger, whitespace, 404s
plyr4 May 9, 2024
81fa5dd
fix: consistent variable naming and whitespace
plyr4 May 9, 2024
17a42d9
chore(queue): rename channels to routes
plyr4 May 9, 2024
eb3417b
fix: lint naming
plyr4 May 9, 2024
a56ea84
enhance: check record not found error instead of nil
plyr4 May 9, 2024
197e42f
enhance: add docker image validation
plyr4 May 10, 2024
1f8ef15
fix: typo
plyr4 May 10, 2024
da472a4
fix: lint fmt
plyr4 May 10, 2024
3a627ef
fix: test case flags
plyr4 May 10, 2024
f120029
fix: test case lint
plyr4 May 10, 2024
a43546e
fix: clone image compiler test
plyr4 May 10, 2024
4cad3e6
fix: dont update shared pointer
plyr4 May 10, 2024
b6e2672
fix: set ID
plyr4 May 10, 2024
e77df13
Merge branch 'main' into feat/platform-settings
plyr4 May 10, 2024
f54aa67
fix: test merge
plyr4 May 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 148 additions & 0 deletions api/admin/settings.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
// SPDX-License-Identifier: Apache-2.0

//nolint:dupl // ignore similar code

Check failure on line 3 in api/admin/settings.go

View workflow job for this annotation

GitHub Actions / golangci

[golangci] api/admin/settings.go#L3

directive `//nolint:dupl // ignore similar code` is unused for linter "dupl" (nolintlint)
Raw output
api/admin/settings.go:3:1: directive `//nolint:dupl // ignore similar code` is unused for linter "dupl" (nolintlint)
//nolint:dupl // ignore similar code
^
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
package admin

import (
"fmt"
"net/http"
"strings"

"github.com/gin-gonic/gin"
settings "github.com/go-vela/server/api/types/settings"
"github.com/go-vela/server/database"
sMiddleware "github.com/go-vela/server/router/middleware/settings"
"github.com/go-vela/server/util"
"github.com/sirupsen/logrus"
)

// swagger:operation GET /api/v1/admin/settings admin GetSettings
//
// Get the currently configured settings.
//
// ---
// produces:
// - application/json
// security:
// - ApiKeyAuth: []
// responses:
// '200':
// description: Successfully retrieved settings from the database
// schema:
// type: array
// items:
// "$ref": "#/definitions/Settings"
// '500':
// description: Unable to retrieve settings from the database
// schema:
// "$ref": "#/definitions/Error"

// GetSettings represents the API handler to
// captures settings stored in the database.
func GetSettings(c *gin.Context) {
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
// capture middleware values
s := sMiddleware.FromContext(c)

logrus.Info("Admin: reading settings")

output := strings.ToLower(c.Query("output"))

switch output {
case "env":
exported := s.ToEnv()
c.String(http.StatusOK, exported)
case "yaml":
exported := s.ToYAML()
c.String(http.StatusOK, exported)
case "json":
fallthrough
default:
c.JSON(http.StatusOK, s)
}
}

// swagger:operation PUT /api/v1/admin/settings admin AdminUpdateSettings
//
// Update the settings singleton in the database
//
// ---
// produces:
// - application/json
// parameters:
// - in: body
// name: body
// description: Payload containing settings to update
// required: true
// schema:
// "$ref": "#/definitions/Settings"
// security:
// - ApiKeyAuth: []
// responses:
// '200':
// description: Successfully updated the settings in the database
// schema:
// "$ref": "#/definitions/Settings"
// '404':
// description: Unable to update the settings in the database
// schema:
// "$ref": "#/definitions/Error"
// '501':
// description: Unable to update the settings in the database
// schema:
// "$ref": "#/definitions/Error"

// UpdateSettings represents the API handler to
// update the settings singleton stored in the database.
func UpdateSettings(c *gin.Context) {
// capture middleware values
s := sMiddleware.FromContext(c)
ctx := c.Request.Context()

logrus.Info("Admin: updating settings")

// capture body from API request
input := new(settings.Platform)

err := c.Bind(input)
if err != nil {
retErr := fmt.Errorf("unable to decode JSON for settings: %w", err)

util.HandleError(c, http.StatusBadRequest, retErr)
plyr4 marked this conversation as resolved.
Show resolved Hide resolved

return
}

if input.CloneImage != nil {
s.CloneImage = input.CloneImage
}

if input.TemplateDepth != nil {
s.TemplateDepth = input.TemplateDepth
}

if input.StarlarkExecLimit != nil {
s.StarlarkExecLimit = input.StarlarkExecLimit
}

if input.Routes != nil {
// update routes if set
s.SetRoutes(input.GetRoutes())
}

if input.RepoAllowlist != nil {
// update allowlist if set
s.SetRepoAllowlist(input.GetRepoAllowlist())
}

// send API call to update the repo
s, err = database.FromContext(c).UpdateSettings(ctx, s)
if err != nil {
retErr := fmt.Errorf("unable to update settings: %w", err)

util.HandleError(c, http.StatusInternalServerError, retErr)
plyr4 marked this conversation as resolved.
Show resolved Hide resolved

return
}

c.JSON(http.StatusOK, s)
}
95 changes: 95 additions & 0 deletions api/types/settings/compiler.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
// SPDX-License-Identifier: Apache-2.0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [golangci] reported by reviewdog 🐶
: # github.com/go-vela/server/api/types/settings [github.com/go-vela/server/api/types/settings.test]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what now? lol

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

golangci had an absolute meltdown yesterday, randomly. 50+ comments of nonsense


package settings

type Compiler struct {
CloneImage *string `json:"clone_image,omitempty"`
TemplateDepth *int `json:"template_depth,omitempty"`
StarlarkExecLimit *uint64 `json:"starlark_exec_limit,omitempty"`
}

// GetCloneImage returns the CloneImage field.
//
// When the provided CompilerSettings type is nil, or the field within
// the type is nil, it returns the zero value for the field.
func (cs *Compiler) GetCloneImage() string {
// return zero value if Settings type or CloneImage field is nil
if cs == nil || cs.CloneImage == nil {
return ""
}

return *cs.CloneImage
}

// GetTemplateDepth returns the TemplateDepth field.
//
// When the provided CompilerSettings type is nil, or the field within
// the type is nil, it returns the zero value for the field.
func (cs *Compiler) GetTemplateDepth() int {
// return zero value if Settings type or TemplateDepth field is nil
if cs == nil || cs.TemplateDepth == nil {
return 0
}

return *cs.TemplateDepth
}

// GetStarlarkExecLimit returns the StarlarkExecLimit field.
//
// When the provided CompilerSettings type is nil, or the field within
// the type is nil, it returns the zero value for the field.
func (cs *Compiler) GetStarlarkExecLimit() uint64 {
// return zero value if Settings type or StarlarkExecLimit field is nil
if cs == nil || cs.StarlarkExecLimit == nil {
return 0
}

return *cs.StarlarkExecLimit
}

// SetCloneImage sets the CloneImage field.
//
// When the provided CompilerSettings type is nil, it
// will set nothing and immediately return.
func (cs *Compiler) SetCloneImage(v string) {
// return if Settings type is nil
if cs == nil {
return
}

cs.CloneImage = &v
}

// SetTemplateDepth sets the TemplateDepth field.
//
// When the provided CompilerSettings type is nil, it
// will set nothing and immediately return.
func (cs *Compiler) SetTemplateDepth(v int) {
// return if Settings type is nil
if cs == nil {
return
}

cs.TemplateDepth = &v
}

// SetStarlarkExecLimit sets the StarlarkExecLimit field.
//
// When the provided CompilerSettings type is nil, it
// will set nothing and immediately return.
func (cs *Compiler) SetStarlarkExecLimit(v uint64) {
// return if Settings type is nil
if cs == nil {
return
}

cs.StarlarkExecLimit = &v
}

func CompilerMockEmpty() Compiler {
cs := Compiler{}
cs.SetCloneImage("")
cs.SetTemplateDepth(0)
cs.SetStarlarkExecLimit(0)
return cs

Check failure on line 94 in api/types/settings/compiler.go

View workflow job for this annotation

GitHub Actions / golangci

[golangci] api/types/settings/compiler.go#L94

return statements should not be cuddled if block has more than two lines (wsl)
Raw output
api/types/settings/compiler.go:94:2: return statements should not be cuddled if block has more than two lines (wsl)
	return cs
	^
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
plyr4 marked this conversation as resolved.
Show resolved Hide resolved
}
133 changes: 133 additions & 0 deletions api/types/settings/platform.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
// SPDX-License-Identifier: Apache-2.0

package settings

import (
"fmt"
)

// Platform is the API representation of platform settings.
//
// swagger:model Platform
type Platform struct {
ID *int64 `json:"id,omitempty"`

*Queue `json:"queue,omitempty"`
*Compiler `json:"compiler,omitempty"`

// misc
RepoAllowlist *[]string `json:"repo_allowlist,omitempty"`
}

// GetID returns the ID field.
//
// When the provided Settings type is nil, or the field within
// the type is nil, it returns the zero value for the field.
func (s *Platform) GetID() int64 {
// return zero value if Settings type or ID field is nil
if s == nil || s.ID == nil {
return 0
}

return *s.ID
}

// SetID sets the ID field.
//
// When the provided Settings type is nil, it
// will set nothing and immediately return.
func (s *Platform) SetID(v int64) {
// return if Settings type is nil
if s == nil {
return
}

s.ID = &v
}

// SetCompilerSettings sets the CompilerSettings field.
//
// When the provided CompilerSettings type is nil, it
// will set nothing and immediately return.
func (s *Platform) SetCompilerSettings(cs Compiler) {
// return if Settings type is nil
if s == nil {
return
}

s.Compiler = &cs
}

// SetQueueSettings sets the QueueSettings field.
//
// When the provided QueueSettings type is nil, it
// will set nothing and immediately return.
func (s *Platform) SetQueueSettings(qs Queue) {
// return if Settings type is nil
if s == nil {
return
}

s.Queue = &qs
}

// GetRepoAllowlist returns the RepoAllowlist field.
//
// When the provided Settings type is nil, or the field within
// the type is nil, it returns the zero value for the field.
func (s *Platform) GetRepoAllowlist() []string {
// return zero value if Settings type or RepoAllowlist field is nil
if s == nil || s.RepoAllowlist == nil {
return []string{}
}

return *s.RepoAllowlist
}

// SetRepoAllowlist sets the RepoAllowlist field.
//
// When the provided Settings type is nil, it
// will set nothing and immediately return.
func (s *Platform) SetRepoAllowlist(v []string) {
// return if Settings type is nil
if s == nil {
return
}

s.RepoAllowlist = &v
}

// String implements the Stringer interface for the Settings type.
func (s *Platform) String() string {
return fmt.Sprintf(`{
ID: %d,
CloneImage: %s,
QueueRoutes: %v,
other stuff: %v,
}`,
s.GetID(),
s.GetCloneImage(),
s.GetRoutes(),
s.GetRoutes(),
)
}

// ToEnv converts the Settings type to a string format compatible with standard posix environments.
func (s *Platform) ToEnv() string {
return fmt.Sprintf(`VELA_CLONE_IMAGE='%s'
VELA_QUEUE_ROUTES='%v'
`,
s.GetCloneImage(),
s.GetRoutes(),
)
}

// ToYAML converts the Settings type to a YAML string.
func (s *Platform) ToYAML() string {
return fmt.Sprintf(`VELA_CLONE_IMAGE: '%s'
VELA_QUEUE_ROUTES: '%s'
`,
s.GetCloneImage(),
s.GetRoutes(),
)
}
Loading
Loading