Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: dynamic file loader #96

Merged
merged 82 commits into from
Feb 7, 2024
Merged

feat: dynamic file loader #96

merged 82 commits into from
Feb 7, 2024

Conversation

lvlcn-t
Copy link
Collaborator

@lvlcn-t lvlcn-t commented Jan 31, 2024 •
edited
Loading

Motivation

To also dynamically load the runtime config locally with a file.

Closes #85

Changes

I've adjusted the file loader to run in the same interval as the http loader.

Additionally I've fixed some race conditions in the http loader tests and harmonized both loaders.

For additional information look at the commits.

Tests done

I've provided some unit tests for the file loader.

  • Unit tests succeeded.
  • E2E tests succeeded.

Manual E2E test

Logs (with debug logs):

$ go run main.go run --config .tmp/config/start-config.yaml 
Using config file: .tmp/config/start-config.yaml
{"time":"2024-01-31T16:11:21.857325647+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/cmd.NewCmdRun.run.func1","file":"/home/installadm/dev/github/sparrow/cmd/run.go","line":81},"msg":"Running sparrow"}
{"time":"2024-01-31T16:11:21.857438975+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow.(*Sparrow).api.func1","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/api.go","line":81},"msg":"Serving Api","addr":":8080"}
{"time":"2024-01-31T16:11:21.857460883+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).Reconcile","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":81},"msg":"Starting global gitlabTargetManager reconciler"}
{"time":"2024-01-31T16:11:51.872199193+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/config.(*FileLoader).Run","file":"/home/installadm/dev/github/sparrow/pkg/config/file.go","line":79},"msg":"Successfully got local runtime configuration"}
{"time":"2024-01-31T16:11:51.872274206+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).Startup","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":121},"msg":"Initializing health check"}
{"time":"2024-01-31T16:11:51.872306316+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).Run","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":93},"msg":"Starting healthcheck","interval":"20s"}
{"time":"2024-01-31T16:12:11.872464888+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":194},"msg":"Checking health"}
{"time":"2024-01-31T16:12:11.872526771+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":199},"msg":"Getting health status for each target in separate routine","amount":2}
{"time":"2024-01-31T16:12:11.872569466+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":236},"msg":"Waiting for all routines to finish"}
{"time":"2024-01-31T16:12:11.872594262+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":221},"msg":"Starting retry routine to get health status","target":"https://www.google.com"}
{"time":"2024-01-31T16:12:11.8726504+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":221},"msg":"Starting retry routine to get health status","target":"https://www.example.com"}
{"time":"2024-01-31T16:12:12.006130511+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":227},"msg":"Successfully got health status of target","target":"https://www.google.com","status":"healthy"}
{"time":"2024-01-31T16:12:12.287731427+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":227},"msg":"Successfully got health status of target","target":"https://www.example.com","status":"healthy"}
{"time":"2024-01-31T16:12:12.287776716+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":239},"msg":"Successfully got health status from all targets"}
{"time":"2024-01-31T16:12:12.287801659+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).Run","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":113},"msg":"Successfully finished health check run"}
{"time":"2024-01-31T16:12:21.863085495+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).fetchFileList","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":203},"msg":"Fetching file list from gitlab"}
{"time":"2024-01-31T16:12:21.873305744+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/config.(*FileLoader).Run","file":"/home/installadm/dev/github/sparrow/pkg/config/file.go","line":79},"msg":"Successfully got local runtime configuration"}
{"time":"2024-01-31T16:12:21.948094557+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).fetchFileList","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":253},"msg":"Successfully fetched file list","files":2}
{"time":"2024-01-31T16:12:22.022215896+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).fetchFile","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":195},"msg":"Successfully fetched file","file":"dev-tom.local.sparrow.com.json"}
{"time":"2024-01-31T16:12:22.105700518+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).fetchFile","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":195},"msg":"Successfully fetched file","file":"tom.local.sparrow.de.json"}
{"time":"2024-01-31T16:12:22.10572891+01:00","level":"INFO","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).FetchFiles","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":149},"msg":"Successfully fetched all target files","files":2}
{"time":"2024-01-31T16:12:22.105741206+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).refreshTargets","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":215},"msg":"Skipping unhealthy target","target":{"url":"https://dev-tom.local.sparrow.com","lastSeen":"2024-01-26T18:03:45.46397814Z"}}
{"time":"2024-01-31T16:12:22.105764532+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).refreshTargets","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":211},"msg":"Found self as global target","lastSeenMin":1.6363695812333332}
{"time":"2024-01-31T16:12:22.105776707+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).refreshTargets","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":222},"msg":"Updated global targets","targets":1}
{"time":"2024-01-31T16:12:22.105793738+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).updateRegistration","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":160},"msg":"Updating registration"}
{"time":"2024-01-31T16:12:22.105814393+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/gitlab.(*Client).PutFile","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/gitlab/gitlab.go","line":261},"msg":"Registering sparrow instance to gitlab"}
{"time":"2024-01-31T16:12:22.4599661+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/sparrow/targets.(*gitlabTargetManager).updateRegistration","file":"/home/installadm/dev/github/sparrow/pkg/sparrow/targets/gitlab.go","line":179},"msg":"Successfully updated registration"}
{"time":"2024-01-31T16:12:32.288176868+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":194},"msg":"Checking health"}
{"time":"2024-01-31T16:12:32.288229653+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":199},"msg":"Getting health status for each target in separate routine","amount":3}
{"time":"2024-01-31T16:12:32.288256339+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":236},"msg":"Waiting for all routines to finish"}
{"time":"2024-01-31T16:12:32.288267021+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":221},"msg":"Starting retry routine to get health status","target":"https://www.telekom.de"}
{"time":"2024-01-31T16:12:32.288295019+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":221},"msg":"Starting retry routine to get health status","target":"https://www.example.com"}
{"time":"2024-01-31T16:12:32.288325164+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":221},"msg":"Starting retry routine to get health status","target":"https://www.google.com"}
{"time":"2024-01-31T16:12:32.351792774+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":227},"msg":"Successfully got health status of target","target":"https://www.google.com","status":"healthy"}
{"time":"2024-01-31T16:12:32.355092395+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":227},"msg":"Successfully got health status of target","target":"https://www.telekom.de","status":"healthy"}
{"time":"2024-01-31T16:12:32.391033677+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check.func2","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":227},"msg":"Successfully got health status of target","target":"https://www.example.com","status":"healthy"}
{"time":"2024-01-31T16:12:32.391116899+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).check","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":239},"msg":"Successfully got health status from all targets"}
{"time":"2024-01-31T16:12:32.391135177+01:00","level":"DEBUG","source":{"function":"github.com/caas-team/sparrow/pkg/checks/health.(*Health).Run","file":"/home/installadm/dev/github/sparrow/pkg/checks/health/health.go","line":113},"msg":"Successfully finished health check run"}

TODO

  • I've assigned this PR to myself
  • I've labeled this PR correctly

niklastreml and others added 30 commits January 16, 2024 18:20
@niklastreml
chore: updated helm chart version to 0.0.4
6f759d9
@niklastreml
chore: bump appVersion to v0.3.0
820cd55
@lvlcn-t
fix: incorrect formatting on empty labels
da317cc
@puffitos
refactor: remove any configurations
02e06b5 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
chore: removed decode (RIP)
078de7c 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@lvlcn-t
fix: newline indent
975e77f
@puffitos
chore: merged from main
4a2d58f 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
refactor: restructure checks pkg
68d066a 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@niklastreml
fix: remove default runtime config in helm chart
761d7c4
@niklastreml
feat: simplify helm chart
2fa7fb2
@niklastreml
fix: remove -n
b9abd15
@niklastreml
fix: register healthcheck for e2e test
8bb3f07
@niklastreml
feat: separate checks and sparrow config
c8c764a
@niklastreml
fix: e2e tests
8f0d564
@niklastreml
fix: bug where env variables would not be read correctly
551f07c
@niklastreml
feat: allow mounting extra secrets as env variables
d81ad4a
@puffitos
Merge remote-tracking branch 'origin/feat/simplify-helm-chart-config'…
3f40604 
… into refactor/typed-runtime-cfg
@puffitos
Merge remote-tracking branch 'origin' into refactor/typed-runtime-cfg
7fa28db
@niklastreml
Merge branch 'main' into chore/bump-chart-version
fc72395
@niklastreml
Merge branch 'chore/bump-chart-version' into feat/simplify-helm-chart…
2a1cb4c 
…-config
@puffitos
chore: added helm docs
0bd9674 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
Merge branch 'feat/simplify-helm-chart-config' of github.com:caas-tea…
c6f0ca7 
…m/sparrow into feat/simplify-helm-chart-config
@niklastreml
docs: envfromSecrets
0c4042f
@puffitos
chore: merge from main
9339cdd 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
Merge branch 'feat/simplify-helm-chart-config' into refactor/typed-ru…
90988e7 
…ntime-cfg
@puffitos
fix: checks are properly configured on start
3d0971f 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
fix: target manager nil check when enriching
2f6333b 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
chore: merge from main
eb5fc0a 
Signed-off-by: Bruno Bressi <bruno.bressi@telekom.de>
@puffitos
Merge remote-tracking branch 'origin/main' into refactor/typed-runtim…
2742590 
…e-cfg
@lvlcn-t
feat: add dns check
5cc774f
@lvlcn-t lvlcn-t added the feature Introduces a new feature label Jan 31, 2024
@lvlcn-t lvlcn-t self-assigned this Jan 31, 2024
@lvlcn-t lvlcn-t force-pushed the feat/dynamic-file-loader branch from 72e6a60 to 3ea387e Compare January 31, 2024 15:28
puffitos
puffitos requested changes Feb 2, 2024
View reviewed changes
Copy link
Collaborator

@puffitos puffitos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must still review the tests.

pkg/config/http.go Outdated Show resolved Hide resolved
pkg/config/file.go Show resolved Hide resolved
pkg/config/file.go Outdated Show resolved Hide resolved
pkg/config/file.go Outdated Show resolved Hide resolved
@lvlcn-t
chore: PR review
5b74e0f 
* chore: adjust godoc
* chore: log with path
* fix: handle file close error
* test: add unit test for file close error
* feat: validate file path config
* test: add file path config validation tests
@lvlcn-t
chore: licensing
a913ce9
@lvlcn-t
fix: validate only empty string
48b04d2
puffitos
puffitos reviewed Feb 3, 2024
View reviewed changes
Copy link
Collaborator

@puffitos puffitos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, everything seems fine by me. I would additionally comment the methods of the test package which mock the fs.File interface.

pkg/config/file_test.go Outdated Show resolved Hide resolved
pkg/config/file.go Outdated Show resolved Hide resolved
@lvlcn-t lvlcn-t merged commit 577d5c7 into main Feb 7, 2024
11 checks passed
@lvlcn-t lvlcn-t deleted the feat/dynamic-file-loader branch February 7, 2024 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@niklastreml niklastreml niklastreml left review comments

@puffitos puffitos puffitos approved these changes

@y-eight y-eight Awaiting requested review from y-eight y-eight is a code owner

@eumel8 eumel8 Awaiting requested review from eumel8

Assignees

@lvlcn-t lvlcn-t

Labels
feature Introduces a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature: File Loader should also dynamically load files
3 participants
@lvlcn-t @niklastreml @puffitos