Skip to content
This repository has been archived by the owner on Jan 26, 2023. It is now read-only.

Make attache-control a long running sidecar #24

Merged
merged 8 commits into from
Feb 4, 2022

Conversation

beautifulentropy
Copy link
Member

@beautifulentropy beautifulentropy commented Jan 24, 2022

Previous to this change, we've avoided hashicorp/nomad#10058 by adding a long
sleep before attache-control would otherwise exit successfully. Unfortunately
this still means that subsequent scaling deployments will fail if the sleep
incurred from the initial deployment has expired.

This change makes attache-control a proper sidecar (long running ephemeral
post-start task). Instead of exiting after a successful run, attache-control
will stop the ticker channel and continue running until it receives a kill
signal.

  • Configure attache-control as a side-car in the example nomad job
  • Remove attempt-limit flag and cliOpts.attemptLimit
  • Split the main func of attache-control into helpers
  • Add helper methods to scalingOpts
  • Replace calls to time.Tick with time.NewTicker
  • Add renewChan (chan struct{}) and acquired (bool) to the lock struct
  • lock.Acquire() sets lock.acquired to true, initializes lock.renewChan, and calls
    lock.periodicallyRenew() in a long-running go-routine
  • lock.Cleanup() closes lock.renewChan and sets lock.acquired to false

Fixes #6
Part Of #11

@beautifulentropy
Copy link
Member Author

@pgporada just a heads up on this. There will need to be some changes to the redis-cluster nomad job:

  • Set sidecar = true in the lifecycle stanza of the attache-control task
  • Remove flag attempt-limit from attache-control if you currently set one

@beautifulentropy
Copy link
Member Author

Added a basic repro of the subsequent deployment failures to hashicorp/nomad#10058:
hashicorp/nomad#10058 (comment)

- Move lock.Cleanup() deferral to post-acquisition attempt
- Add ticker.Stop() deferral before shard slot rebalance loop
- Remove un-necessary Continue from attache-control main
@beautifulentropy beautifulentropy merged commit 6e528e5 into main Feb 4, 2022
@beautifulentropy beautifulentropy deleted the un-nest-control-main branch February 4, 2022 23:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cleanup the nested loops in the main func of attache-control
2 participants