Merge pull request #456 from bottlerocket-os/main

Move to Prod: ECS Updater DOcs
bottlerocket-os · Mar 19, 2024 · 15ca306 · 15ca306
2 parents 7fa1a32 + bf809dd
commit 15ca306
Show file tree

Hide file tree

Showing 19 changed files with 715 additions and 6 deletions.
diff --git a/assets/scss/_styles_project.scss b/assets/scss/_styles_project.scss
@@ -517,14 +517,60 @@ nav.foldable-nav .with-child.depad {
         }
     }
 }
+.ecs-updater-diagram {
+    text {
+        text-anchor: middle;
+        font-size: 12px;
+    }
+    .aws-service rect,
+    .fargate-task rect {
+        rx: 9; 
+    }
+    .fargate-task {
+        rect {
+            fill: $tan;
+        }
+        text {
+            fill: $dark-blue;
+        }
+    }
+    .aws-service {
+        rect {
+            fill: $dark-orange;           
+        }
+        text {
+
+            fill: $white;
+        }
+    }
+    .shared-responsibility-boundary {
+        fill: none;
+        stroke: $dark;
+        stroke-width: 2px;
+        stroke-miterlimit: 10;
+        stroke-dasharray: 2 6;
+    }
+
+    .line-arrow-connector {
+        .mid-label,
+        .label {
+            text-anchor: middle;
+        }
 
+        .mid-label {
+            font-size: 11px;
+            font-weight: 100;
+        }
+    }
+}
+
+.ecs-updater-diagram,
 .brupop-diagram,
 .brupop-state-machine {
     .line-arrow-connector {
         stroke-miterlimit: 10;
         stroke: $dark-blue;
         .connector {
-
             fill: none;
             pointer-events : stroke;
             &.dotted {
@@ -537,6 +583,7 @@ nav.foldable-nav .with-child.depad {
 
     }
 
+    .mid-label,
     .label {
         font-size: 12px;
         font-family: $td-fonts-serif;
@@ -562,21 +609,27 @@ nav.foldable-nav .with-child.depad {
     }
 }
 
-.brupop-diagram {
+.brupop-diagram,
+.ecs-updater-diagram {
+    .aws-service .rect,
     .node,
     .agent,
     .api-server,
     .controller,
     .unused-container,
     .unused-volume,
     .active-volume,
+    .control-container, 
     .line-arrow-connector .arrow-head,
     .ellipses,
     .future-volume,
     .label-backer,
     .wait  {
         pointer-events: all;
     }
+}
+.ecs-updater-diagram,
+.brupop-diagram {
     .node {
         fill: $tan;
         rx: 3;
@@ -587,7 +640,8 @@ nav.foldable-nav .with-child.depad {
     .unused-container,
     .unused-volume,
     .active-volume,
-    .future-volume {
+    .future-volume,
+    .control-container {
         rx: 2;
     }
     .agent {
@@ -596,7 +650,8 @@ nav.foldable-nav .with-child.depad {
 
     .label-backer,
     .unused-container,
-    .unused-volume {
+    .unused-volume,
+    .control-container {
         fill: $white;
     }
 
@@ -614,6 +669,9 @@ nav.foldable-nav .with-child.depad {
         fill: none;
     }
 
+    .control-container {
+        fill: $light-blue;
+    }
     .api-server {
         fill: $dark-orange;
     }

diff --git a/content/en/ecs-updater/0.2.x/_index.markdown b/content/en/ecs-updater/0.2.x/_index.markdown
@@ -0,0 +1,4 @@
++++
+type="docs"
+title="0.2.x (Current)"
++++
diff --git a/content/en/ecs-updater/0.2.x/concepts/index.markdown b/content/en/ecs-updater/0.2.x/concepts/index.markdown
@@ -0,0 +1,55 @@
++++
+title = "Concepts"
+type = "docs"
+description = "Introduction to the components and concepts used in the Bottlerocket ECS Updater" 
+weight = 1
++++
+
+You can update Bottlerocket in a couple of ways:
+
+* **node replacement** where new instances with a new version of the OS replace nodes with older versions of the OS,
+* **in-place updates** where the node downloads and reboots into a new version of the OS while maintaining the same instance/machine.
+
+There is no single preferred nor advised method to update a node; both methods have pros and cons depending on your situation.
+You can trigger an {{< cross-project-current-link project="os" url="/en/os/x.x.x/update/methods/in-place/#apiclient-commands">}}in-place update manually with the API{{< /cross-project-current-link >}} or you can use the Bottlerocket ECS Updater.
+**The ECS Updater is a service that checks for (and applies) Bottlerocket updates when your nodes are ready.**
+
+If you use Bottlerocket on Kubernetes (see [Brupop](../../brupop/)) or intend to replace nodes in [Amazon ECS](https://aws.amazon.com/ecs/), ECS Updater is not for you.
+If you plan to use [EC2 Spot in your ECS cluster](../troubleshoot/#spot-instances-never-update-to-a-new-version-of-bottlerocket), ECS Updater is not for you.
+Even if you do plan to do in-place updates, you don’t need to use ECS Updater as you can manage in-place updates in other ways.
+However, ECS updater offers an automated way to manage in-place Bottlerocket updates.
+
+## Overview
+
+The ECS Updater periodically evaluates data from the ECS control plane API and information from each Bottlerocket node’s API server to determine what nodes need an update *and* which nodes are eligible for replacement.
+The upgradable, eligible nodes drain, reboot into the latest update, and finally reactivate.
+
+## Automated updates
+
+The ECS Updater task runs from [AWS Fargate](https://aws.amazon.com/fargate/), triggered periodically by [Amazon EventBridge](https://aws.amazon.com/eventbridge/).
+The updater task first determines the Bottlerocket nodes in a given cluster by querying the ECS API.
+
+{{< ecs-updater/eventbridge-api-task >}}
+
+Next, the ECS Updater runs an [AWS Systems Manager](https://aws.amazon.com/systems-manager/) [(SSM) Document](https://docs.aws.amazon.com/systems-manager/latest/userguide/documents.html) on each Bottlerocket node against the control container.
+This document invokes the API Client to check for updates.
+With the list of nodes that have available updates, the ECS Updater task queries the ECS API for information on the type of workloads running on the node.
+If a node is running one or more non-service ([standalone](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/standalone-tasks.html) or [scheduled](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html)) tasks, the node is deemed not updateable.
+
+{{< ecs-updater/updateable >}}
+
+ECS Updater uses the ECS API to initiate node draining and checks the number of running tasks on the node until none remain.
+
+{{< ecs-updater/node-3-ready >}}
+
+Once the node has no remaining tasks, the ECS Updater runs an SSM Document against the control container to initiate the update through the {{< cross-project-current-link project="os" url="/en/os/x.x.x/concepts/api-driven/">}}Bottlerocket API{{</ cross-project-current-link >}}.
+
+Effectively, this is the same as running  `apiclient update apply --reboot`  from an interactive SSM session with the control container.
+After the reboot, the node will be running the newer version of Bottlerocket.
+
+{{< ecs-updater/drain-ssm >}}
+
+Once confirmed to be running with the new version, the ECS Updater uses the ECS API to update the node status to active.
+At this point ECS will resume assigning tasks to the updated node.
+
+{{< ecs-updater/node-ready-for-tasks >}}
diff --git a/content/en/ecs-updater/0.2.x/operate/index.markdown b/content/en/ecs-updater/0.2.x/operate/index.markdown
@@ -0,0 +1,48 @@
++++
+title = "Operate & Observe"
+type = "docs"
+description = "Understanding the day-to-day use of ECS Updater" 
+weight = 20
++++
+
+By default, the ECS Updater runs every 12 hours (using the [default template](https://github.com/bottlerocket-os/bottlerocket-ecs-updater/blob/develop/stacks/bottlerocket-ecs-updater.yaml), schedule defined at at `.Resources.BottlerocketUpdaterSchedule.Properties.ScheduleExpression`).
+Each time it runs, the ECS Updater creates a new log stream in Amazon CloudWatch Logs in the log group you specified when deploying.
+You can view the logs in the [CloudWatch Logs Console](https://console.aws.amazon.com/cloudwatch/), with your own tools, or in the console.
+
+## ECS Updater logs in the console
+
+First, you’ll need to [retrieve the log stream](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/describe-log-streams.html); likely you’re interested in the most recent update check.
+Using the environment variables established in the Setup documentation, run the following command to get the most recent stream and store it in an environment variable:
+
+```shell
+export LAST_ECS_UPDATER_LOG_STREAM=$(aws logs describe-log-streams \
+   --log-group-name ${ECS_UPDATER_LOG_GROUP} \
+   --order-by LastEventTime \
+   --query "logStreams[0].logStreamName" \
+   --descending --no-cli-pager --output text --no-paginate)
+```
+
+Then confirm that the results look correct by running `echo $LAST_ECS_UPDATER_LOG_STREAM` , this should return something like `/ecs/bottlerocket-updater/some-ecs-cluster/BottlerocketEcsUpdaterService/0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a`
+
+Once you have the `LAST_ECS_UPDATER_LOG_STREAM` environment variable, you can [get the events](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/get-log-events.html):
+
+```shell
+aws logs get-log-events --log-group-name ${ECS_UPDATER_LOG_GROUP} \
+   --log-stream-name ${LAST_ECS_UPDATER_LOG_STREAM} \
+   --query "events[*].message" \
+   --no-cli-pager --output table
+```
+
+This returns a table formatted list of the events, starting with the oldest and moving to the most recent.
+
+A few notable messages and meanings:
+
+* `Filtering container instances running Bottlerocket OS` and subsequently the message `Bottlerocket instance "i-<instance>" detected` .
+You should see a match for all the Bottlerocket nodes in your ECS cluster.
+* ECS Updater emits the message  `Filtering instances with available updates` when evaluating both the availability of of updates as well as the ability to drain the node.
+* `Sending SSM document "bottlerocket-ecs-updater-UpdateApplyCommand...`  signals the start of ECS Updater directing the control container to apply the update.
+* After finishing the update, the ECS Updater commands the control container to reboot, being logged as `Sending SSM document "bottlerocket-ecs-updater-RebootCommand...`
+* Following the reboot, ECS Updater awaits the successful reboot and emits  `Waiting for instance "i-<instance>" to reach Ok status`
+* Once reaching active, ECS Updater emits  `Container instance "arn:aws:ecs:<region>:<instance>:container-instance/<cluster>/<...>" state changed to ACTIVE successfully!`
+* Before ensuring that the update happened correctly, ECS Updater logs `Verifying update by checking there is no new version available to update and validate the active version`
+* In the scope of the frequency at which ECS Updater runs compared to the release cadence, most log streams end with “`No instances to update`”