Ensure amazon-ssm-agent is up-to-date if present

[AndrewUnderwoodAtFanatics]

Description of changes:

In the original PR (#1501), the intention was to optimize the AMI build process by avoiding the reinstallation of the amazon-ssm-agent if it was already installed. The original logic was sound for its purpose—there's no need to install something that's already there. However, this introduced a regression: the script no longer ensured that the amazon-ssm-agent was updated to the latest version if it was pre-installed in the upstream AMI. The downstream effect of this change was that any AMIs built from this script inadvertently shipped with an out-of-date version of the amazon-ssm-agent, version 3.2.1705.0-1, which contains several CVEs, including ALAS-2023-2303.

This change retain the efficiency of not re-installing the agent unnecessarily while addressing the packing regression. The yum update command will pull the latest version from the repositories, ensuring that the secure version, amazon-ssm-agent 3.2.1705.0-1.amzn2, is correctly shipped as before the regression occurred.

Testing Done

On the latest AMI amazon-eks-node-1.25-v20231116 running the upgrade command for the amazon-ssm-agent correctly installs the 3.2.1705.0-1.amzn2 version of the package which has the latest security updates.

sh-4.2$ yum list | grep amazon-ssm-agent
amazon-ssm-agent.x86_64                  3.2.1705.0-1                 installed
amazon-ssm-agent.x86_64                  3.2.1705.0-1.amzn2           amzn2-core
sh-4.2$ sudo yum upgrade -y amazon-ssm-agent
Loaded plugins: priorities, update-motd, versionlock
amzn2-core                                                                                                                                                                                  | 3.6 kB  00:00:00
amzn2extra-docker                                                                                                                                                                           | 2.9 kB  00:00:00
amzn2extra-kernel-5.10                                                                                                                                                                      | 3.0 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package amazon-ssm-agent.x86_64 0:3.2.1705.0-1 will be updated
---> Package amazon-ssm-agent.x86_64 0:3.2.1705.0-1.amzn2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                               Arch                                        Version                                                   Repository                                       Size
===================================================================================================================================================================================================================
Updating:
 amazon-ssm-agent                                      x86_64                                      3.2.1705.0-1.amzn2                                        amzn2-core                                       24 M

Transaction Summary
===================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 24 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
amazon-ssm-agent-3.2.1705.0-1.amzn2.x86_64.rpm                                                                                                                                              |  24 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : amazon-ssm-agent-3.2.1705.0-1.amzn2.x86_64                                                                                                                                                      1/2
  Cleanup    : amazon-ssm-agent-3.2.1705.0-1.x86_64                                                                                                                                                            2/2
  Verifying  : amazon-ssm-agent-3.2.1705.0-1.amzn2.x86_64                                                                                                                                                      1/2
  Verifying  : amazon-ssm-agent-3.2.1705.0-1.x86_64                                                                                                                                                            2/2

Updated:
  amazon-ssm-agent.x86_64 0:3.2.1705.0-1.amzn2

Complete!
sh-4.2$ yum list | grep amazon-ssm-agent
amazon-ssm-agent.x86_64                  3.2.1705.0-1.amzn2           @amzn2-core

[cartermckinnon]

We already do a blanket yum update at the beginning of this script:

amazon-eks-ami/scripts/install-worker.sh

Line 63 in dd3e341

sudo yum update -y

So it seems strange that yum update -y amazon-ssm-agent would do something different. I'm wondering if some AL2 repodata was changed, as I didn't observe this behavior a week or so ago.

FWICT, the amzn2 binary for this SSM agent version was re-compiled with some newer go dependencies, likely similar to the changes made in the latest SSM agent version (in today's AMI release), 3.2.1798.0-1: https://github.com/aws/amazon-ssm-agent/releases/tag/3.2.1798.0

I think the proper fix is to swap our installation method back to the AL2 RPM, but I need to confirm with the SSM agent team that it will remain up to date.

[stewartsmith]

Upstream SSM agent does all sorts of releases that we don't bring into Amazon Linux. We only bring in specific versions intentionally to ensure stability. Things such as

amazon-eks-ami/scripts/install-worker.sh

Line 517 in 8b87d03

sudo yum install -y https://s3.${BINARY_BUCKET_REGION}.${S3_DOMAIN}/amazon-ssm-${BINARY_BUCKET_REGION}/${SSM_AGENT_VERSION}/linux_${ARCH}/amazon-ssm-agent.rpm

where it's grabbed out of an S3 bucket have caused teams issues in the past as it's not versioned and you can't easily roll back (like you can with yum repos where the old versions are readily available)

[AndrewUnderwoodAtFanatics]

Got it. In this case it sounds like the balance between stability and freshness may be tipped a little too far towards stability since it's causing the image to ship with CVEs. Is there something that can be done to this process to ensure the agent stays a bit more up to date?

[stevo-f3]

Seems fixed with #1531