WARNING: Be wary about upgrading AWS SDK to v1.25.38 or Higher, Introduces Breaking Changes in Certain Conditions

[ekristen]

Certain Conditions: Not running SOPS on the EC2 host (ie docker, containerd, kubernetes, or anywhere there's an extra network hop) and using ec2 metadata to authenticate and disabl

After spending FAR too much time (days over a couple of weeks) debugging a MASSIVE lag in using SOPS programmatically with Flux in AWS it all comes down to AWS making a BREAKING change in their SDK and the EC2 Metadata Service and the SDK taking minutes to timeout all the new EC2 Metadata calls.

Starting in v1.25.38 and later they turned on a new authentication method for EC2 Metadata that you CANNOT disable (see aws/aws-sdk-go#2980) and it's unlikely they are going to support disabling based on some comments.

This really only applies to people wanting to use SOPS with AWS KMS and using something like Instance Profiles to do the authentication.

For example I am using flux to sync secrets to kubernetes clusters and I'm using SOPS to decrypt them, Flux has a newer version of AWS SDK that has this new change whereas SOPS itself does not.

More details: fluxcd/flux#3186 (comment)

Hopefully someone sees this and it saves them some time or at least allows devs to go into an SDK upgrade with eyes wide open.

[autrilla]

Thanks for letting everyone know!