Error removing Image docker.domain.com/myapp:1.1 Error: No such image

[sabretus]

Summary

ECS agent throws following errors in logs:

Feb 4 09:45:28 ecs-10-0-4-182.dev.domain.com ecs-agent ERROR Error removing Image docker.domain.com/myapp:1.1 - Error: No such image: docker.domain.com/myapp:1.1

Description

This happens for different image tagss with old versions which are outdated and doesn't exist on instance. For example current version of image tag is 1.5, but ECS agent still tries to delete 1.1 and fails, because it doesn't exist.

From all it seems that it doesn't affect anything, just weird errors in logs.

Expected Behavior

Do not try to delete image tag, which already doesn't exist.

Observed Behavior

Trying to delete image tag which doesn't exist. Just for testing purpose I have tagged generic nginx image with tag of my app, tag was deleted and errors gone for that image, however later they appear again for other tags.

Environment Details

Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1025-aws
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.91GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
Product License: Community Engine

[rhuddleston]

Looks like this started happening sometime after the 17.3 release

[sharanyad]

@sabretus ECS Agent will keep a record of all the images used by the tasks and removes them periodically. if image docker.domain.com/myapp:1.1 was pulled by agent, it will try to clean it up. Was this image removed manually from the instance or do you have other cleanup scripts running?

[rhuddleston]

Yes we have other cleanup scripts that can run but those scripts haven't changed in years. In version 17.3 and before it would throw one error and that was it. Now in newer versions it throws an error every 30 minutes and never gives up trying to remove and image that will never exist again.

[shubham2892]

We have a PR #1897 out to fix the issue.

[shubham2892]

The PR is merged and should be out in Agent version 1.26.0. Please reopen if you see the issue again after the release.