How to recover portus deleted image tags

[gopal-gemini]

We have installed Portus using "head" version last month ( not sure the exact version number because upgraded with latest stable version Version: 2.3.5@35b9c801bf5275537a8bd8152d2f5769d6c24b09), the images got deleted automatically from portus. I see that the blobs layers are still available in the blobs folder.

I do not see the issues after upgraded with stable but How do we recover the deleted images tag and bring back it available?

last bug report
#1160

[mssola]

I do not see the issues after upgraded with stable but How do we recover the deleted images tag and bring back it available?

So the thing is that blobs weren't deleted (and you haven't run the garbage collector that you can find inside of the docker registry, right?). In this case, then the only thing you are missing is the manifest that Portus deleted. You could just push the images up again, and it should already re-use the blobs you had uploaded before the delete. This is the way to go imho.

got deleted automatically from portus

I'd need logs and the configuration to give you a better response 😅

[hggh]

I have the same problem. Portus automatically deletes image tags without any user interaction. It happens from time to time.

I'm running a clean installation of Portus 2.4 wih docker-compose.

The portus background logs that the job is now deleting a image and I see the delete call in the registry logs but no user submitted it.

On the dashboard inside the web UI there is only a message "portus deleted fooo/bar:test-201812-27"

Log from Portus background container

[catalog] Removed the tag 'test-201812-27'.
Handling 'delete' event:
{
  "id": "31cf9827-c223-4465-9742-c6786b742119",
  "timestamp": "2018-12-17T09:26:11.630596986Z",
  "action": "delete",
  "target": {
    "digest": "sha256:cdf8ba51628b9615196f90093865f72269a528c9217eb2d375f4e3ae6ace5151",
    "repository": "fooo/bar"
  },
  "request": {
    "id": "ae24028a-75a8-4b99-875b-49ab60e84556",
    "addr": "0.0.0.0",
    "host": "example.com",
    "method": "DELETE",
    "useragent": "Ruby"
  },
  "actor": {
    "name": "portus"
  },
  "source": {
    "addr": "9853afadbed0:5000",
    "instanceID": "cbb1dbc1-3d19-463b-b3b6-f88267b908a2"
  }
}

Deployment information

Configuration:

version: "2"

services:
  portus:
    image: opensuse/portus:2.4
    environment:
      - PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}

      # SSL
      - PORTUS_PUMA_TLS_KEY=/certificates/portus.key
      - PORTUS_PUMA_TLS_CERT=/certificates/portus.crt

      - PORTUS_CHECK_SSL_USAGE_ENABLED=false  

      # NGinx is serving the assets instead of Puma. If you want to change this,
      # uncomment this line.
      - RAILS_SERVE_STATIC_FILES='true'
    ports:
      - 3000:3000
    links:
      - db
    volumes:
      - ./secrets:/certificates:ro
      - static:/srv/Portus/public
    restart: unless-stopped

  background:
    image: opensuse/portus:2.4
    depends_on:
      - portus
      - db
    environment:
      # Theoretically not needed, but cconfig's been buggy on this...
      - CCONFIG_PREFIX=PORTUS
      - PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
      - PORTUS_KEY_PATH=/certificates/portus.key
      - PORTUS_PASSWORD=${PORTUS_PASSWORD}

      - PORTUS_BACKGROUND=true
      - PORTUS_BACKGROUND_SYNC_ENABLED=false
    links:
      - db
    volumes:
      - ./secrets:/certificates:ro
    restart: unless-stopped

  db:
    image: library/mariadb:10.0.23
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
    environment:
      - MYSQL_DATABASE=portus_production

      # Again, the password shouldn't be handled like this.
      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
    volumes:
      - /media/data1/hub/mariadb:/var/lib/mysql
    restart: unless-stopped

  registry:
    image: library/registry:2.6.2
    command: ["/bin/sh", "/etc/docker/registry/init"]
    environment:
      # Authentication
      REGISTRY_AUTH_TOKEN_REALM: https://${MACHINE_FQDN}/v2/token
      REGISTRY_AUTH_TOKEN_SERVICE: ${MACHINE_FQDN}
      REGISTRY_AUTH_TOKEN_ISSUER: ${MACHINE_FQDN}
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/portus.crt

      # SSL
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus.crt
      REGISTRY_HTTP_TLS_KEY: /secrets/portus.key

      # Portus endpoint
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://${MACHINE_FQDN}/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s
    volumes:
      - /media/data1/hub/registry:/var/lib/registry
      - ./secrets:/secrets:ro
      - ./registry/config.yml:/etc/docker/registry/config.yml:ro
      - ./registry/init:/etc/docker/registry/init:ro
    ports:
      - 5000:5000
      - 5001:5001 # required to access debug service
    links:
      - portus:portus
    restart: unless-stopped

  nginx:
    image: library/nginx:alpine
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./secrets:/secrets:ro
      - static:/srv/Portus/public:ro
    ports:
      - 80:80
      - 443:443
    links:
      - registry:registry
      - portus:portus
    restart: unless-stopped

volumes:
  static:
    driver: local

Portus version: 2.4.0@2.4.0

docker ps

c1a7851de984        opensuse/portus:2.4   "/init"                  3 weeks ago         Up 11 days          3000/tcp                                   hub_background_1
766f9e025274        nginx:alpine          "nginx -g 'daemon of…"   2 months ago        Up 11 days          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   hub_nginx_1
9853afadbed0        registry:2.6.2        "/entrypoint.sh /bin…"   2 months ago        Up 11 days          0.0.0.0:5000-5001->5000-5001/tcp           hub_registry_1
40f473acf0c5        opensuse/portus:2.4   "/init"                  2 months ago        Up 11 days          0.0.0.0:3000->3000/tcp                     hub_portus_1
98f61817efe6        mariadb:10.0.23       "/docker-entrypoint.…"   2 months ago        Up 11 days          3306/tcp                                   hub_db_1

[maleficxp]

Same issue affects me.
Portus deletes images itself.
I found, that each image was deleted after 30 day it was added, It looks like, the garbage collector did it, but it is not enabled in my config (I use default config):

delete:
  enabled: true
  contributors: false
  garbage_collector:
    enabled: false
    older_than: 30
    tag: ''

Portus version: 2.5.0-dev@c98a51eb0cc2ee772c19ee64269d2a86ec0fe0dc

portus-background container log:

[catalog] Removed the tag 'latest'.
[catalog] Removed the image 'cuda10'.
Handling 'delete' event:
{
  "id": "be4128a1-299f-4479-a46a-dd2d86f3fbcf",
  "timestamp": "2019-01-10T12:17:28.108994516Z",
  "action": "delete",
  "target": {
    "digest": "sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d",
    "repository": "dwx315410/cuda10"
  },
  "request": {
    "id": "9c7d04e3-0c02-4aaa-b32d-ba9dbdb5c5dd",
    "addr": "172.18.0.5",
    "host": "docker.matis.huawei.com",
    "method": "DELETE",
    "useragent": "Ruby"
  },
  "actor": {
    "name": "portus"
  },
  "source": {
    "addr": "f196d255bd82:5000",
    "instanceID": "cc9e6a8f-62a7-45a1-9711-9d3cd222fe14"
  }
}

registry container log:

time="2019-01-10T12:17:26Z" level=debug msg="authorizing request" go.version=go1.7.6 http.request.host=docker.matis.huawei.com http.request.id=a6f1618a-5ce4-4580-8bdb-92949e7e4f57 http.request.method=DELETE http.request.remoteaddr=172.18.0.5 http.request.uri="/v2/dwx315410/cuda10/manifests/sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d" http.request.useragent=Ruby instance.id=cc9e6a8f-62a7-45a1-9711-9d3cd222fe14 vars.name="dwx315410/cuda10" vars.reference="sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d" version=v2.6.2
time="2019-01-10T12:17:26Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=docker.matis.huawei.com http.request.id=a6f1618a-5ce4-4580-8bdb-92949e7e4f57 http.request.method=DELETE http.request.remoteaddr=172.18.0.5 http.request.uri="/v2/dwx315410/cuda10/manifests/sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d" http.request.useragent=Ruby instance.id=cc9e6a8f-62a7-45a1-9711-9d3cd222fe14 vars.name="dwx315410/cuda10" vars.reference="sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d" version=v2.6.2
172.18.0.1 - - [10/Jan/2019:12:17:26 +0000] "DELETE /v2/dwx315410/cuda10/manifests/sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d HTTP/1.0" 401 156 "" "Ruby"
...
time="2019-01-10T12:17:28Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=docker.matis.huawei.com http.request.id=9c7d04e3-0c02-4aaa-b32d-ba9dbdb5c5dd http.request.method=DELETE http.request.remoteaddr=172.18.0.5 http.request.uri="/v2/dwx315410/cuda10/manifests/sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d" http.request.useragent=Ruby http.response.duration=13.694751ms http.response.status=202 http.response.written=0 instance.id=cc9e6a8f-62a7-45a1-9711-9d3cd222fe14 version=v2.6.2
172.18.0.1 - - [10/Jan/2019:12:17:28 +0000] "DELETE /v2/dwx315410/cuda10/manifests/sha256:302496805dadfe2804ff60d8fa835e488f447c6422d0f897d0a4b12e8a366b9d HTTP/1.0" 202 0 "" "Ruby"

[vitoravelino]

@maleficxp this bug was fixed at #2018. Your image is probably older than that. Also, we've imlemented the suggestions of #2051. If you upgrade your instance you can take advantage of that too.

[vitoravelino]

To be honest I think we can close this issue. What do you think, @mssola?

[mssola]

Agreed. Thanks @vitoravelino for the heads up 👍