background job fails when trying to update

[stevenmcastano]

Description

At some point one or two of my docker containers running portus was down while I was doing work, so new images got pushed in, but did not show up in the portus web interface. Once I got everything up and running, NEW images with new tags will show up, but the old ones do not. I tried to configure the background container to use the "update" strategy, but now when it looks and tries to do the update it errors out, crashes and restarts.

Here is the error message it gives:

/srv/Portus/lib/portus/registry_client.rb:92:in `calculate_tag_size': undefined method `[]' for nil:NilClass (NoMethodError)
	from /srv/Portus/lib/portus/registry_client.rb:75:in `manifest'
	from /srv/Portus/app/models/repository.rb:263:in `block in create_tags'
	from /srv/Portus/app/models/repository.rb:260:in `each'
	from /srv/Portus/app/models/repository.rb:260:in `create_tags'
	from /srv/Portus/app/models/repository.rb:224:in `create_or_update!'
	from /srv/Portus/lib/portus/background/sync.rb:113:in `block in update_registry!'
	from /srv/Portus/lib/portus/background/sync.rb:108:in `each'
	from /srv/Portus/lib/portus/background/sync.rb:108:in `update_registry!'
	from /srv/Portus/lib/portus/background/sync.rb:68:in `block (2 levels) in execute!'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
	from /usr/lib64/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
	from /srv/Portus/lib/portus/background/sync.rb:68:in `block in execute!'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:70:in `each'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:70:in `block in find_each'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `loop'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:222:in `in_batches'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/relation/batches.rb:69:in `find_each'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/querying.rb:11:in `find_each'
	from /srv/Portus/lib/portus/background/sync.rb:62:in `execute!'
	from /srv/Portus/bin/background.rb:61:in `block (2 levels) in <top (required)>'
	from /srv/Portus/bin/background.rb:58:in `each'
	from /srv/Portus/bin/background.rb:58:in `each_with_index'
	from /srv/Portus/bin/background.rb:58:in `block in <top (required)>'
	from /srv/Portus/bin/background.rb:57:in `loop'
	from /srv/Portus/bin/background.rb:57:in `<top (required)>'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/commands/runner/runner_command.rb:38:in `load'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/commands/runner/runner_command.rb:38:in `perform'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/command/base.rb:65:in `perform'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/command.rb:46:in `invoke'
	from /srv/Portus/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:12:in `require'
	from bin/rails:12:in `<main>'

This is based on the docker-compose file for ssl with clair and the opensuse/portus:head image.

[stale]

Thanks for all your contributions!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[xfh]

I'm having exactly the same problem with the latest image. Are you also using Docker Registry 2.7?

[HonoluluHenk]

Same here

[Jean-Baptiste-Lasselle]

Ok, I think I have an idea here, based on the env variable PORTUS_BACKGROUND_SERVICE_FQDN I propose in this remark I made :

#2243 (comment)

The idea is that if your background container is down and not receiving (and sending back confirmation), and we make sure the portus container never gets any HTTP REQUEST notifications from the registry (a webhook), then the registry will keep sending the webhooks notifying about those entries you guys want to "update" after bringing back background service.
And the docker registry WILL keep retrying sending those notifications, all of them, until background is up n running.
Alright, I see here that the number one priority about Portus Monitoring, is to monitor the background process, and the notification flow in full details, using elastic stack, grafana and prometheus.

@xfh I'm using Docker Registry 2.6

[stale]

Thanks for all your contributions!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.