AWS EC2 inventory ends in failure

[nasjomach]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

AWS EC2 inventory end in failure after some weeks/days using it.
Job status: Failed

The job used to run correctly for some time and then at some point ends in failures.

Recreating the the same inventory source with the exact same options, credentials, etc makes it ends in Successful status.

Failed job log last entries:
22.907 DEBUG Adding child group zones to parent all
22.909 INFO Loaded 815 groups, 240 hosts
13k output lines (debug)

Successfull job last entries:
47.597 INFO Inventory import completed for in 34.0s
24k output lines (debug)

This seems to not be happening with: AWX 19.5.0.

AWX version

21.1.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Creat AWS EC2 inventory source, use it for some time.

Expected results

Inventory source ends in failure.

Actual results

Failed job log last entries:
22.907 DEBUG Adding child group zones to parent all
22.909 INFO Loaded 815 groups, 240 hosts
13k output lines (debug)

Additional information

Recreating the the same inventory source with the exact same options, credentials, etc makes it ends in success status.

[sarabrajsingh]

hey @nasjomach can you copy/paste the output of the API details for this inventory update?

endpoint: /api/v2/inventory_updates/<id>

where <id> is the id number of the job corresponding to the inventory update.

thanks,
AWX Team

[mick1627]

Here the output of a failed inventory update

[sarabrajsingh]

might be a duplicate of - #12530

[mick1627]

In our case, the pods to do the inventory sync start and finished successfully. The pods run about 15 secondes.
Then, the task in AWX stay in "running" mode for about 15 - 20 secondes and finally finished with error.

[mick1627]

We observed that if we Uncheck the Update options box "Overwrite" on the source inventory, the inventory sync is working.
Maybe related to #12277
From a new inventory, we tried to stop an instance, re-run the inventory, the instance stopped is removed from the inventory it works well.
We will try the same thing in few days with an host used in several job template.

[AlanCoding]

This is a rare log for us to see:

awx.main.dispatch worker pid:437716 is gone (exit=-9)

awx/awx/main/dispatch/pool.py

Line 391 in cfc1255

logger.error('worker pid:{} is gone (exit={})'.format(w.pid, w.exitcode))

This is very descriptive. The process that was saving the inventory data died.