Fix RDS Enhanced Monitoring bug #712

marknet15 · 2021-09-15T16:12:35Z

SUMMARY

This is a fix for an issue when an RDS instance already exists and you wish to enable enhanced monitoring, for the full details see the linked old reported issue:
ansible/ansible#51772

But in summary currently if you enable enhanced monitoring on an RDS instance that already exists where it isn't already enabled then the following is returned:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'MonitoringRoleArn'
fatal: [localhost_eu-west-1-pdv-qa-1 -> 127.0.0.1]: FAILED! => changed=false 
  module_stderr: |-
    Traceback (most recent call last):
      File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 975, in _run
        self._run_code(code, mod)
      File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 939, in _run_code
        exec(code, vars(mod))
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1245, in <module>
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1210, in main
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 855, in get_parameters
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 885, in get_options_with_changing_values
      File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 983, in get_changing_options_with_consistent_keys
    KeyError: 'MonitoringRoleArn'
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error

Depends-On: mattclay/aws-terminator#164

Other changes
A load of issues have surfaced in the integration tests due to how slow RDS is to create / modify etc. I've condensed down the tests where possible reducing the number of inventory jobs to 6 and bumped serial to 6 so that hopefully all tests can run at once and finish within the 1 hr AWS session duration.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

rds_instance

ansibullbot · 2021-09-15T16:17:22Z

cc @jillr @markuman @s-hertel @tremble @wimnat
click here for bot help

alinabuzachis · 2021-09-16T18:18:02Z

@marknet15 Thank you for working on this. Would you be so kind to add some integration tests to test this functionality?

marknet15 · 2021-09-17T10:55:02Z

@alinabuzachis thanks for the comment 👍🏻 I've added some integration tests, but upon inspection it doesn't look like zuul has run any, it seems to be skipping and I can't see why, would you be able to assist? I'm a little stuck, as "tests/integration/targets/" seems correct:

source ~/venv/bin/activate; ansible-test integration --diff --no-temp-workdir --skip-tags False --retry-on-error --continue-on-error --python 3.6 -vvvv rds_instance
stdout
RLIMIT_NOFILE: (1024, 524288)
MAXFD: -1
Falling back to tests in "tests/integration/targets/" because "roles/test/" was not found.
WARNING: Excluding tests marked "unsupported" which require --allow-unsupported or prefixing with "unsupported/": rds_instance
Run command: ssh-keygen -m PEM -q -t rsa -N '' -f /home/zuul/.ansible/test/id_rsa
Working directory: /home/zuul/.ansible/collections/ansible_collections/community/aws
Program found: /usr/bin/ssh-keygen
HOME=/home/zuul
LC_ALL=en_US.UTF-8
PATH=/home/zuul/venv/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Command exited with status 0 after 0.33289432525634766 seconds.
WARNING: All targets skipped.

https://dashboard.zuul.ansible.com/t/ansible/build/8075a26688a749f69fed570912827d6a/console

marknet15 · 2021-09-18T11:26:16Z

@alinabuzachis I think If I'm following this correctly the rds_instance integration tests are actually disabled:
file: community.aws/tests/integration/targets/rds_instance/aliases

# reason: missing-policy
# reason: slow
unsupported

What would be the next steps ?

tests/integration/targets/rds_instance/aliases

alinabuzachis · 2021-09-22T09:00:50Z

recheck

Co-authored-by: Mark Chappell <mchappel@redhat.com>

- Condense tests where possible - Remove irrelevant snapshot tests - Up concurrency to 6

jillr

A few things I ran across while testing the policy PR. I did still run into the 60 minutes STS timeout but it got much further this time. Thanks so much for all you're doing for these tests!

jillr · 2021-09-29T17:19:26Z

tests/integration/targets/rds_instance/roles/rds_instance/tasks/test_modification.yml

@@ -100,6 +108,8 @@
      # TODO: test modifying db_subnet_group_name, db_security_groups, db_parameter_group_name, option_group_name,
      # monitoring_role_arn, monitoring_interval, domain, domain_iam_role_name, cloudwatch_logs_export_configuration

+      # Test multiple modifications including enabling enhanced monitoring


RDS doesn't like the version or parameter combination in this one.
"msg": "Unable to modify DB instance: An error occurred (InvalidParameterCombination) when calling the ModifyDBInstance operation: Cannot upgrade mariadb from 10.3.20 to 10.4.8",

Hmm according to the docs AWS allows 10.3 to 10.4 as a major version upgrade:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MariaDB.html

But might be some inconsistencies in the tests, I've updated things slightly

tests/integration/targets/rds_instance/roles/rds_instance/tasks/test_modification.yml

ansibullbot · 2021-10-08T08:40:21Z

@marknet15 this PR contains the following merge commits:

16ececf

Please rebase your branch to remove these commits.

click here for bot help

Co-authored-by: Mark Chappell <mchappel@redhat.com>

- Condense tests where possible - Remove irrelevant snapshot tests - Up concurrency to 6

This reverts commit 8d9aacc.

…into rds-fix

ansibullbot · 2021-10-08T08:51:58Z

@marknet15 this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

ansibullbot · 2021-10-08T08:51:58Z

@marknet15 This PR was evaluated as a potentially problematic PR for the following reasons:

More than 50 changed files.

Such PR can only be merged by human. Contact a Core team member to review this PR on IRC: #ansible-devel on irc.freenode.net

click here for bot help

marknet15 · 2021-10-08T08:54:20Z

Apologies I completely messed up my branch 👎🏻 and created a clean one:
#747

Rds enhanced monitoring bug fix SUMMARY (a copy of #712 as I messed up my branch by accident) This is a fix for an issue when an RDS instance already exists and you wish to enable enhanced monitoring, for the full details see the linked old reported issue: ansible/ansible#51772 But in summary currently if you enable enhanced monitoring on an RDS instance that already exists where it isn't already enabled then the following is returned: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'MonitoringRoleArn' fatal: [localhost_eu-west-1-pdv-qa-1 -> 127.0.0.1]: FAILED! => changed=false module_stderr: |- Traceback (most recent call last): File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 975, in _run self._run_code(code, mod) File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 939, in _run_code exec(code, vars(mod)) File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1245, in <module> File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1210, in main File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 855, in get_parameters File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 885, in get_options_with_changing_values File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 983, in get_changing_options_with_consistent_keys KeyError: 'MonitoringRoleArn' module_stdout: '' msg: |- MODULE FAILURE See stdout/stderr for the exact error Originally-Depends-On: mattclay/aws-terminator#164 Other changes A load of issues have surfaced in the integration tests due to how slow RDS is to create / modify etc. I've condensed down the tests where possible reducing the number of inventory jobs to 6 and bumped serial to 6 so that hopefully all tests can run at once and finish within the 1 hr AWS session duration. ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_instance Reviewed-by: Mark Chappell <None> Reviewed-by: Mark Woolley <mw@marknet15.com> Reviewed-by: None <None>

Pin pytest version SUMMARY ISSUE TYPE Bugfix Pull Request Docs Pull Request Feature Pull Request New Module Pull Request COMPONENT NAME ADDITIONAL INFORMATION

Rds enhanced monitoring bug fix SUMMARY (a copy of ansible-collections#712 as I messed up my branch by accident) This is a fix for an issue when an RDS instance already exists and you wish to enable enhanced monitoring, for the full details see the linked old reported issue: ansible/ansible#51772 But in summary currently if you enable enhanced monitoring on an RDS instance that already exists where it isn't already enabled then the following is returned: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'MonitoringRoleArn' fatal: [localhost_eu-west-1-pdv-qa-1 -> 127.0.0.1]: FAILED! => changed=false module_stderr: |- Traceback (most recent call last): File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 975, in _run self._run_code(code, mod) File "master:/opt/mitogen/mitogen-0.2.9/ansible_mitogen/runner.py", line 939, in _run_code exec(code, vars(mod)) File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1245, in <module> File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 1210, in main File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 855, in get_parameters File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 885, in get_options_with_changing_values File "master:/tmp/build/4bef5c86/framework/library/cloud/aws/rds_instance.py", line 983, in get_changing_options_with_consistent_keys KeyError: 'MonitoringRoleArn' module_stdout: '' msg: |- MODULE FAILURE See stdout/stderr for the exact error Originally-Depends-On: mattclay/aws-terminator#164 Other changes A load of issues have surfaced in the integration tests due to how slow RDS is to create / modify etc. I've condensed down the tests where possible reducing the number of inventory jobs to 6 and bumped serial to 6 so that hopefully all tests can run at once and finish within the 1 hr AWS session duration. ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_instance Reviewed-by: Mark Chappell <None> Reviewed-by: Mark Woolley <mw@marknet15.com> Reviewed-by: None <None> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@8fe00cb

marknet15 added 2 commits September 15, 2021 17:08

Fix RDS bug

1adfc2f

add changelog

f1717a4

ansibullbot added bug This issue/PR relates to a bug community_review module module needs_triage plugins plugin (any type) small_patch Hopefully easy to review traceback labels Sep 15, 2021

marknet15 added 2 commits September 17, 2021 11:23

add tests and make further bug fix tweak

2e6b3f0

test update

0191738

ansibullbot added integration tests/integration tests tests and removed small_patch Hopefully easy to review labels Sep 17, 2021

marknet15 added 2 commits September 18, 2021 12:28

correct test var

8530978

Enable tests

9a36eed

marknet15 mentioned this pull request Sep 20, 2021

Add RDS missing permissions mattclay/aws-terminator#164

Merged

tremble reviewed Sep 21, 2021

View reviewed changes

tests/integration/targets/rds_instance/aliases Show resolved Hide resolved

marknet15 and others added 4 commits September 27, 2021 11:52

Update tests/integration/targets/rds_instance/aliases

fd62423

Co-authored-by: Mark Chappell <mchappel@redhat.com>

try leaving out the iam deletion for enhanced monitoring

25fcc7a

move IAM creation to earlier on

ad93153

The tests are exceeding a 1hr session limit

c541e83

- Condense tests where possible - Remove irrelevant snapshot tests - Up concurrency to 6

jillr reviewed Sep 29, 2021

View reviewed changes

marknet15 added 2 commits October 1, 2021 15:51

PR feedback

98d9cb3

disable wait on final delete

f177d0b

ansibullbot added merge_commit This PR contains at least one merge commit. Please resolve! needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html and removed community_review labels Oct 8, 2021

marknet15 and others added 15 commits October 8, 2021 09:46

Fix RDS bug

2e854d4

add changelog

f2906d8

add tests and make further bug fix tweak

eecfccb

test update

cfa1a69

correct test var

feeae5b

Enable tests

3e38cef

Update tests/integration/targets/rds_instance/aliases

4d7d794

Co-authored-by: Mark Chappell <mchappel@redhat.com>

try leaving out the iam deletion for enhanced monitoring

7cbf05e

move IAM creation to earlier on

eb7ba05

The tests are exceeding a 1hr session limit

ebb27b7

- Condense tests where possible - Remove irrelevant snapshot tests - Up concurrency to 6

PR feedback

1c21ac2

disable wait on final delete

4c222aa

further adjust

5f4a204

Revert "Squashed commit of the following:"

54cf169

This reverts commit 8d9aacc.

Merge branch 'rds-fix' of https://github.com/marknet15/community.aws …

8c6770f

…into rds-fix

ansibullbot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR label Oct 8, 2021

marknet15 mentioned this pull request Oct 8, 2021

Rds enhanced monitoring bug fix #747

Merged

marknet15 closed this Oct 8, 2021

marknet15 deleted the rds-fix branch October 8, 2021 08:54

marknet15 mentioned this pull request Oct 10, 2021

Rds enhanced monitoring fix (retry) #751

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix RDS Enhanced Monitoring bug #712

Fix RDS Enhanced Monitoring bug #712

marknet15 commented Sep 15, 2021 •

edited

Loading

ansibullbot commented Sep 15, 2021

alinabuzachis commented Sep 16, 2021

marknet15 commented Sep 17, 2021

marknet15 commented Sep 18, 2021 •

edited

Loading

alinabuzachis commented Sep 22, 2021

jillr left a comment

jillr Sep 29, 2021

marknet15 Oct 1, 2021

ansibullbot commented Oct 8, 2021

ansibullbot commented Oct 8, 2021

ansibullbot commented Oct 8, 2021

marknet15 commented Oct 8, 2021

Fix RDS Enhanced Monitoring bug #712

Fix RDS Enhanced Monitoring bug #712

Conversation

marknet15 commented Sep 15, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ansibullbot commented Sep 15, 2021

alinabuzachis commented Sep 16, 2021

marknet15 commented Sep 17, 2021

marknet15 commented Sep 18, 2021 • edited Loading

alinabuzachis commented Sep 22, 2021

jillr left a comment

Choose a reason for hiding this comment

jillr Sep 29, 2021

Choose a reason for hiding this comment

marknet15 Oct 1, 2021

Choose a reason for hiding this comment

ansibullbot commented Oct 8, 2021

ansibullbot commented Oct 8, 2021

ansibullbot commented Oct 8, 2021

marknet15 commented Oct 8, 2021

marknet15 commented Sep 15, 2021 •

edited

Loading

marknet15 commented Sep 18, 2021 •

edited

Loading