Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compute.oslogin.service_account_ssh_test: test_main failed #7277

Closed
flaky-bot bot opened this issue Dec 21, 2021 · 15 comments · Fixed by #7370, #8208 or #8244
Closed

compute.oslogin.service_account_ssh_test: test_main failed #7277

flaky-bot bot opened this issue Dec 21, 2021 · 15 comments · Fixed by #7370, #8208 or #8244
Assignees
Labels
api: compute Issues related to the Compute Engine API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p3 Desirable enhancement or fix. May not be included in next release. 🚨 This issue needs some love. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Dec 21, 2021

Note: #6894 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: 5ceff25
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 64, in test_main
    machine_type, account_email)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 201, in setup_resources
    body=instance_config).execute()
  File "/workspace/compute/oslogin/.nox/py-3-6/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-6/lib/python3.6/site-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 68, in test_main
raise Exception('Could not set up the necessary test resources.')
Exception: Could not set up the necessary test resources.

@flaky-bot flaky-bot bot added flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 21, 2021
@product-auto-label product-auto-label bot added api: compute Issues related to the Compute Engine API. samples Issues that are directly related to samples. labels Dec 21, 2021
@leahecole leahecole assigned leahecole and unassigned m-strzelczyk Dec 21, 2021
@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Dec 26, 2021
@leahecole leahecole added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Dec 28, 2021
@busunkim96 busunkim96 added external This issue is blocked on a bug with the actual product. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. external This issue is blocked on a bug with the actual product. labels Dec 30, 2021
@yoshi-automation yoshi-automation removed the 🚨 This issue needs some love. label Dec 30, 2021
@leahecole
Copy link
Collaborator

This should be resolved now that b/211071739 is resolved

@flaky-bot flaky-bot bot reopened this Jan 7, 2022
@flaky-bot flaky-bot bot added the flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. label Jan 7, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Jan 7, 2022

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.


commit: 32950b9
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 97, in test_main
    ssh_login()
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/six.py", line 719, in reraise
    raise value
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 90, in ssh_login
    main(cmd, project, test_id, zone, oslogin, account, hostname)
  File "/workspace/compute/oslogin/service_account_ssh.py", line 116, in main
    private_key_file = create_ssh_key(oslogin, account)
  File "/workspace/compute/oslogin/service_account_ssh.py", line 79, in create_ssh_key
    oslogin.users().importSshPublicKey(parent=account, body=body).execute()
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/googleapiclient/http.py", line 922, in execute
    resp, content = _retry_request(
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/googleapiclient/http.py", line 190, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/google_auth_httplib2.py", line 245, in request
    self.credentials.refresh(self._request)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/google/oauth2/service_account.py", line 410, in refresh
    access_token, expiry, _ = _client.jwt_grant(
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/google/oauth2/_client.py", line 193, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/google/oauth2/_client.py", line 165, in _token_endpoint_request
    _handle_error_response(response_data)
  File "/workspace/compute/oslogin/.nox/py-3-10/lib/python3.10/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_data)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid grant: account not found', {'error': 'invalid_grant', 'error_description': 'Invalid grant: account not found'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 99, in test_main
raise Exception('SSH to the test instance failed.')
Exception: SSH to the test instance failed.

@leahecole leahecole assigned m-strzelczyk and unassigned leahecole Jan 11, 2022
@leahecole
Copy link
Collaborator

Hi @m-strzelczyk - I need some help debugging this. Please reach out if you need any help - it may have to do with the default network being recreated. I'll ping you with a way to request more permissions if needed

@m-strzelczyk
Copy link
Contributor

Seems like the Jenkins account is missing the roles/iam.serviceAccountKeyAdmin and roles/iam.serviceAccountUser roles in the Python 3.10 project. Let's discuss this in DMs.

@m-strzelczyk m-strzelczyk mentioned this issue Jan 13, 2022
8 tasks
@m-strzelczyk
Copy link
Contributor

I have made pull request, to play around with the test a bit (#7370 ), but it turns out it works quite fine there...

I have noticed that the error we got ({'error': 'invalid_grant', 'error_description': 'Invalid grant: account not found'}) is rather generic and can happen for a lot of different reasons.

I'm starting to think there's not much I could do here, as the test seems to work and the reason it failed has just went away 🤔

@flaky-bot flaky-bot bot reopened this Apr 17, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Apr 17, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: e7379a8
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 62, in test_main
    service_account_key = setup_resources(
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 211, in setup_resources
    compute.instances().setIamPolicy(
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 68, in test_main
raise Exception('Could not set up the necessary test resources.')
Exception: Could not set up the necessary test resources.

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Jun 19, 2022
m-strzelczyk added a commit that referenced this issue Jul 15, 2022
m-strzelczyk added a commit that referenced this issue Jul 22, 2022
* WIP: Triggering a new PR to have a stage for running tests and experimenting.

* fix(ssh-test): Trying to fix #7277

* Import reordering
@flaky-bot flaky-bot bot reopened this Jul 23, 2022
@flaky-bot
Copy link
Author

flaky-bot bot commented Jul 23, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 884d81d
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 97, in test_main
    ssh_login()
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/six.py", line 719, in reraise
    raise value
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 90, in ssh_login
    main(cmd, project, test_id, zone, oslogin, account, hostname)
  File "/workspace/compute/oslogin/service_account_ssh.py", line 131, in main
    profile = oslogin.users().getLoginProfile(name=account).execute()
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/http.py", line 922, in execute
    resp, content = _retry_request(
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/http.py", line 190, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/google_auth_httplib2.py", line 245, in request
    self.credentials.refresh(self._request)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/google/oauth2/service_account.py", line 410, in refresh
    access_token, expiry, _ = _client.jwt_grant(
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/google/oauth2/_client.py", line 193, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/google/oauth2/_client.py", line 165, in _token_endpoint_request
    _handle_error_response(response_data)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_data)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid grant: account not found', {'error': 'invalid_grant', 'error_description': 'Invalid grant: account not found'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 99, in test_main
raise Exception('SSH to the test instance failed.')
Exception: SSH to the test instance failed.

m-strzelczyk added a commit that referenced this issue Aug 4, 2022
* WIP: Triggering a new PR to have a stage for running tests and experimenting.

* fix(ssh-test): Trying to fix #7277

* Import reordering

* fix(compute-ssh): Another fix for #7277

* Changing debian from 9 to 11

* Fixing something.

* Testing things.

Co-authored-by: Vadym Matsishevskyi <25311427+vam-google@users.noreply.github.com>
@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 4, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: b868f40
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 62, in test_main
    service_account_key = setup_resources(
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 198, in setup_resources
    operation = compute.instances().insert(
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-9/lib/python3.9/site-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 68, in test_main
raise Exception('Could not set up the necessary test resources.')
Exception: Could not set up the necessary test resources.

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 10, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 1cd2422
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 97, in test_main
    ssh_login()
  File "/workspace/compute/oslogin/.nox/py-3-8/lib/python3.8/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/workspace/compute/oslogin/.nox/py-3-8/lib/python3.8/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/workspace/compute/oslogin/.nox/py-3-8/lib/python3.8/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/workspace/compute/oslogin/.nox/py-3-8/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/workspace/compute/oslogin/.nox/py-3-8/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 93, in ssh_login
    assert assert_value in out
AssertionError: assert 'Linux oslogin-test-525107' in "Creating key users/oslogin-test-525107@python-docs-samples-tests.iam.gserviceaccount.com and {'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaw2VfkFEi5lx/dV123SeFcLpTosbkFqFY+UOTCGi0qf+VOqP6b7QnmTmS+LQzDUsfpliShu3K6f+deifnRK1vEhS5m9h0rKwdjpWupRGQWjIelmen4GOQ8EKIXKdgOrNAcdzLHllfIvOi21lKennhdqirdTYgg5pOfdnyP5FcWPnVYbjuntgvBs6TTWs4FL+qRhAZNvaZjUecJ5d387aBUAJFAeHLmTP9rfakYRP3w0aJ93BvqdQ0SGz9vPbH0TePd8e7LmU696O5/4VkqEf9Jrs0CpRzDYhQSuhhsNab2zYdLPqAVk4fOFvlWaCP7KjuRRYs5aA7Fpn/Oh7e2kX7Q58NGdxhZyb3AZxMlryLI6vYxGgl2R6y60ZrqbZl10gTo+ir79RTczi2mBrWGmA/yS2NhFCNHcvwTUurT9MoRdyt6nhjsKDdEHACfmpiLPoOUokeWjhcJmZSDZtiBb1K5oFpWiuR/a1y8K6rQf+ldlec7NJ8GS67P9a9QEZmw80= kbuilder@kokoro-gcp-ubuntu-prod-750669605', 'expirationTimeUsec': 1660131791927780}\nssh: connect to host 34.138.210.107 port 22: Connection timed out\n"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 99, in test_main
raise Exception('SSH to the test instance failed.')
Exception: SSH to the test instance failed.

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 11, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 54c72b8
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 97, in test_main
    ssh_login()
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/six.py", line 719, in reraise
    raise value
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 93, in ssh_login
    assert assert_value in out
AssertionError: assert 'Linux oslogin-test-115031' in "Creating key users/oslogin-test-115031@python-docs-samples-tests.iam.gserviceaccount.com and {'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbJRHSGBlt3CRWbQy97a/HFURfT8nKOlr+xrQccJR2ULgCaTN5eUPXXFEAxlWI1HC6EP9m02hUrmjpanHINbhsWcxPKscsDjpt4bdBk44mkTVBEST6jx3OIpjIMRHweVqA59WIjLkrYfSQkVJdE/5YDljrYAtv3FLVJzBZwiQtvEr6y9zIgxrOu2q80+dcxyzhMtqtbJKtlfta6lZbJIrUcb3MUtoGoVSyanaQ0dZaUTepNrm9TlGEoxLbT8L0xPampher1A47V2D2rMFj1DVCsXytyXPUlCMBH9vPgSCSbrVwAo8p47kiSTsw79nE0GjTYERWayZMY3mKtb4sr64IZ5W5FuwpD4BNvlB0X9QyvnU0/A0XedssSiz+T1P/nafeF2D4Sp+gEa0Ck/XBfWgxi1Z8UdSdussJJ0RHkCqNrGsgEzrrBHJbgQVJeSHa/oH352TeTmx1fJYKVw0AG5QHQsLSMM6WgJzZTP3msjUM+GYtR+m69ZdBN29tbElTl3k= kbuilder@kokoro-gcp-ubuntu-prod-431126807', 'expirationTimeUsec': 1660246238189151}\nssh: connect to host 34.148.5.215 port 22: Connection timed out\n"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 99, in test_main
raise Exception('SSH to the test instance failed.')
Exception: SSH to the test instance failed.

@flaky-bot
Copy link
Author

flaky-bot bot commented Aug 30, 2022

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.


commit: 4762700
buildURL: Build Status, Sponge
status: failed

Test output
Traceback (most recent call last):
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 66, in test_main
    machine_type, account_email)
  File "/workspace/compute/oslogin/service_account_ssh_test.py", line 225, in setup_resources
    'role': 'roles/compute.osLogin'
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 131, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/workspace/compute/oslogin/.nox/py-3-7/lib/python3.7/site-packages/googleapiclient/http.py", line 937, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/compute/oslogin/service_account_ssh_test.py", line 70, in test_main
raise Exception('Could not set up the necessary test resources.')
Exception: Could not set up the necessary test resources.

@flaky-bot flaky-bot bot reopened this Aug 30, 2022
@leahecole
Copy link
Collaborator

Fwiw this test is also leaving leftovers - I just cleaned a bunch up in the test project because we had VM leftovers giving us quota issues

@m-strzelczyk m-strzelczyk added priority: p3 Desirable enhancement or fix. May not be included in next release. and removed priority: p2 Moderately-important priority. Fix may not be included in next release. labels Sep 28, 2022
@kweinmeister
Copy link
Contributor

Hi @leahecole, do you happen to have any more details on which resources are left over?

cleanup_resources() looks like it deletes several types of resources. It's possible that these leftovers are caused when an exception is thrown, or the cleanup code is missing resource types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: compute Issues related to the Compute Engine API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p3 Desirable enhancement or fix. May not be included in next release. 🚨 This issue needs some love. samples Issues that are directly related to samples. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
7 participants