Update Venafi module #55286

arykalin · 2019-11-13T15:01:18Z

What does this PR do?

Transitions solution to Venafi preferred integration interface (VCert-Python) and updated to align with Venafi standard use cases for DevOps

Copy of #54695 merged with master branch

What issues does this PR fix or reference?

Restores interoperability with Venafi Cloud

Previous Behavior

Supported only Venafi Cloud

New Behavior

Adds support for Venafi Trust Protection Platform

Tests written?

Yes

Commits signed with GPG?

No

waynew

@arykalin thanks for the PR and getting everything shifted to master!

This is mostly looking good - there are a couple of minor style changes that will make this 💯- I don't see any glaring issues.

Though there are some test failures, that I believe are due to requirements. IIRC we need to run pre-commit to rebuild the dependencies. You should be able to just run pre-commit run -av, which will say "failed" the first time.

Let me know if you run into any problems!

tests/integration/externalapi/test_venafiapi.py

salt/runners/venafiapi.py

tests/integration/externalapi/test_venafiapi.py

dwoz · 2019-12-10T00:31:58Z

@arykalin The conflicts need to be resolved, The requirements from master should be used.

arykalin · 2019-12-11T07:24:00Z

@dwoz this requirements was added by pre-commit run -av which @waynew recommended to run since our tests require vcert dependency. How I can add this dependency to tests?

…e-venafi-module # Conflicts: # requirements/static/py2.7/linux.txt # requirements/static/py3.4/linux.txt # requirements/static/py3.5/linux.txt # requirements/static/py3.6/linux.txt # requirements/static/py3.7/linux.txt

arykalin · 2019-12-11T09:11:46Z

I merged with master and run pre-commit run -av again. Will it fit?

waynew · 2019-12-17T15:49:36Z

Hey @arykalin thanks for the updates! There is one issue unrelated to this PR in the failed tests, but there are a couple of failures that are related to Venafi:

https://jenkinsci.saltstack.com/job/pr-ubuntu1804-py3/job/PR-55286/18/testReport/

Traceback (most recent call last):
  File "/tmp/kitchen/testing/tests/integration/externalapi/test_venafiapi.py", line 38, in wrapper
    return func(self, _random_name(prefix='salt-test-'), *args, **kwargs)
  File "/tmp/kitchen/testing/tests/integration/externalapi/test_venafiapi.py", line 59, in test_request
    cert = x509.load_pem_x509_certificate(cert_output.encode(), default_backend())
  File "/tmp/kitchen/testing/.nox/runtests-parametrized-3-coverage-true-crypto-none-transport-zeromq/lib/python3.6/site-packages/cryptography/x509/base.py", line 50, in load_pem_x509_certificate
    return backend.load_pem_x509_certificate(data)
  File "/tmp/kitchen/testing/.nox/runtests-parametrized-3-coverage-true-crypto-none-transport-zeromq/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1170, in load_pem_x509_certificate
    "Unable to load certificate. See https://cryptography.io/en/la"
ValueError: Unable to load certificate. See https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for more details.

Do you need any help looking into that?

arykalin · 2019-12-31T14:26:31Z

@waynew fixed tests

waynew · 2020-01-02T22:07:15Z

Almost there - looks like https://jenkinsci.saltstack.com/job/pr-macosxmojave-py2/job/PR-55286/19/ failed as did the py3 Mojave build.

waynew · 2020-01-07T04:51:29Z

@arykalin Looks like a dependency isn't getting installed?

 ('return is', {u'fun': 'venafi.request', u'jid': u'20200106233150621490', u'return': u"'venafi.request' is not available.", u'out': ["'venafi.request' is not available."]})
cert out is: '

arykalin · 2020-01-09T12:36:07Z

@arykalin Looks like a dependency isn't getting installed?

 ('return is', {u'fun': 'venafi.request', u'jid': u'20200106233150621490', u'return': u"'venafi.request' is not available.", u'out': ["'venafi.request' is not available."]})
cert out is: '

@waynew yes, I didn't add vcert to darwin.in file

waynew · 2020-01-09T15:28:46Z

We can probably remove the print lines now, too. Or we can wait until the mac builds run 🙃

If you do remove them, you could rebase the changes on master, too.

…e-venafi-module

codecov · 2020-01-09T21:56:35Z

Codecov Report

Merging #55286 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #55286   +/-   ##
=======================================
  Coverage   18.79%   18.79%           
=======================================
  Files         821      821           
  Lines      175155   175155           
  Branches    37695    37695           
=======================================
  Hits        32910    32910           
  Misses     139568   139568           
  Partials     2677     2677

Flag	Coverage Δ
#archlts	`18.07% <0%> (ø)`	⬆️
#centos7	`23.74% <0%> (ø)`	⬆️
#proxy	`23.78% <0%> (ø)`	⬆️
#py2	`18.59% <0%> (ø)`	⬆️
#py3	`18.42% <0%> (ø)`	⬆️
#runtests	`18.79% <0%> (ø)`	⬆️
#ubuntu1604	`23.71% <0%> (ø)`	⬆️
#zeromq	`18.79% <0%> (ø)`	⬆️

codecov · 2020-01-09T22:04:36Z

Codecov Report

Merging #55286 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #55286   +/-   ##
=======================================
  Coverage   18.79%   18.79%           
=======================================
  Files         821      821           
  Lines      175155   175155           
  Branches    37695    37695           
=======================================
  Hits        32910    32910           
  Misses     139568   139568           
  Partials     2677     2677

Flag	Coverage Δ
#archlts	`18.07% <0%> (ø)`	⬆️
#centos7	`23.74% <0%> (ø)`	⬆️
#proxy	`23.78% <0%> (ø)`	⬆️
#py2	`18.59% <0%> (ø)`	⬆️
#py3	`18.42% <0%> (ø)`	⬆️
#runtests	`18.79% <0%> (ø)`	⬆️
#ubuntu1604	`23.71% <0%> (ø)`	⬆️
#zeromq	`18.79% <0%> (ø)`	⬆️

codecov · 2020-01-09T22:04:50Z

Codecov Report

Merging #55286 into master will increase coverage by 0.83%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #55286      +/-   ##
==========================================
+ Coverage   17.97%   18.79%   +0.83%     
==========================================
  Files        1238      821     -417     
  Lines      240782   175155   -65627     
  Branches    52795    37695   -15100     
==========================================
- Hits        43265    32910   -10355     
+ Misses     193686   139568   -54118     
+ Partials     3831     2677    -1154

Flag	Coverage Δ
#archlts	`18.07% <ø> (+1.82%)`	⬆️
#centos7	`23.74% <ø> (-0.95%)`	⬇️
#cloud	`?`
#proxy	`23.78% <ø> (-0.03%)`	⬇️
#py2	`18.59% <ø> (+1.76%)`	⬆️
#py3	`18.42% <ø> (+0.73%)`	⬆️
#runtests	`18.79% <ø> (+0.83%)`	⬆️
#ubuntu1604	`23.71% <ø> (-0.03%)`	⬇️
#zeromq	`18.79% <ø> (+1.8%)`	⬆️

Impacted Files	Coverage Δ
salt/utils/xmlutil.py	`13.64% <0%> (-40.9%)`	⬇️
salt/utils/smb.py	`19.84% <0%> (-39.27%)`	⬇️
salt/utils/cloud.py	`9.32% <0%> (-31.96%)`	⬇️
salt/utils/vt.py	`10.76% <0%> (-28.27%)`	⬇️
salt/cloud/__init__.py	`7.78% <0%> (-20.82%)`	⬇️
salt/output/yaml_out.py	`45% <0%> (-20%)`	⬇️
salt/config/__init__.py	`27.63% <0%> (-16.84%)`	⬇️
salt/utils/http.py	`11.31% <0%> (-15.3%)`	⬇️
salt/utils/validate/path.py	`40% <0%> (-10%)`	⬇️
salt/utils/decorators/__init__.py	`50.43% <0%> (-8.35%)`	⬇️
... and 546 more

…e-venafi-module

…date-venafi-module

codecov · 2020-01-10T09:52:56Z

Codecov Report

❗ No coverage uploaded for pull request base (master@511eaa9). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #55286   +/-   ##
=========================================
  Coverage          ?   18.79%           
=========================================
  Files             ?      821           
  Lines             ?   175155           
  Branches          ?    37695           
=========================================
  Hits              ?    32910           
  Misses            ?   139568           
  Partials          ?     2677

Flag	Coverage Δ
#archlts	`18.07% <0%> (?)`
#centos7	`23.74% <0%> (?)`
#proxy	`23.78% <0%> (?)`
#py2	`18.59% <0%> (?)`
#py3	`18.42% <0%> (?)`
#runtests	`18.79% <0%> (?)`
#ubuntu1604	`23.71% <0%> (?)`
#zeromq	`18.79% <0%> (?)`

arykalin · 2020-01-10T14:10:32Z

Hmm, can't understand why pylint fails, it was passing before :(

codecov · 2020-01-10T14:26:21Z

Codecov Report

❗ No coverage uploaded for pull request base (master@511eaa9). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #55286   +/-   ##
=========================================
  Coverage          ?   18.79%           
=========================================
  Files             ?      821           
  Lines             ?   175155           
  Branches          ?    37695           
=========================================
  Hits              ?    32910           
  Misses            ?   139568           
  Partials          ?     2677

Flag	Coverage Δ
#archlts	`18.07% <0%> (?)`
#centos7	`23.74% <0%> (?)`
#proxy	`23.78% <0%> (?)`
#py2	`18.59% <0%> (?)`
#py3	`18.42% <0%> (?)`
#runtests	`18.79% <0%> (?)`
#ubuntu1604	`23.71% <0%> (?)`
#zeromq	`18.79% <0%> (?)`

salt/runners/venafiapi.py

arykalin · 2020-01-13T13:16:37Z

@waynew ci/py3/ubuntu1604 build failing with unknown reason, could you look?

waynew · 2020-01-13T16:06:20Z

Not sure exactly what failed there either - I went ahead and triggered a rebuild.

waynew

🎉

waynew · 2020-01-13T21:10:04Z

@arykalin woohoo! We've got the build passing now, success!

s0undt3ch

There's a bug when our compile requirement pre-commit script runs in some situations.

I'll update requirements and push the fixed requirements.

s0undt3ch · 2020-01-14T06:24:02Z

This PR was resubmitted with a more concise git history and fixed requirements files in #55858 since I didn't have push access to the venafi fork.

Copy pasting changes from branch venafi-fix

44745e8

arykalin requested a review from a team as a code owner November 13, 2019 15:01

ghost requested a review from dwoz November 13, 2019 15:01

arykalin mentioned this pull request Nov 13, 2019

Updates to Venafi module #55267

Closed

Adding new line to the end of file

d6c8565

waynew suggested changes Nov 13, 2019

View reviewed changes

arykalin added 9 commits November 19, 2019 16:29

Updating dependencies and removing commented @destructivetest

570183a

Updating code

4e713a3

Updating code

acc6b21

Update vcert, fixing private key recieving which was changed.

b1f904f

Fixing code for vcert refactoring

7e990c0

removing print

e3918a7

removing commented code

4277e27

Fixing lint

d2b04ce

Running pre-commit run -av

bb93e8e

arykalin requested a review from waynew November 27, 2019 10:28

arykalin added 3 commits December 9, 2019 15:45

Removing log from test

3ef6013

Avoid using break

d82bcb3

dib't use pytest.fail

2216faa

arykalin added 2 commits December 11, 2019 10:24

running pre-commit run -av with vcert

94e6948

arykalin and others added 3 commits December 11, 2019 18:20

fixing PEP8 E231: missing whitespace after ','

f03337b

removing unused import

28b43bb

Merge branch 'master' into update-venafi-module

8f8329d

waynew mentioned this pull request Dec 17, 2019

Lint fixes to master #55671

Merged

Adding temporary debug

23bc4be

arykalin and others added 2 commits January 5, 2020 14:20

Adding print to debug test on macos

6c3cd68

more debugging

325ac51

Running pre-commit run -av for darwin

b825433

arykalin added 2 commits January 10, 2020 00:20

removing print

16cea5f

Merge branch 'master' of https://github.com/saltstack/salt into updat…

511eaa9

…e-venafi-module

arykalin and others added 3 commits January 10, 2020 12:51

Merge branch 'master' into update-venafi-module

8672d51

Merge branch 'master' of https://github.com/saltstack/salt into updat…

b61bda7

…e-venafi-module

Merge branch 'update-venafi-module' of github.com:Venafi/salt into up…

2616d69

…date-venafi-module

waynew suggested changes Jan 10, 2020

View reviewed changes

salt/runners/venafiapi.py Outdated Show resolved Hide resolved

salt/runners/venafiapi.py Outdated Show resolved Hide resolved

arykalin added 2 commits January 11, 2020 10:13

raise exception instead of sys.exit

01bdec4

Fixing pep warnings

7421fda

waynew approved these changes Jan 13, 2020

View reviewed changes

s0undt3ch suggested changes Jan 13, 2020

View reviewed changes

s0undt3ch mentioned this pull request Jan 14, 2020

Update Venfi Module #55858

Merged

s0undt3ch closed this Jan 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Venafi module #55286

Update Venafi module #55286

arykalin commented Nov 13, 2019

waynew left a comment

dwoz commented Dec 10, 2019

arykalin commented Dec 11, 2019

arykalin commented Dec 11, 2019

waynew commented Dec 17, 2019

arykalin commented Dec 31, 2019

waynew commented Jan 2, 2020

waynew commented Jan 7, 2020

arykalin commented Jan 9, 2020

waynew commented Jan 9, 2020

codecov bot commented Jan 9, 2020

codecov bot commented Jan 9, 2020

codecov bot commented Jan 9, 2020

codecov bot commented Jan 10, 2020

arykalin commented Jan 10, 2020

codecov bot commented Jan 10, 2020

arykalin commented Jan 13, 2020

waynew commented Jan 13, 2020

waynew left a comment

waynew commented Jan 13, 2020

s0undt3ch left a comment

s0undt3ch commented Jan 14, 2020

Update Venafi module #55286

Update Venafi module #55286

Conversation

arykalin commented Nov 13, 2019

What does this PR do?

What issues does this PR fix or reference?

Previous Behavior

New Behavior

Tests written?

Commits signed with GPG?

waynew left a comment

Choose a reason for hiding this comment

dwoz commented Dec 10, 2019

arykalin commented Dec 11, 2019

arykalin commented Dec 11, 2019

waynew commented Dec 17, 2019

arykalin commented Dec 31, 2019

waynew commented Jan 2, 2020

waynew commented Jan 7, 2020

arykalin commented Jan 9, 2020

waynew commented Jan 9, 2020

codecov bot commented Jan 9, 2020

Codecov Report

codecov bot commented Jan 9, 2020

Codecov Report

codecov bot commented Jan 9, 2020

Codecov Report

codecov bot commented Jan 10, 2020

Codecov Report

arykalin commented Jan 10, 2020

codecov bot commented Jan 10, 2020

Codecov Report

arykalin commented Jan 13, 2020

waynew commented Jan 13, 2020

waynew left a comment

Choose a reason for hiding this comment

waynew commented Jan 13, 2020

s0undt3ch left a comment

Choose a reason for hiding this comment

s0undt3ch commented Jan 14, 2020