antrea OvS not installing #198

jayunit100 · 2022-07-04T17:37:47Z

Looks like as of today, somehow ovs isnt starting....

Any ideas on why ?

Reproducing:

vagrant destroy --force ; make all;

Debugging:

vagrant ssh winw1 ; Start-Service *ovs* ; 
# check C:/openvswitch logs for details

Nothing in the openvswitch logs....

PS C:\Users\vagrant> cat C:\openvswitch\var\log\openvswitch\
cat : Could not find a part of the path 'C:\openvswitch\var\log\openvswitch\'.
At line:1 char:1
+ cat C:\openvswitch\var\log\openvswitch\
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\openvswitch\var\log\openvswitch\:String) [Get-Content], DirectoryN
   otFoundException
    + FullyQualifiedErrorId : GetContentReaderDirectoryNotFoundError,Microsoft.PowerShell.Commands.GetContentCommand

PS C:\Users\vagrant> ls C:\openvswitch\var\log\openvswitch\
PS C:\Users\vagrant> ls C:\openvswitch\var\log\openvswitch\
PS C:\Users\vagrant>

Full logs i see are:

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         7/4/2022  10:15 AM                net.d
7/4/2022 10:18 AM Installation log location: C:\k\antrea\install_ovs.log

7/4/2022 10:18 AM Downloading OVS package from https://downloads.antrea.io/ovs/ovs-2.15.2-antrea.0-win64.zip to C:\k\antrea\ovs-win64.zip

7/4/2022 10:18 AM Download OVS package success.

7/4/2022 10:18 AM Extract C:\k\antrea\ovs-win64.zip to C:\k\antrea

7/4/2022 10:18 AM Copying OVS package from C:\k\antrea\openvswitch to C:\openvswitch

7/4/2022 10:18 AM Installing OVS driver certificate.


PSPath                   : Microsoft.PowerShell.Security\Certificate::LocalMachine\TrustedPublisher\9E27F7B475263235DF5
                           A75A4936D0B21460F4630
PSParentPath             : Microsoft.PowerShell.Security\Certificate::LocalMachine\TrustedPublisher
PSChildName              : 9E27F7B475263235DF5A75A4936D0B21460F4630
PSIsContainer            : False
Archived                 : False
Extensions               : {System.Security.Cryptography.Oid, System.Security.Cryptography.Oid}
FriendlyName             :
IssuerName               : System.Security.Cryptography.X509Certificates.X500DistinguishedName
NotAfter                 : 10/12/2031 5:00:00 PM
NotBefore                : 10/13/2021 4:44:07 AM
HasPrivateKey            : False
PrivateKey               :
PublicKey                : System.Security.Cryptography.X509Certificates.PublicKey
RawData                  : {48, 130, 3, 18...}
SerialNumber             : 1F2362D86CB36598457D26A3A4317421
SubjectName              : System.Security.Cryptography.X509Certificates.X500DistinguishedName
SignatureAlgorithm       : System.Security.Cryptography.Oid
Thumbprint               : 9E27F7B475263235DF5A75A4936D0B21460F4630
Version                  : 3
Handle                   : 1963336863792
Issuer                   : CN="WDKTestCert appveyor,132785990465004813"
Subject                  : CN="WDKTestCert appveyor,132785990465004813"
EnhancedKeyUsageList     : {Code Signing (1.3.6.1.5.5.7.3.3)}
DnsNameList              : {WDKTestCert appveyor,132785990465004813}
SendAsTrustedIssuer      : False
EnrollmentPolicyEndPoint : Microsoft.CertificateServices.Commands.EnrollmentEndPointProperty
EnrollmentServerEndPoint : Microsoft.CertificateServices.Commands.EnrollmentEndPointProperty
PolicyId                 :


PSPath                   : Microsoft.PowerShell.Security\Certificate::LocalMachine\Root\9E27F7B475263235DF5A75A4936D0B2
                           1460F4630
PSParentPath             : Microsoft.PowerShell.Security\Certificate::LocalMachine\Root
PSChildName              : 9E27F7B475263235DF5A75A4936D0B21460F4630
PSIsContainer            : False
Archived                 : False
Extensions               : {System.Security.Cryptography.Oid, System.Security.Cryptography.Oid}
FriendlyName             :
IssuerName               : System.Security.Cryptography.X509Certificates.X500DistinguishedName
NotAfter                 : 10/12/2031 5:00:00 PM
NotBefore                : 10/13/2021 4:44:07 AM
HasPrivateKey            : False
PrivateKey               :
PublicKey                : System.Security.Cryptography.X509Certificates.PublicKey
RawData                  : {48, 130, 3, 18...}
SerialNumber             : 1F2362D86CB36598457D26A3A4317421
SubjectName              : System.Security.Cryptography.X509Certificates.X500DistinguishedName
SignatureAlgorithm       : System.Security.Cryptography.Oid
Thumbprint               : 9E27F7B475263235DF5A75A4936D0B21460F4630
Version                  : 3
Handle                   : 1963339904416
Issuer                   : CN="WDKTestCert appveyor,132785990465004813"
Subject                  : CN="WDKTestCert appveyor,132785990465004813"
EnhancedKeyUsageList     : {Code Signing (1.3.6.1.5.5.7.3.3)}
DnsNameList              : {WDKTestCert appveyor,132785990465004813}
SendAsTrustedIssuer      : False
EnrollmentPolicyEndPoint : Microsoft.CertificateServices.Commands.EnrollmentEndPointProperty
EnrollmentServerEndPoint : Microsoft.CertificateServices.Commands.EnrollmentEndPointProperty
PolicyId                 :

7/4/2022 10:18 AM Installing OVS kernel driver

7/4/2022 10:18 AM Hyper-V Virtual Machine Management service status: Running


C:\openvswitch\driver>netcfg -l .\ovsext.inf -c s -i OVSExt
Trying to install OVSExt ...

... .\ovsext.inf was copied to C:\Windows\INF\oem2.inf.

... done.


C:\openvswitch\driver>net stop vmms
The Hyper-V Virtual Machine Management service is stopping.
The Hyper-V Virtual Machine Management service was stopped successfully.


C:\openvswitch\driver>net start vmms
The Hyper-V Virtual Machine Management service is starting.
The Hyper-V Virtual Machine Management service was started successfully.

7/4/2022 10:18 AM Creating ovsdb file

7/4/2022 10:18 AM Create and start ovsdb-server service

[SC] CreateService SUCCESS
[SC] ChangeServiceConfig2 SUCCESS


Stderr from the command:

Start-Service : Service 'ovsdb-server (ovsdb-server)' cannot be started due to the following error: Cannot start
service ovsdb-server on computer '.'.
At C:\k\antrea\Install-OVS.ps1:214 char:5
+     Start-Service ovsdb-server
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
   ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

The text was updated successfully, but these errors were encountered:

wenyingd · 2022-07-06T02:00:13Z

Could you provide your steps to install Antrea and OVS?

jayunit100 · 2022-07-07T01:33:57Z

Hi wenying

The scripts in the repo run antrea install via powershell they are in the fork/ directory

Running make all you can see them run

Can add more details if that doesn't help or debug live together

wenyingd · 2022-07-11T02:34:20Z

@jayunit100 Thanks for providing the OVS installation logs. It looks the service of ovsdb-server fails to start which blocks the later installation steps. The issue should be related with several missing DLLs on for OpenSSL on the Windows host. Would you try to download and install this exe file (https://slproweb.com/download/Win64OpenSSL-3_0_5.exe) on the issued host, and then re-install OVS?

jayunit100 · 2022-07-13T14:28:42Z

right now we use "Win64OpenSSL-3_0_3.exe" ... is that ok ? or we need 3_0_5 ?

jayunit100 · 2022-07-13T14:30:29Z

@wenyingd how did you find that SSL error ?
testing w/
https://slproweb.com/download/Win64OpenSSL-3_0_5.exe now will let you know if it works.
meanwhile. @wenyingd ... Do you see any flaws with our existing installer for this ?
https://github.com/kubernetes-sigs/sig-windows-dev-tools/tree/master/sync/windows

jayunit100 · 2022-07-13T15:59:16Z

@wenyingd whats the "right" way to install OpenSSL from our powershell automation ?

jayunit100 · 2022-07-13T16:12:11Z

looks like

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

from @marosset working? will see if that can be used for ssh, maybe i can do same w ssl

jayunit100 · 2022-07-13T16:27:32Z

per mark rosetti something like

add openssl.ps1 script to https://github.com/kubernetes-sigs/sig-windows-dev-tools/tree/master/sync/windows
have it wget opeenssl msi
then use msiexec.exe w/ no-ui

will try that later, then will retry the antrea install

wenyingd · 2022-07-14T06:34:32Z

Having some offline discussions with @knabben , and we come to a problem when installing OpenSSL exe file with EULA , a popup is on the console to wait for user agreement input, which is not acceptable for the automation. And @knabben has another patch to copy the dll files directly to system path. But this might introduce conclict on different OS versions.

Personally, I prefer to install SSL dlls via some focusing software installation, but EULA popup is the biggest challenge for automation. Another thought to install OpenSSH could install the required dll files, but it also introduce other un-needed files to be installed.

wenyingd · 2022-07-14T06:42:21Z

I remembered @knabben used to say that the working version of openssl is 1.0.2u, which is not found from the original valid download link. I think that version is working because of no EULA stuck. So maybe we could try to download it from this link https://indy.fulgan.com/SSL/ ( I got a link to download openssl_x64_1.0.2u in this site), which is also a recommended link for Windows OpenSSL suggested for OVS installation.

jayunit100 · 2022-07-15T01:48:08Z

you mean https://indy.fulgan.com/SSL/openssl-1.0.2u-x64_86-win64.zip ?

jayunit100 · 2022-07-15T01:57:03Z

ok i uploaded https://storage.googleapis.com/jayunit100/openssl.exe to try it in the recipe along w the Open SSL license to the same buckettt... testing now

wenyingd · 2022-07-15T02:00:31Z

you mean https://indy.fulgan.com/SSL/openssl-1.0.2u-x64_86-win64.zip ?

Yes, this is a recommended link to download a version which works.

jayunit100 · 2022-07-16T01:13:36Z

that didnt seem to work. trying

choco install openssl --yes

... will update tomorrow...

PS @wenyingd ... why does Antrea need OpenSSL ?

jayunit100 · 2022-07-17T02:08:26Z

i didnt have any luck with this , either.
We'll need to dig further into: whats the right sustainable path to install ovsdb-server on windows nodes...

wenyingd · 2022-07-18T03:11:02Z

that didnt seem to work. trying
choco install openssl --yes
... will update tomorrow...

PS @wenyingd ... why does Antrea need OpenSSL ?

OpenSSL is required by OVS not Antrea. The OVS bits compilation needs OpenSSL support, so it also needs it in the runtime.

jayunit100 · 2022-07-19T03:09:10Z

so, when we say runtime here, why ? isnt OpenSSL just a protocol that is implemented by a library compiled into the binary for ovs? sorry for the naive question...

i dont think we should need to do things like choco install openssl and so on to put a .exe onto the host to run antrea agent... but maybe im missing something.

wenyingd · 2022-07-19T03:14:16Z

My word is not clear. My point is, when we run ovsdb-server/ovs-vswitch processes, they search for the dll files on the Windows host. But if they fail to find the dependent dlls, the bits are not able to run.

choco install openssl actually places the two dlls files "libeay32.dll" and "ssleay32.dll" to Windows system root path (C:\windows\system32), which is a path used for dll searching. So after the installation of openssl, ovsdb-server/ovs-vswitchd processes can find the dll files when we run them. We con't want ssl exe file itself, but OVS needs two dll files installed along with the exe.

dougsland · 2022-08-06T14:14:54Z

let me dig this one.
/assign

jayunit100 · 2022-08-07T14:38:19Z

thanks @dougsland !

jayunit100 · 2022-08-07T14:41:09Z

ok,

so the key requirement is "install DLL files for openSSL into "C:/windows/system32"
If we use choco install openssl, we then expect the 'ovsdb-server (ovsdb-server)' cannot be started due to the following error: Cannot start service ovsdb-server on computer '.'. error to go away ?

@wenyingd ? if so i think that will allow doug to finish implementing this.

QUESTION

@wenyingd ... In one of the experimenets above, I ran choco install openssl --yes but , it didnt seem to fix this issue - was i missing some detail

wenyingd · 2022-08-08T03:14:32Z

ok,

so the key requirement is "install DLL files for openSSL into "C:/windows/system32"

If we use choco install openssl, we then expect the 'ovsdb-server (ovsdb-server)' cannot be started due to the following error: Cannot start service ovsdb-server on computer '.'. error to go away ?

@wenyingd ? if so i think that will allow doug to finish implementing this.

QUESTION

@wenyingd ... In one of the experimenets above, I ran choco install openssl --yes but , it didnt seem to fix this issue - was i missing some detail

You are correct, the key requirement is "install DLL files for openSSL into "C:/windows/system32", one thing I want to add is, except for path C:/windows/system32, installing DLL files into path "c:/openvswitch/usr/sbin" also works to resolve the ovsdb-server start failure issue.

As for your question that choco install openssl --yes doesn't resolve the issue according to your experience, could you help check if the two files "ssleay32.dll and libeay32.dll" are located in path "C:\windows\system32". If no, this is possible not a valid resolution Besides. upstream antrea has merged a fixe for the issue (antrea-io/antrea#4027). So using the latest code in antrea main branch, no additional operations are expected to copy the dll files. But the fix is not involved in the release branches, so it may happen if the code in release branches are used.

jayunit100 · 2022-08-10T10:55:43Z

ok so , I think we're saying

that choco install openssl --yes might do the wrong path.
the fix in (Install-OVS.ps1 failed to start ovsdb service without ssl files on windows antrea-io/antrea#4027) will download OpenSSL FOR US so that if its not there, we just get a fresh installation

So, there are two possible fixes we can do?

wenyingd · 2022-08-10T10:57:23Z

ok so , I think we're saying

that choco install openssl --yes might do the wrong path.

the fix in (Install-OVS.ps1 failed to start ovsdb service without ssl files on windows antrea-io/antrea#4027) will download OpenSSL FOR US so that if its not there, we just get a fresh installation

So, there are two possible fixes we can do?

yes

dougsland · 2022-08-29T13:53:53Z

@wenyingd hey, tried updating the CNI to 1.8.0 and still no good. Please see: #204

If you can review anyway the patch, I believe we must update the CNI as it's far behind. Going to run a new tests tonight.

This was referenced Jul 19, 2022

Install-OVS.ps1 failed to start ovsdb service without ssl files on windows antrea-io/antrea#4027

Closed

doc magic constants for winpipe:// and unix:// to our ovsdb wrapper antrea-io/antrea#4033

Closed

k8s-ci-robot assigned dougsland Aug 6, 2022

dougsland mentioned this issue Aug 31, 2022

CNI: antrea uses different SA during deploy #205

Merged

k8s-ci-robot closed this as completed in #205 Aug 31, 2022

jayunit100 mentioned this issue Sep 13, 2023

Figure out a stable path for SSL installation in install-ovs.ps1 antrea-io/antrea#5479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

antrea OvS not installing #198

antrea OvS not installing #198

jayunit100 commented Jul 4, 2022 •

edited

Loading

wenyingd commented Jul 6, 2022

jayunit100 commented Jul 7, 2022

wenyingd commented Jul 11, 2022

jayunit100 commented Jul 13, 2022

jayunit100 commented Jul 13, 2022 •

edited

Loading

jayunit100 commented Jul 13, 2022

jayunit100 commented Jul 13, 2022 •

edited

Loading

jayunit100 commented Jul 13, 2022

wenyingd commented Jul 14, 2022

wenyingd commented Jul 14, 2022

jayunit100 commented Jul 15, 2022

jayunit100 commented Jul 15, 2022

wenyingd commented Jul 15, 2022

jayunit100 commented Jul 16, 2022

jayunit100 commented Jul 17, 2022

wenyingd commented Jul 18, 2022

jayunit100 commented Jul 19, 2022 •

edited

Loading

wenyingd commented Jul 19, 2022 •

edited

Loading

dougsland commented Aug 6, 2022

jayunit100 commented Aug 7, 2022

jayunit100 commented Aug 7, 2022 •

edited

Loading

wenyingd commented Aug 8, 2022 •

edited

Loading

QUESTION

jayunit100 commented Aug 10, 2022

wenyingd commented Aug 10, 2022

dougsland commented Aug 29, 2022 •

edited

Loading

antrea OvS not installing #198

antrea OvS not installing #198

Comments

jayunit100 commented Jul 4, 2022 • edited Loading

wenyingd commented Jul 6, 2022

jayunit100 commented Jul 7, 2022

wenyingd commented Jul 11, 2022

jayunit100 commented Jul 13, 2022

jayunit100 commented Jul 13, 2022 • edited Loading

jayunit100 commented Jul 13, 2022

jayunit100 commented Jul 13, 2022 • edited Loading

jayunit100 commented Jul 13, 2022

wenyingd commented Jul 14, 2022

wenyingd commented Jul 14, 2022

jayunit100 commented Jul 15, 2022

jayunit100 commented Jul 15, 2022

wenyingd commented Jul 15, 2022

jayunit100 commented Jul 16, 2022

jayunit100 commented Jul 17, 2022

wenyingd commented Jul 18, 2022

jayunit100 commented Jul 19, 2022 • edited Loading

wenyingd commented Jul 19, 2022 • edited Loading

dougsland commented Aug 6, 2022

jayunit100 commented Aug 7, 2022

jayunit100 commented Aug 7, 2022 • edited Loading

QUESTION

wenyingd commented Aug 8, 2022 • edited Loading

QUESTION

jayunit100 commented Aug 10, 2022

wenyingd commented Aug 10, 2022

dougsland commented Aug 29, 2022 • edited Loading

jayunit100 commented Jul 4, 2022 •

edited

Loading

jayunit100 commented Jul 13, 2022 •

edited

Loading

jayunit100 commented Jul 13, 2022 •

edited

Loading

jayunit100 commented Jul 19, 2022 •

edited

Loading

wenyingd commented Jul 19, 2022 •

edited

Loading

jayunit100 commented Aug 7, 2022 •

edited

Loading

wenyingd commented Aug 8, 2022 •

edited

Loading

dougsland commented Aug 29, 2022 •

edited

Loading