VSphere : option to enable soap client debugging #2442

dloucasfx · 2022-09-02T17:36:13Z

The VSphere monitor is hard to debug as the log messages don’t provide markers to where the code has failed, it only returns the low level error from the SDK.

For example, from this error, it’s almost impossible to know which API call caused it.

Aug 30 17:28:41 dtvadb02.bbldtl.int otelcol[31973]: 2022-08-30T17:28:41.873+1000        error        vsphere/runner.go:34        firstTimeSetup failed        {"kind": "receiver", "name": "smartagent/vsphere", "pipeline": "metrics", "monitorType":        "vsphere", "error": "Post \"https://xxxxxxx/sdk\": Forbidden"}

This bug is to allow the user to turn debugging on the SOAP client so we can see the SOAP requests/responses

Signed-off-by: Dani Louca dlouca@splunk.com

Signed-off-by: Dani Louca <dlouca@splunk.com>

pmcollins

Thanks for making this change.

crobert-1 · 2022-09-06T23:25:36Z

Would it be possible to add a test, or at least include some kind of example output when debugging is turned on? I'm pretty unfamiliar with the module being used, so I can't really tell if it's working or what the debugging output would look like.

dloucasfx · 2022-09-06T23:58:46Z

Would it be possible to add a test, or at least include some kind of example output when debugging is turned on? I'm pretty unfamiliar with the module being used, so I can't really tell if it's working or what the debugging output would look like.

Thanks for looking Curtis!
It's not trivial to add a test for this and not worth the effort as this is monitor going away and eventually replaced by the native VCenter OTEL Receiver, besides this will only be used for debugging API calls that fail, which is not very common.
Here is an example of what the logs will look like once this option is enabled:

time="2022-09-02T12:35:32-04:00" level=info msg="POST /sdk HTTP/1.1\r\nHost: 127.0.0.1:8989\r\nContent-Type: text/xml; charset=\"utf-8\"\r\nSoapaction: urn:vim25/7.0\r\nUser-Agent: govmomi/0.29.0 (go1.19;darwin;amd64)\r\n\r\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<Envelope xmlns=\"http://schemas.xmlsoap.org/soap/envelope/\"><Body><RetrieveServiceContent xmlns=\"urn:vim25\"><_this type=\"ServiceInstance\">ServiceInstance</_this></RetrieveServiceContent></Body></Envelope>" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="2022-09-02T12-35-32.826831000 - 0001: " monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="    12ms (*methods.RetrieveServiceContentBody)" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nContent-Type: text/xml; charset=utf-8\r\nDate: Fri, 02 Sep 2022 16:35:32 GMT\r\n\r\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<soapenv:Envelope xmlns:soapenc=\"http://schemas.xmlsoap.org/soap/encoding/\" xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><soapenv:Body><RetrieveServiceContentResponse xmlns=\"urn:vim25\"><returnval><rootFolder type=\"Folder\">group-d1</rootFolder><propertyCollector type=\"PropertyCollector\">propertyCollector</propertyCollector><viewManager type=\"ViewManager\">ViewManager</viewManager><about><name>VMware vCenter Server</name><fullName>VMware vCenter Server 6.5.0 build-5973321 (govmomi simulator)</fullName><vendor>VMware, Inc.</vendor><version>6.5.0</version><build>5973321</build><localeVersion>INTL</localeVersion><localeBuild>000</localeBuild><osType>linux-amd64</osType><productLineId>vpx</productLineId><apiType>VirtualCenter</apiType><apiVersion>6.5</apiVersion><instanceUuid>dbed6e0c-bd88-4ef6-b594-21283e1c677f</instanceUuid><licenseProductName>VMware VirtualCenter Server</licenseProductName><licenseProductVersion>6.0</licenseProductVersion></about><setting type=\"OptionManager\">VpxSettings</setting><userDirectory type=\"UserDirectory\">UserDirectory</userDirectory><sessionManager type=\"SessionManager\">SessionManager</sessionManager><authorizationManager type=\"AuthorizationManager\">AuthorizationManager</authorizationManager><serviceManager type=\"ServiceManager\">ServiceMgr</serviceManager><perfManager type=\"PerformanceManager\">PerfMgr</perfManager><scheduledTaskManager type=\"ScheduledTaskManager\">ScheduledTaskManager</scheduledTaskManager><alarmManager type=\"AlarmManager\">AlarmManager</alarmManager><eventManager type=\"EventManager\">EventManager</eventManager><taskManager type=\"TaskManager\">TaskManager</taskManager><extensionManager type=\"ExtensionManager\">ExtensionManager</extensionManager><customizationSpecManager type=\"CustomizationSpecManager\">CustomizationSpecManager</customizationSpecManager><customFieldsManager type=\"CustomFieldsManager\">CustomFieldsManager</customFieldsManager><diagnosticManager type=\"DiagnosticManager\">DiagMgr</diagnosticManager><licenseManager type=\"LicenseManager\">LicenseManager</licenseManager><searchIndex type=\"SearchIndex\">SearchIndex</searchIndex><fileManager type=\"FileManager\">FileManager</fileManager><datastoreNamespaceManager type=\"DatastoreNamespaceManager\">DatastoreNamespaceManager</datastoreNamespaceManager><virtualDiskManager type=\"VirtualDiskManager\">virtualDiskManager</virtualDiskManager><snmpSystem type=\"HostSnmpSystem\">SnmpSystem</snmpSystem><vmProvisioningChecker type=\"VirtualMachineProvisioningChecker\">ProvChecker</vmProvisioningChecker><vmCompatibilityChecker type=\"VirtualMachineCompatibilityChecker\">CompatChecker</vmCompatibilityChecker><ovfManager type=\"OvfManager\">OvfManager</ovfManager><ipPoolManager type=\"IpPoolManager\">IpPoolManager</ipPoolManager><dvSwitchManager type=\"DistributedVirtualSwitchManager\">DVSManager</dvSwitchManager><hostProfileManager type=\"HostProfileManager\">HostProfileManager</hostProfileManager><clusterProfileManager type=\"ClusterProfileManager\">ClusterProfileManager</clusterProfileManager><complianceManager type=\"ProfileComplianceManager\">MoComplianceManager</complianceManager><localizationManager type=\"LocalizationManager\">LocalizationManager</localizationManager><storageResourceManager type=\"StorageResourceManager\">StorageResourceManager</storageResourceManager><guestOperationsManager type=\"GuestOperationsManager\">guestOperationsManager</guestOperationsManager><overheadMemoryManager type=\"OverheadMemoryManager\">OverheadMemoryManager</overheadMemoryManager><certificateManager type=\"CertificateManager\">certificateManager</certificateManager><ioFilterManager type=\"IoFilterManager\">IoFilterManager</ioFilterManager><vStorageObjectManager type=\"VcenterVStorageObjectManager\">VStorageObjectManager</vStorageObjectManager><hostSpecManager type=\"HostSpecificationManager\">HostSpecificationManager</hostSpecManager><cryptoManager type=\"CryptoManagerKmip\">CryptoManager</cryptoManager><healthUpdateManager type=\"HealthUpdateMan" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="ager\">HealthUpdateManager</healthUpdateManager><failoverClusterConfigurator type=\"FailoverClusterConfigurator\">FailoverClusterConfigurator</failoverClusterConfigurator><failoverClusterManager type=\"FailoverClusterManager\">FailoverClusterManager</failoverClusterManager></returnval></RetrieveServiceContentResponse></soapenv:Body></soapenv:Envelope>" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="POST /sdk HTTP/1.1\r\nHost: 127.0.0.1:8989\r\nContent-Type: text/xml; charset=\"utf-8\"\r\nSoapaction: urn:vim25/7.0\r\nUser-Agent: govmomi/0.29.0 (go1.19;darwin;amd64)\r\n\r\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<Envelope xmlns=\"http://schemas.xmlsoap.org/soap/envelope/\"><Body><Login xmlns=\"urn:vim25\"><_this type=\"SessionManager\">SessionManager</_this><userName>user</userName><password>********</password><locale>en_US</locale></Login></Body></Envelope>" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="2022-09-02T12-35-32.830758000 - 0002: " monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="     2ms (*methods.LoginBody)" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="HTTP/1.1 200 OK\r\nContent-Length: 821\r\nContent-Type: text/xml; charset=utf-8\r\nDate: Fri, 02 Sep 2022 16:35:32 GMT\r\nSet-Cookie: vmware_soap_session=fc061d91-7ee3-47ae-ac39-518a9c3bab94; HttpOnly\r\n\r\n" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<soapenv:Envelope xmlns:soapenc=\"http://schemas.xmlsoap.org/soap/encoding/\" xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><soapenv:Body><LoginResponse xmlns=\"urn:vim25\"><returnval><key>fc061d91-7ee3-47ae-ac39-518a9c3bab94</key><userName>user</userName><fullName>user</fullName><loginTime>2022-09-02T16:35:32.838822442Z</loginTime><lastActiveTime>2022-09-02T16:35:32.838835174Z</lastActiveTime><locale>en_US</locale><messageLocale>en_US</messageLocale><extensionSession>false</extensionSession><ipAddress>172.17.0.1</ipAddress><userAgent>govmomi/0.29.0 (go1.19;darwin;amd64)</userAgent><callCount>1</callCount></returnval></LoginResponse></soapenv:Body></soapenv:Envelope>" monitorID=1 monitorType=vsphere
time="2022-09-02T12:35:32-04:00" level=info msg="POST /sdk HTTP/1.1\r\nHost: 127.0.0.1:8989\r\nContent-Type: text/xml; charset=\"utf-8\"\r\nSoapaction: urn:vim25/7.0\r\nUser-Agent: govmomi/0.29.0 (go1.19;darwin;amd64)\r\n\r\n" monitorID=1 monitorType=vsphere

Full log file https://drive.google.com/file/d/1HVvu701xV3sDN5PhEsUhmOud4srunZ_R/view?usp=sharing

crobert-1 · 2022-09-07T00:02:56Z

It's not trivial to add a test for this and not worth the effort as this is monitor going away and eventually replaced by the native VCenter OTEL Receiver, besides this will only be used for debugging API calls that fail, which is not very common.
Here is an example of what the logs will look like once this option is enabled:

That's a fair point regarding testing. Thanks for including some example output, it helps!

option to enable soap client debugging

fd503ea

Signed-off-by: Dani Louca <dlouca@splunk.com>

dloucasfx requested review from a team, keitwb, asuresh4 and rmfitzpatrick as code owners September 2, 2022 17:36

pmcollins approved these changes Sep 2, 2022

View reviewed changes

dloucasfx changed the title ~~option to enable soap client debugging~~ VSphere : option to enable soap client debugging Sep 6, 2022

crobert-1 approved these changes Sep 7, 2022

View reviewed changes

rmfitzpatrick approved these changes Sep 7, 2022

View reviewed changes

rmfitzpatrick merged commit 0c87d89 into main Sep 7, 2022

atoulme deleted the SWAT-5009 branch October 10, 2022 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VSphere : option to enable soap client debugging #2442

VSphere : option to enable soap client debugging #2442

dloucasfx commented Sep 2, 2022

pmcollins left a comment

crobert-1 commented Sep 6, 2022

dloucasfx commented Sep 6, 2022

crobert-1 commented Sep 7, 2022

VSphere : option to enable soap client debugging #2442

VSphere : option to enable soap client debugging #2442

Conversation

dloucasfx commented Sep 2, 2022

pmcollins left a comment

Choose a reason for hiding this comment

crobert-1 commented Sep 6, 2022

dloucasfx commented Sep 6, 2022

crobert-1 commented Sep 7, 2022