Geoevent upgrade to 11.3: "No Synchronization Service found in platform service"

[Biboba]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Module Version

• 4.3.0

Affected Resource(s)

• ArcGIS_ServerUpgrade

Configuration Files

AnymousGeoEventDevDeployment.txt

Expected Behavior

Should be able to run the configuration after having upgraded the GeoEvent component from 11.1 to 11.3 that is to say, the plateform service "Zookeeper (Synchronization_Service)" should be deployed on ArcGIS Server following upgrade.

Actual Behavior

After successful upgrade of Geovent from 11.1 to 11.3 by running:

Invoke-ArcGISConfiguration -ConfigurationParametersFile C:\pathto\GisGeoEventDevDeployment.json -Credential (Get-Credential) -Mode Upgrade

Running the configuration again not in mode "Upgrade":

Invoke-ArcGISConfiguration -ConfigurationParametersFile C:\pathto\GisGeoEventDevDeployment.json -Credential (Get-Credential)

returns the following error:

PowerShell DSC resource ArcGIS_GeoEvent failed to execute Test-TargetResource functionality with error message: No Synchronization Service found in platform service

Here are the last lines of the log before failing:

16.10.2024 11:35:26: [GISEVENTDEV13]: LCM:  [ Start  Resource ]  [[ArcGIS_xFirewall]GeoEvent_FirewallRules]
16.10.2024 11:35:26: [GISEVENTDEV13]: LCM:  [ Start  Test     ]  [[ArcGIS_xFirewall]GeoEvent_FirewallRules]
16.10.2024 11:35:26: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] TEST: Find rules with specified parameters
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] Get-FirewallRules: Found a Firewall Rule for Name: [ArcGISGeoEventFirewallRules] and DisplayGroup [ArcGIS GeoEvent Extension]
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] TEST: Check each defined parameter against the existing Firewall Rule - ArcGISGeoEventFirewallRules
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] Get-FirewallRuleProperty:  Get all the properties
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] Get-FirewallRuleProperty: Add filter info to rule map
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] Test-RuleHasProperties returning True
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_xFirewall]GeoEvent_FirewallRules] TEST: Returning True
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ End    Test     ]  [[ArcGIS_xFirewall]GeoEvent_FirewallRules]  in 4.6010 seconds.
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ Skip   Set      ]  [[ArcGIS_xFirewall]GeoEvent_FirewallRules]
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ End    Resource ]  [[ArcGIS_xFirewall]GeoEvent_FirewallRules]
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ Start  Resource ]  [[ArcGIS_GeoEvent]ArcGIS_GeoEvent]
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ Start  Test     ]  [[ArcGIS_GeoEvent]ArcGIS_GeoEvent]
16.10.2024 11:35:31: [GISEVENTDEV13]:                            [[ArcGIS_GeoEvent]ArcGIS_GeoEvent] Checking if WebSocketContextURL in sys props is wss://giseventdev13.company.com:6143/arcgis
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ End    Test     ]  [[ArcGIS_GeoEvent]ArcGIS_GeoEvent]  in 0.2270 seconds.
16.10.2024 11:35:31: [GISEVENTDEV13]: LCM:  [ End    Set      ]
16.10.2024 11:35:31: L’opération « Invoquer une méthode CIM » est terminée.

Checking myself the Admin API of the ArcGIS Server, I confirm that in "admin/system/platformservices", there is only "Spark (Compute_Platform)" and the "Zookeeper (Synchronization_Service)" plateform service is missing.

Steps to Reproduce

1. Install and configure Geoevent 11.1.
2. Upgrade to 11.3 using this library in upgrade mode.
3. Run the configuration again without the upgrade mode once at 11.3
4. "No Synchronization Service found in platform service"

Important Factoids

Upgrading manually Geoevent from 11.1 to 11.3 works: the "Zookeeper (Synchronization_Service)" plateform service exists after the upgrade.

References

I wonder if this issue could not be related to our private cloud Openstack. Last time I had an issue with ArcGIS Powershell DSC that I could not reproduce manually, it was the following issue:
#132

Any idea ? What could "unregister" this plateform service ?

Thanks !

/cc @ceddc

[Biboba]

It took me a while to figure out what was going on but here are the results:

• I understood that it was related to the upgrade operation: if ArcGIS Powershell dsc was doing the install but the upgrade was done manually avoiding the following line, it works I had the 2 platform services:
  

  arcgis-powershell-dsc/Modules/ArcGIS/DSCResources/ArcGIS_ServerUpgrade/ArcGIS_ServerUpgrade.psm1

  Line 110 in 4e1ea91

  $Response = Invoke-ArcGISWebRequest -Url $ServerUpgradeUrl -HttpFormParameters @{f = 'json';runAsync='true'} -Referer $Referer -Verbose

• Thinking the upgrade request was done somehow differently, I recorded with Fiddler the request of the upgrade done manually from the server manager (the one that works, ie I have the 2 platforms service at the end of the upgrade) and replayed it again after the installation done by ArcGIS Powershell dsc: it fails once again. So I came to the conclusion that by visiting the ArcGIS Manager, it had side consequences.

• The manager is making the following requests:

https://geoeventserver.company.com:6443/arcgis/rest/info?f=json
https://geoeventserver.company.com:6443/arcgis/admin/security/config?f=json
https://geoeventserver.company.com:6443/arcgis/admin/mode?f=json
https://geoeventserver.company.com:6443/arcgis/admin/?f=json
https://geoeventserver.company.com:6443/arcgis/admin/publicKey?f=json
https://geoeventserver.company.com:6443/arcgis/admin/info?f=json
https://geoeventserver.company.com:6443/arcgis/admin/local?f=json

I tested them one by one:

1. Install GeoEvent 11.3 with ArcGIS Powershell dsc
2. Make a GET request to one of the URL above
3. Make the upgrade request

• I found out that visiting the following URL:

https://geoeventserver.company.com:6443/arcgis/admin/local?f=json

before making the upgrade request, it fixes the issue. So this GET request to arcgis/admin/local has somehow side consequences !

• Adding the following lines to the library just before the upgrade request in ArcGIS/DSCResources/ArcGIS_ServerUpgrade/ArcGIS_ServerUpgrade.psm1#L110, solves the issue !

[string]$ServerLocalUrl = $ServerSiteURL.TrimEnd('/') + "/arcgis/admin/local"
Write-Verbose "Making request to $ServerLocalUrl to fix synchronisation platefeform missing after upgrading the site"
Invoke-ArcGISWebRequest -Url $ServerLocalUrl -HttpFormParameters @{f = 'json'} -Referer $Referer -Verbose

Do you observe the phenomena on your side ?

Thanks

[cameronkroeker]

Hi @Biboba,

Do you use or need the ZooKeeper platform service? I believe this is only needed or used in certain workflows with GeoAnalytics. From my understanding, the ArcGIS GeoEvent Server used to use the ArcGIS Server's ZooKeeper platform service prior to 10.6, which is why the PowerShell Module is checking for it. However, starting at 10.6 and above GeoEvent no longer uses it. So we may look into updating the logic in the module to no longer check for it since its no longer needed for GeoEvent.

I was able to reproduce the issue outside of PowerShell by manually performing the upgrade via the ArcGIS Server admin endpoint as well. In ArcGIS Server logs I see the following entry:

<Msg time="2024-10-22T09:11:47,846" type="DEBUG" code="9999" source="Admin" process="11848" thread="1" methodName="" machine="Machine.domain.com" user="" elapsed="" requestID="e535a662-7f3c-4c24-9a58-10aa1a8ba24b">Spark upgraded and registered with the site {"id":"76270df1-be12-47bc-94a3-dc01604881cb","type":"COMPUTE_PLATFORM","provider":"Spark","info":{"secretKey":"{crypt}Ru8Fub4JH9gQXnCrAjiJQqRIAZ2bgeIeJRV+0o8PI/Mm7U2FjeFCj76uri8KZBio","clusterURL":"spark://Machine.domain.com:7077","enableWebUI":false,"ssl":false,"version":"3.5.1","minutesBeforeRestart":4}}</Msg>
<Msg time="2024-10-22T09:11:47,849" type="DEBUG" code="9999" source="Admin" process="11848" thread="1" methodName="" machine="Machine.domain.com" user="" elapsed="" requestID="e535a662-7f3c-4c24-9a58-10aa1a8ba24b">Error while configuring platform services </Msg>
<Msg time="2024-10-22T09:11:47,849" type="DEBUG" code="9999" source="Admin" process="11848" thread="1" methodName="" machine="Machine.domain.com" user="" elapsed="" requestID="e535a662-7f3c-4c24-9a58-10aa1a8ba24b">java.lang.NullPointerException: Cannot invoke "com.esri.arcgis.discovery.admin.machine.ServerMachineConfig.getMachineName()" because the return value of "com.esri.arcgis.discovery.admin.machine.ServerMachineConfigurator.getServerMachineConfig()" is null
	at com.esri.arcgis.discovery.admin.system.platformservices.ZooKeeper.e(ZooKeeper.java:547)
	at com.esri.arcgis.discovery.admin.system.platformservices.ZooKeeper.a(ZooKeeper.java:703)
	at com.esri.arcgis.discovery.admin.system.platformservices.ZooKeeper.createConnectionString(ZooKeeper.java:698)
	at com.esri.arcgis.discovery.admin.SiteManager$PlatformServicesModule$c.&lt;init&gt;(SiteManager$PlatformServicesModule$c.java:6336)
	at com.esri.arcgis.discovery.admin.SiteManager$PlatformServicesModule.c(SiteManager.java:6330)
	at com.esri.arcgis.discovery.admin.SiteManager$PlatformServicesModule.configure(SiteManager.java:6415)
	at com.esri.arcgis.discovery.admin.SiteManager.upgrade(SiteManager.java:10546)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at com.esri.arcgis.discovery.admin.system.AsyncJobsManager$a.run(AsyncJobsManager$a.java:280)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
</Msg>
<Msg time="2024-10-22T09:11:47,849" type="INFO" code="7538" source="Admin" process="11848" thread="1" methodName="" machine="Machine.domain.com" user="" elapsed="0.0" requestID="e535a662-7f3c-4c24-9a58-10aa1a8ba24b">ArcGIS Server upgrade step 2 of 2 : Configured platform services.</Msg>

I suspect the workaround you found works because the /admin/local?f=json pre-loads the machineName avoiding the null exception.

[Biboba]

Hi @cameronkroeker,

Thanks for your reply and for the reproduction.
To be honest, I did not know the existence of ZooKeeper platform service before this issue. So I don't use it or need it but I thought it was maybe required/used by Geoevent without my knowledge.
Don't you think it should be logged by technical support somehow ? It's a bit weird this GET request that initializes the machineName... maybe it has other consequences than the platform service being missing.

Thanks !

[cameronkroeker]

Hi @cameronkroeker,

Thanks for your reply and for the reproduction. To be honest, I did not know the existence of ZooKeeper platform service before this issue. So I don't use it or need it but I thought it was maybe required/used by Geoevent without my knowledge. Don't you think it should be logged by technical support somehow ? It's a bit weird this GET request that initializes the machineName... maybe it has other consequences than the platform service being missing.

Thanks !

Hi @Biboba,

Yes, it should be logged as an ArcGIS Server bug with Esri Technical Support. I have also observed this same behavior with upgrading a standalone ArcGIS Server without GeoEvent. In the next release of the PowerShell module we will make the adjustment of no longer checking for the platform service when deploying ArcGIS GeoEvent Server since its no longer used/needed since 10.6.

Thanks,
Cameron K.

[cameronkroeker]

In the DSC ArcGIS Module v4.4.0 we did 2 things to resolve this.

1. Prior to upgrade a GET request will be sent to https://geoeventserver.company.com:6443/arcgis/admin/local?f=json
   

   arcgis-powershell-dsc/Modules/ArcGIS/DSCResources/ArcGIS_ServerUpgrade/ArcGIS_ServerUpgrade.psm1

   Lines 106 to 108 in fa485d2

   	[string]$ServerLocalUrl = $ServerSiteURL.TrimEnd('/') + "/arcgis/admin/local"
   	Write-Verbose "Making request to $ServerLocalUrl before upgrading the site"
   	Invoke-ArcGISWebRequest -Url $ServerLocalUrl -HttpFormParameters @{f = 'json'} -Referer $Referer -Verbose

2. Removed the platform service check as it was legacy code that is no longer needed/used

https://github.com/Esri/arcgis-powershell-dsc/releases/tag/v4.4.0