Instance startup failes to load org.glassfish.main.admin.gf-restadmin-connector bundle via OSGi

[Imadevops]

Payara Verison: payara-4.1.152.1
OS: CentOS 6.5 64bit
Java: 7u72

Every morning we have a job that will start the domain and instances on our Payara server that runs 'asadmin start-local-instance --sync full ' locally. Instances are started up sequentially and it is never the same instance that will fail to start when one does fail. We are starting between 4 and 8 local-instances.

Recently we have been having instances fail to start with the following stack trace snippet:

Caused by: org.osgi.framework.BundleException: Unable to acquire global lock for resolve.
at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3980)
at org.apache.felix.framework.Felix.startBundle(Felix.java:2037)
at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:955)
at org.jvnet.hk2.osgiadapter.OSGiModuleImpl.start(OSGiModuleImpl.java:210)

We are able to start the instance manually without a problem. We have not been able to reproduce this reliably and consistently. We have not been able to come up with an concrete reasoning behind why an instance will fail

I have provided the output from Payara at the point of the instance attempting to start this morning. I have scrubbed the DOMAIN and INSTANCE names in the document.

shortened_server.txt

[mikecroft]

Just off the top of my head - when you use --sync full, all of the configuration on every node is removed and replaced with configuration from the server. It is quite possible that this is the process that is failing.

From the GlassFish docs:

Caution - If the DAS is not running or is unreachable from the host where you are running this subcommand, do not set the --sync option to full. To perform a full synchronization, the subcommand removes the instance's cache. If the DAS cannot be contacted to replace the cache, the subcommand fails and the instance cannot be restarted until it is resynchronized with the DAS.

The alternative is to use --sync normal (which will only synchronise things which have changed since the last sync) or --sync none (which will synchronise nothing). I would suggest that normal is probably good enough for most cases, unless you have a specific reason to use full

[Imadevops]

Thanks @mikecroft . I am going to give --sync normal and --sync none and see what happens over the course of the next few weeks. We will also attempt to recreate this outside of our nightly processes by stopping and starting the DAS and instances on test servers constantly. I will update you on our results with both options.

[mikecroft]

Hi @Imadevops did you find any change with different --sync types?

[mikecroft]

Hi - is this still a problem?

[mikecroft]

Are you still seeing sporadic failures with different --sync types, @Imadevops? Should I close this issue?

[Imadevops]

The issue has gone away after taking your suggestions. We are working on upgrading the latest version of Payara as well to make sure that we are on the most up to date version. Thank you!