PAYARA-486 - Fixed wrong session count after failover

[MeroRai]

No description provided.

[payara-ci]

Can one of the admins verify this patch?

[payara-ci]

Can one of the admins verify this patch?

[smillidge]

jenkins test please

[payara-ci]

All tests have passed

[hdhoang]

At random time, I have threads stuck around the HashMap (mostly .put) and BoundedFifoBuffer (in AbstractCollection.remove) added in this PR. I think sessionTimestamps and buffer should have synchronization around them, as activeSessionsCount does.

Here are some more information from server.log and jstack. Soon after this exception appears in server.log, more and more threads will be stuck with the following stack trace, with sessionCreatedEvent, sessionDestroyedEvent, and sessionActivatedEndEvent:

[2016-11-11T14:34:23.172+0000] [Payara 4.1] [INFO] [] [] [tid: _ThreadID=703 _ThreadName=ajp-thread-pool(14)] [timeMillis: 1478842463172] [levelValue: 800] [[
  org.glassfish.flashlight.impl.client.ReflectiveClientInvoker
java.lang.reflect.InvocationTargetException
CAUSE:  java.lang.ArrayIndexOutOfBoundsException: 11
id=47
target=org.glassfish.web.admin.monitor.SessionStatsProvider@436edf80
method=public void org.glassfish.web.admin.monitor.SessionStatsProvider.sessionCreatedEvent(java.lang.String,java.lang.String,java.lang.String)
paramNames=[sessionId, appName, hostName]
probeIndices=[0, 1, 2]
useProbeArgs=true
hasComputedParams=false]]

"ajp-thread-pool(23)" daemon prio=10 tid=0x00007f83c00e1800 nid=0x589f runnable [0x00007f8465d51000]
   java.lang.Thread.State: RUNNABLE
 at org.glassfish.web.admin.monitor.BoundedFifoBuffer.access$300(BoundedFifoBuffer.java:61)
 at org.glassfish.web.admin.monitor.BoundedFifoBuffer$1.next(BoundedFifoBuffer.java:332)
 at java.util.AbstractCollection.remove(AbstractCollection.java:290)
 at org.glassfish.web.admin.monitor.SessionStatsProvider.sessionDestroyedEvent(SessionStatsProvider.java:199)
 at sun.reflect.GeneratedMethodAccessor4056.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.glassfish.flashlight.impl.client.ReflectiveClientInvoker.methodInvoke(ReflectiveClientInvoker.java:173)
 at org.glassfish.flashlight.impl.client.ReflectiveClientInvoker.invoke(ReflectiveClientInvoker.java:155) at org.glassfish.flashlight.provider.FlashlightProbe.fireProbe(FlashlightProbe.java:180)
 at org.glassfish.flashlight.provider.ProbeRegistry.invokeProbe(ProbeRegistry.java:134)
 at org.glassfish.web.admin.monitor.SessionProbeProvider.sessionDestroyedEvent(SessionProbeProvider.java)
 at com.sun.enterprise.web.WebModule.sessionDestroyedEvent(WebModule.java:2221)
.....elided.....

Those threads quickly pile up and consume all CPUs.

At other time, one thread gets stuck like this, and spins on one core:

"ajp-thread-pool(1)" daemon prio=10 tid=0x00007f92582b1000 nid=0xb19 runnable [0x00007f921b0ee000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:494)
        at org.glassfish.web.admin.monitor.SessionStatsProvider.sessionCreatedEvent(SessionStatsProvider.java:174)
        at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.flashlight.impl.client.ReflectiveClientInvoker.methodInvoke(ReflectiveClientInvoker.java:173)
        at org.glassfish.flashlight.impl.client.ReflectiveClientInvoker.invoke(ReflectiveClientInvoker.java:155)
        at org.glassfish.flashlight.provider.FlashlightProbe.fireProbe(FlashlightProbe.java:180)
        at org.glassfish.flashlight.provider.ProbeRegistry.invokeProbe(ProbeRegistry.java:134)
        at org.glassfish.web.admin.monitor.SessionProbeProvider.sessionCreatedEvent(SessionProbeProvider.java)
        at com.sun.enterprise.web.WebModule.sessionCreatedEvent(WebModule.java:2215)
.....elided....

Gradually, more threads fall into this state, each consumes one core fully. There are no relevant output in server.log

[smillidge]

I've created an internal issue PAYARA-1224 for this to make it thread safe.