Bean identifier index inconsistency after Hot re-deployment from NetBeans

[ratcashdev]

Editing a CDI bean in netBeans and hitting save redeploys the application in GF.
After visiting a JAX-RS resoruce using the CDI bean we get the error below:

<pre>org.jboss.weld.exceptions.IllegalStateException: WELD-000227: Bean identifier index inconsistency detected - the distributed container probably does not work with identical applications
Expected hash: 813144575
Current index: BeanIdentifierIndex [hash=2099704167, indexed=12]:
   0: WELD&#37;AbstractSyntheticBean&#37;JSR196OAuthLogin-1.0-SNAPSHOT&#37;HttpSession
   1: WELD&#37;AbstractSyntheticBean&#37;com.ibm.jbatch.container.cdi.BatchCDIInjectionExtension&#37;HttpSession
   2: WELD&#37;AbstractSyntheticBean&#37;com.sun.faces.flow.FlowCDIExtension&#37;HttpSession
   3: WELD&#37;AbstractSyntheticBean&#37;fish.payara.cdi.jsr107.PayaraHazelcastCDIExtension&#37;HttpSession
   4: WELD&#37;AbstractSyntheticBean&#37;org.glassfish.cdi.transaction.TransactionalExtension&#37;HttpSession

[ratcashdev]

Likely cause of the error is the following code:

BeanManager beanManager = getBeanManager();
        Bean<MyBean> bean = (Bean<MyBean>) beanManager.resolve(beanManager.getBeans(MyBean.class));
        CreationalContext ctx = beanManager.createCreationalContext(bean);
        MyBean mBean= (MyBean) beanManager.getReference(bean, MyBean.class, ctx);

This snippet is used inside a ServerAuthModule. I am trying to release the bean using
ctx.release(); as well as bean.destroy(mBean,ctx); but it does not help.
If I stop trying to use CDI in the ServerAuthModule alltogether, then WELD is not complaining after hot-redeploy. However, I need to get this working.

[smillidge]

How are you registering/deregistering the JASPIC SAM in your code?

[ratcashdev]

The JASPIC code/jar is copied to the glassfish/lib folder and registered as a MessageSecurity through the admin console. Instructions taken from here: https://www.ibm.com/developerworks/java/library/j-authenticroast/ (see section container configuration).

The actual JASPIC code that is deployed deployed is here: https://github.com/ratcashdev/authenticroast
You'll have to compile it yourself. Make sure to copy both the API and the IMPL jars.
The app through which I test: https://github.com/ratcashdev/JSR196OAuthLogin
(has a dependency on the API)

Try to change the SampleOAuth.java class while it's deployed 2-3 times, and do
GET /api/test/secure/oauth
with the header: Authorization: BEARER mytoken

BTW: Is there a more straightforward way to register the JASPIC code without messing inside the admin console?

[smillidge]

Take a look here for an example of how to register a SAM programmatically

https://github.com/javaee-samples/javaee7-samples/blob/master/jaspic/custom-principal/src/main/java/org/javaee7/jaspic/customprincipal/sam/SamAutoRegistrationListener.java

[ratcashdev]

Tried with the code from the sample (JaspicUtils). Thank you for pointing me to this solution. Simplifies it a lot. However, it did not help with the exception I was getting in the OP.

I have even tried to register a bean (as a JASPIC ServerAuthModule) that was injected by CDI into my @WebListener but after some edit of SampleOAuth.java and redeploy I got the same result: Bean identifier index inconsistency

[smillidge]

Can you try removing the registration of the SAM module on undeploy?

[ratcashdev]

Unfortunately, even after a successful deregistration of the SAM module and redeploy the exception remains.
BTW: all sample codes with programmatic registration completely ignore the request policy (which in this case is 'null'). Is it really so, that this is completely useless data (even if I found a way to set it)?

Edit: I think I got closer:
Severe: The web application [/JSR196OAuthLogin] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@298e8b71]) and a value of type [name.aikesommer.authenticator.DefaultPrincipalStore] (value [name.aikesommer.authenticator.DefaultPrincipalStore@2dffca54]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Edit2: eliminated all the ThreadLocals in the old code. The exception still remains.

[ratcashdev]

Tried replaced the weld-osgi-bundle jar with a newer version (2.2.10-2.2.12 Final), but it did not help.

[ratcashdev]

Hi Steve, is there anything else I can do to help replicate/solve this issue?

[smillidge]

We just need to try and replicate the issue. We haven't had a chance to replicate with your test application yet. I'll let you know.

[smillidge]

Hi I've tried to reproduce this but can't can you provide me step by step instructions?
I have compiled the AuthenticRoast jars and copied them into the GlassFish lib directory.
I have compiled your war and opened it in NetBeans
I have ran your war from NetBeans so it fires up Payara and deploys the application.
I have then used curl
curl -G -H "Authorization: BEARER mytoken" http://127.0.0.1:8080/JSR196OAuthLogin/api/test/secure/oauth

To access the application and I get a 401 UnAuthorized error and the server log is;
Info: Retrieving main CDI bean.
Info: Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@f4dfa38
Severe: No valid EE environment for injection of name.aikesommer.authenticator.DefaultPrincipalStore

I have then edited SampleOAuth and the application redeploys with server log

Info: RegId = __311_HttpServletserver /JSR196OAuthLogin
Info: result = true
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
Info: visiting unvisited references
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] public org.glassfish.jms.injection.JMSCDIExtension.processAnnotatedType(@observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] org.glassfish.sse.impl.ServerSentEventCdiExtension.processAnnotatedType(@observes ProcessAnnotatedType, BeanManager) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] org.glassfish.sse.impl.ServerSentEventCdiExtension.processAnnotatedType(@observes ProcessAnnotatedType, BeanManager) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-000411: Observer method [BackedAnnotatedMethod] public org.glassfish.jms.injection.JMSCDIExtension.processAnnotatedType(@observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
WARN: WELD-001440: Scope type @javax.enterprise.context.SessionScoped() used on injection point [BackedAnnotatedField] @Inject @SessionScoped com.ratcash.multilogin.authenticators.AppAuthModule.store
Info: registration returned: __311_HttpServletserver /JSR196OAuthLogin
Info: Loading application [JSR196OAuthLogin] at [/JSR196OAuthLogin]
Info: JSR196OAuthLogin was successfully deployed in 1,210 milliseconds.

Severe: No valid EE environment for injection of com.ratcash.multilogin.authenticators.AppAuthModule
Info: Cr eated
Info: Retrieving main CDI bean.
Info: Cr eated
Severe: No valid EE environment for injection of com.ratcash.multilogin.authenticators.SampleOAuth
Info: Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@75014671
Severe: No valid EE environment for injection of name.aikesommer.authenticator.DefaultPrincipalStore

I have done this multiple times but not received the IllegalStateException
I then use curl again and get the same 401 response and the server log is

[smillidge]

BTW
Scope type @javax.enterprise.context.SessionScoped() used on injection point [BackedAnnotatedField] @Inject @SessionScoped com.ratcash.multilogin.authenticators.AppAuthModule.store

I don't think you should put a scope annotation on an injection point

[ratcashdev]

@Inject: you're right. I misinterpreted some other sources on the internet. The definitive seems to be https://docs.oracle.com/javaee/6/tutorial/doc/gmgil.html

[ratcashdev]

As for the Sample app: commit #d386dc2a2f192f3d519f21f9ca12326c4ab48527 needs the jars in the lib folder of glassfish, however, current master doesn't (does the programmatic-registration)
To get the exception, I merely changed the strings inside System.out.println statements to trigger hotdeployment (added or removed a space character):
System.out.println("Called. Token=" + access_token);

The third redeploy attempt always caused the exception. I will try it on a virgin installation of payara.

[smillidge]

OK thanks I will try deleting the jars from the lib directory and see what happens

[smillidge]

wrt to @Inject you need to create a producer method like;

https://github.com/payara/Payara/blob/master/appserver/payara-appserver-modules/payara-jsr107/src/main/java/fish/payara/cdi/jsr107/JSR107Producer.java

to get the non-CDI object injected

[ratcashdev]

behaves the same on a virgin installation of Payara 4.1.152.1 with the embedded domain 'domain1'. What's even more disturbing, that after some edits, back and forth, suddenly it works again, regardless of the file's actual content (i.e. whether it actually equals to the original content when deployed for the first time).
Windows 7, 64bit, with Netbeans 8.0.2, java 1.8.0_45

[smillidge]

I removed the jars from the lib directory and redeployed via NetBeans many times and can not reproduce the issue on Linux 64bit 1.8.0_45-b14 + NetBeans 8.0.2 + Payara 4.1.153-SNAPSHOT

[smillidge]

I'm afraid also can't reproduce on Payara 4.1.152.1 on Linux

[ratcashdev]

Strange. The only notable difference seems to be the OS, but I strongly doubt it could cause these kind of issues. What else do you suggest to try/check/eliminate?

[ratcashdev]

As a matter of fact, I can reproduce the issue on two different laptops with Windows 7. One DOES have an anti-virus, the other doesn't. Maybe some windows services (background shadow copy, defender, whatnot) are interfering? But in such a case a lot of ppl should have had similar issues.

[ratcashdev]

Got rid of the @SessionScoped annotation on the injection point. The initial deployment is still OK, everything as expected. After a hotdeployment, I am still getting the inconsistency exception, but this time also a second exception:

Info:   JSR196OAuthLogin was successfully deployed in 1�494 milliseconds.
Warning:   Error invoking requestInitialized method on ServletRequestListener org.jboss.weld.servlet.WeldListener
org.jboss.weld.exceptions.IllegalStateException: WELD-000227: Bean identifier index inconsistency detected - the distributed container probably does not work with identical applications
Expected hash: 1876076154
Current index: BeanIdentifierIndex [hash=-275291188, indexed=14]:
   0: WELD%AbstractSyntheticBean%JSR196OAuthLogin-1.0-SNAPSHOT%HttpSession
   1: WELD%AbstractSyntheticBean%WEB-INF/lib/AuthenticRoast-Impl-0.3.4-SNAPSHOT%HttpSession
   2: WELD%AbstractSyntheticBean%com.ibm.jbatch.container.cdi.BatchCDIInjectionExtension%HttpSession
   3: WELD%AbstractSyntheticBean%com.sun.faces.flow.FlowCDIExtension%HttpSession
   4: WELD%AbstractSyntheticBean%fish.payara.cdi.jsr107.PayaraHazelcastCDIExtension%HttpSession
   5: WELD%AbstractSyntheticBean%org.glassfish.cdi.transaction.TransactionScopedContextExtension%HttpSession
   6: WELD%AbstractSyntheticBean%org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider%HttpSession
...

Severe:   No valid EE environment for injection of com.ratcash.multilogin.authenticators.AppAuthModule
Info:   Creat ed
Info:   Retrieving main CDI bean.
Info:   Creat ed
Severe:   No valid EE environment for injection of com.ratcash.multilogin.authenticators.SampleOAuth
Info:   Retrieved: com.ratcash.multilogin.authenticators.SampleOAuth@35ce12ab
Severe:   org.jboss.weld.context.ContextNotActiveException: WELD-001303: No active contexts for scope type javax.enterprise.context.SessionScoped
    at org.jboss.weld.manager.BeanManagerImpl.getContext(BeanManagerImpl.java:708)
    at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:90)
    at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:165)

So it seems this has a close relationship with #299 .

[ratcashdev]

Tested it on a plain old vanilla glassfish 4.1 from oracle. That version did not throw the inconsistency exception on the same machine(win 7 64 bit)/java(1.8.0_45) version, whereas payara 4.1.152.x and 4.1.151 did.

[ratcashdev]

Is it possible to somewhere disable the "distributed container"?

[Pandrex247]

Hi, I've tried to recreate this on a Windows environment, and I'm getting the same results as Steve.
Can you lay out some steps to recreate the errors from a clean install to make sure we're not misunderstanding somewhere please?

I did the following on 64 bit Windows:

• Cloned master branch of https://github.com/ratcashdev/JSR196OAuthLogin
• Downloaded and extracted Payara 4.1.152.1
• Downloaded and installed Oracle JDK 1.8.0_45
• Downloaded and installed NetBeans
• Added Payara domain domain1 to NetBeans
• Build and added the 2 JARs from https://github.com/ratcashdev/authenticroast master branch
• Built JSR196OAuthLogin using NetBeans, and clicked Run to deploy
  • This deployed successfully
• Ran the GET request using cURL and RESTClient Firefox plugin
  • Same result that Steve gave
• Made a change to the String and clicked Run again
  • Same result as Steve
• Repeated previous step a few times, still same results
• Stopped the server, removed the 2 JARs, and redid deployments
  • Same result as Steve

[ratcashdev]

The procedure seems to me right. I will try to remove EVERYTHING (incl. JAVA and MAVEN) from my windows machine and repeat the steps you outlined.
Meanwhile I have realized that maybe the path of the payara/glassfish installation may play a role. My contains spaces, i.e.
C:\JAVA Environment\glassfish4\

The maven version is (note the paths):

c:\JAVA Environment\maven\bin>mvn --version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T13:57:37+02:00)
Maven home: c:\JAVA Environment\maven\bin\..
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_45\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

JAVA home is visible from the output as well.

[ratcashdev]

Removed JAVA, Netbeans and maven. Reinstalled everything from scratch including a virgin payara distribution. Path still contained spaced. Result: Got the "inconsitency" exception.
If Netbeans was using the bundled maven distribution (3.0.5), the result was still the same.
On the other PC the paths to glassfish/maven do not contain spaces and the result is still the same (bean inconsistency). I think we can rule out the path as being the difference/culprit.

[ratcashdev]

I just realized, that you're clicking on "RUN" multiple times. This is not what I am doing. I click RUN only the very first time (which in turn asks me which app server to use/start) and I tell Netbeans to remember my selection during the session.

• Do the REST call
• edit SampleOauth.java
• click save
• !!! Wait till Netbeans performs hotdeploy. I do NOT click RUN any more
• Do the REST call

If I click RUN after SAVE, the inconsistency exception is not thrown.

[Pandrex247]

OK I've managed to reproduce your error now, just clicking save does seem to make it appear after three times. We'll look into a bit more for you.

Now that I know what to do, I'll see if I can reproduce it on Linux as well.

[Pandrex247]

I've managed to reproduce this error on Linux with Payara 4.1.152.1. The error doesn't seem to be present in 4.1.151 though, so I thought I'd check the GlassFish upstream, and the error does get thrown there as well.

With that, it looks like it'll be something we've brought in from one of GlassFish upstream cherry-picks since 4.1.151.

[Pandrex247]

I just tried recreating it on clean install of 4.1.151 again on a different VM, and couldn't.
Would you mind retrying it again on a clean install of Payara 4.1.151 when you have the chance please?

[ratcashdev]

I have just re-tested on 4.1.151 (downloaded from http://bit.ly/1CGCtI9 ) I could not reproduce the bug. So I guess I have to apologize for my former statement.

[Pandrex247]

No problem, I was just trying to determine if it was an upstream bug or something we'd put in ourselves.

[ratcashdev]

I was checking the issues labelled as "upstream" for Payara and the related commits. I am willing to do the bisection straight on Payara, if I could get access to the nighly builds. Are those reachable somewhere? Could not download them from your jenkins.

[ratcashdev]

OK, tracked it down. Plain Glassfish nightly glassfish-4.1-b13-03_26_2015 is OK, using Weld 2.2.2 (Final). The nighly glassfish-4.1-b13-03_27_2015 is bad, exhibits the problem described here. The changelog of the latter mentions a single thing: "Upgrade to Weld 2.2.10.Final"

[ratcashdev]

Downgrading to Weld 2.2.8 (Final) resolves the issue. 2.2.9 already throws the exception. Upgrading to 2.2.14 does not help. After reading the changelog of 2.2.9, it seems this behavior is not really a bug, but rather a feature:
https://issues.jboss.org/browse/WELD-1818
So now the question is,

1. what causes WELD to think that we're in a cluster?
2. what causes WELD to discover the incorrect index hash?

[smillidge]

OK thanks for the diagnostic. We can't revert Weld due to fixes we need in the later weld so we'll need to spend some time looking into this in more detail. In particular the new Weld functionality and why it causes problems with hot deployment via NetBeans and not on a full redeploy.

[ratcashdev]

https://issues.jboss.org/browse/WELD-1887 is interesting. It says it can be disabled somehow.

[smillidge]

I think this is related to the fact that I believe NetBeans uses keepState when using "deploy on save" which means it will attempt to retain the session state by serializing and deserializing the session objects. Can you try setting an explicit serialVersionUID on any classes you put into the Session.

Also the property you refer to is org.jboss.weld.serialization.beanIdentifierIndexOptimization can you try setting that as a System property to false on the server NetBeans is deploying to. Although, at this stage, I don't know if setting a system property will be used by Weld.

[ratcashdev]

created a weld.properties file under /src/main/resources with the content
org.jboss.weld.serialization.beanIdentifierIndexOptimization = false

This masked the apparent underlying problems and the exception was not thrown. Setting it to true caused the exception to appear again.

The only code that is session scoped is the PrincipalHolder. Giving him a new serialVersionUID for every deployment also helps avoid the exception.

[smillidge]

OK I think this is the weld bug you refer to

[ratcashdev]

WELD-1887 is marked as resolved in 2.2.10 (and I also tried 2.2.14). I guess in our case it must be something else.

[smillidge]

It's resolved in that they have set a property to switch off the bean index optimisation. Not that they have fixed the problem with hot deploy and sessions. Which fixes the problem in your case as well.

[ratcashdev]

I see now. I also found the clarification in the Weld RefGuide. Pasting the link here for completeness.
https://docs.jboss.org/weld/reference/latest/en-US/html/configure.html#_bean_identifier_index_optimization

This means this ticket can be closed as resolved.

[smillidge]

We need to explicitly set the property org.jboss.weld.serialization.beanIdentifierIndexOptimization = false in Payara similar to the Weld Servlet.

[ratcashdev]

To my understanding, this new 'feature' of WELD only affects development installations. Is it actually desirable to disable this optimization in production environments? I have not tested this in a clustered setup...

[smillidge]

This also causes problems on redeploys with keep session state.

[smillidge]

Also causes problems in shared session with Payara Micro