back port the fix from JBPAPP-2737-JMS feature does not work after cluster failover

We have configured Seam to work in a clustered environment by using a custom JMSConnectionProvider that fetches the UIL2ConnectionFactory from HA-JNDI. This arrangements works perfectly fine while both servers are up - i.e. a web client connected to the slave server will correctly receive Asynchronous Seam notifications from a JMS topic hosted on the master server.

However, when the master server fails over to the slave, the Seam JMS support on the slave stops working. We get the following exception:

06-02 12:52:45.405 ERROR [seam.remoting.messaging.SubscriptionRegistry] (http-192.168.100.140-80-2 org.jboss.mq.SpyJMSException: Cannot get the Topic from the provider; - nested throwable: (java.io.IOE
xception: Client is not connected)
06-02 12:52:45.406 ERROR [jboss.seam.remoting.Remoting] (http-192.168.100.140-80-2 Error
java.lang.NullPointerException
at org.jboss.seam.remoting.messaging.SubscriptionRequest.marshal(SubscriptionRequest.java:31)
at org.jboss.seam.remoting.SubscriptionHandler.marshalResponse(SubscriptionHandler.java:107)
at org.jboss.seam.remoting.SubscriptionHandler.handle(SubscriptionHandler.java:96)
at org.jboss.seam.remoting.Remoting.getResource(Remoting.java:111)
at org.jboss.seam.servlet.SeamResourceServlet.doGet(SeamResourceServlet.java:75)
at org.jboss.seam.servlet.SeamResourceServlet.doPost(SeamResourceServlet.java:92)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
at org.jboss.seam.debug.hot.HotDeployFilter.doFilter(HotDeployFilter.java:68)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:677)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:595)

We have diagnosed this issue and have developed a fix. The problem is as follows:

• once failover occurs, the topicConnection field in the SubscriptionRegistry object becomes invalid, as the connection to the master server has died.
• every subsequent call to SubscriptionRegistry.subscribe() will fail with SpyJMSException: Cannot get the Topic from the provider; Client is not connected.
• even if the master server recovers, this connection is dead and all Seam JMS subscriptions will no longer work.

The fix is to catch the Exception and retry with a new topicConnection if a subscribe request fails. The following patch on jboss-seam-2.0.2.SP1/src/remoting/org/jboss/seam/remoting/messaging/SubscriptionRegistry.java works for us:

Index: SubscriptionRegistry.java
===================================================================
— SubscriptionRegistry.java (revision 5528)
+++ SubscriptionRegistry.java (working copy)
@@ -139,7 +139,21 @@
RemoteSubscriber sub = new RemoteSubscriber(UUID.randomUUID().toString(), topicName);

try {

• sub.subscribe(getTopicConnection());
  + try { + sub.subscribe(getTopicConnection()); + }

  catch (Exception e) {
  + // If we failed during the first attempt, we might have a dead JMS connection.
  + // Clear the topic connection and try again.
  + if (topicConnection != null)

  Unknown macro: {+ try { + topicConnection.close(); + } catch (Exception e1) { + // Ignore any exception during the close, since this connection is probably dead anyway + }+ }

  + topicConnection = null;
  + sub.subscribe(getTopicConnection());
  + }
  subscriptions.put(sub.getToken(), sub);

// Save the client's token in their session context