Uploaded image for project: 'RiftSaw'
  1. RiftSaw
  2. RIFTSAW-404

TransactionRequiredException during cluster failover

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.3.0.Final
    • Fix Version/s: 2.3.1.Final, 2.4.0
    • Component/s: Integration
    • Labels:
      None
    • Environment:

      Windows XP, java version "1.6.0_13", jboss 5.1.0, riftsaw 2.3.0.Final, jbossws cxf 3.4.0

    • Steps to Reproduce:
      Hide

      1. Unzip/Install jboss 5.1.0 GA JDK6
      2. Unzip jbossws cxf 3.4.0
      3. Unzip riftsaw 2.3.0 final
      4. Edit the JBossWS ant.properties.example - rename it to ant.properties and set jboss home path, and server config to 'all' (instead of default)
      5. In the JBossWS folder run ant deploy-jboss510
      6. Edit riftsaw/install/deployment.properties, set jboss home, server config 'all' and add clustering.support=true
      7. In the RiftSaw folder run ant deploy
      8. We'll run both nodes on the same machine, so copy jboss-home\server\all to jboss-home\server\all2
      9. Start the server config 'all', using the command 'run.bat -c all' in the jboss-home\bin folder. Wait for it to startup completely.
      10. Start the server config 'all2', using the command 'run.bat -c all2 -Djboss.service.binding.set=ports-01 -Djboss.messaging.ServerPeerID=2' in the jboss-home\bin folder. Wait for it to startup completely.
      11. You should see the cluster view change in the logs for the 'all' server, like below:

      18:23:25,446 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 1, delta: 1) : [127.0.0.1:1099, 127.0.0.1:1199]
      18:23:25,461 INFO [DefaultPartition] I am (127.0.0.1:1099) received membershipChanged event:
      18:23:25,461 INFO [DefaultPartition] Dead members: 0 ([])
      18:23:25,461 INFO [DefaultPartition] New Members : 1 ([127.0.0.1:1199])
      18:23:25,461 INFO [DefaultPartition] All Members : 2 ([127.0.0.1:1099, 127.0.0.1:1199])
      18:23:25,461 INFO [ODEJobClusterListener] The available nodes now are [127.0.0.1:1099, 127.0.0.1:1199]
      18:23:25,993 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0.0.1:4692]
      18:24:12,900 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0.0.1:4692]
      18:24:36,479 INFO [GroupMember] org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@1b64fe4 got new view [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0.
      0.1:4692], old view is [127.0.0.1:4648|0] [127.0.0.1:4648]
      18:24:36,495 INFO [GroupMember] I am (127.0.0.1:4648)
      18:24:36,495 INFO [GroupMember] New Members : 1 ([127.0.0.1:4692])
      18:24:36,495 INFO [GroupMember] All Members : 2 ([127.0.0.1:4648, 127.0.0.1:4692])

      12. Stop the 'all2' server.
      13. The 'all' server tries to fail over and the error below is logged to the server log for the 'all' server:

      18:26:34,874 INFO [GroupMember] I am (127.0.0.1:4648)
      18:26:36,733 WARN [GMS] 127.0.0.1:4648 failed to collect all ACKs (1) for mcasted view [127.0.0.1:4648|2] [127.0.0.1:4648] after 2000ms, missing ACKs from [127.0.0.1:4648] (r
      eceived=[]), local_addr=127.0.0.1:4648
      18:26:37,077 INFO [GroupMember] Dead members: 1 ([127.0.0.1:4692])
      18:26:37,077 INFO [GroupMember] All Members : 1 ([127.0.0.1:4648])
      18:26:40,108 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|2] [127.0.0.1:4648]
      18:26:40,296 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 2, delta: -1) : [127.0.0.1:1099]
      18:26:40,296 INFO [DefaultPartition] I am (127.0.0.1:1099) received membershipChanged event:
      18:26:40,296 INFO [DefaultPartition] Dead members: 1 ([127.0.0.1:1199])
      18:26:40,311 INFO [DefaultPartition] New Members : 0 ([])
      18:26:40,311 INFO [DefaultPartition] All Members : 1 ([127.0.0.1:1099])
      18:26:40,311 INFO [ODEJobClusterListener] The available nodes now are [127.0.0.1:1099]
      18:26:40,327 ERROR [AbstractEntityManagerImpl] Unable to mark for rollback on PersistenceException:
      java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.nosuchtx] [com.arjuna.ats.internal.jta.transaction.arjunacore.nosuchtx] No such transactio
      n!
      at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.setRollbackOnly(BaseTransaction.java:191)
      at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.setRollbackOnly(BaseTransactionManagerDelegate.java:123)
      at org.hibernate.ejb.AbstractEntityManagerImpl.markAsRollback(AbstractEntityManagerImpl.java:421)
      at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:576)
      at org.hibernate.ejb.QueryImpl.executeUpdate(QueryImpl.java:48)
      at org.apache.ode.dao.jpa.scheduler.SchedulerDAOConnectionImpl.updateReassign(SchedulerDAOConnectionImpl.java:177)
      at org.jboss.soa.bpel.clustering.ODEJobClusterListener.membershipChanged(ODEJobClusterListener.java:79)
      at org.jboss.ha.framework.server.ClusterPartition.notifyListeners(ClusterPartition.java:1589)
      at org.jboss.ha.framework.server.ClusterPartition.processEvent(ClusterPartition.java:1437)
      at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:108)
      at java.lang.Thread.run(Thread.java:619)
      18:26:40,358 WARN [DefaultPartition] HAMembershipListener callback failure: org.jboss.soa.bpel.clustering.ODEJobClusterListener@1c0ef03
      javax.persistence.TransactionRequiredException: Executing an update/delete query
      at org.hibernate.ejb.QueryImpl.executeUpdate(QueryImpl.java:48)
      at org.apache.ode.dao.jpa.scheduler.SchedulerDAOConnectionImpl.updateReassign(SchedulerDAOConnectionImpl.java:177)
      at org.jboss.soa.bpel.clustering.ODEJobClusterListener.membershipChanged(ODEJobClusterListener.java:79)
      at org.jboss.ha.framework.server.ClusterPartition.notifyListeners(ClusterPartition.java:1589)
      at org.jboss.ha.framework.server.ClusterPartition.processEvent(ClusterPartition.java:1437)
      at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:108)
      at java.lang.Thread.run(Thread.java:619)
      18:26:40,561 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|2] [127.0.0.1:4648]

      Show
      1. Unzip/Install jboss 5.1.0 GA JDK6 2. Unzip jbossws cxf 3.4.0 3. Unzip riftsaw 2.3.0 final 4. Edit the JBossWS ant.properties.example - rename it to ant.properties and set jboss home path, and server config to 'all' (instead of default) 5. In the JBossWS folder run ant deploy-jboss510 6. Edit riftsaw/install/deployment.properties, set jboss home, server config 'all' and add clustering.support=true 7. In the RiftSaw folder run ant deploy 8. We'll run both nodes on the same machine, so copy jboss-home\server\all to jboss-home\server\all2 9. Start the server config 'all', using the command 'run.bat -c all' in the jboss-home\bin folder. Wait for it to startup completely. 10. Start the server config 'all2', using the command 'run.bat -c all2 -Djboss.service.binding.set=ports-01 -Djboss.messaging.ServerPeerID=2' in the jboss-home\bin folder. Wait for it to startup completely. 11. You should see the cluster view change in the logs for the 'all' server, like below: 18:23:25,446 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 1, delta: 1) : [127.0.0.1:1099, 127.0.0.1:1199] 18:23:25,461 INFO [DefaultPartition] I am (127.0.0.1:1099) received membershipChanged event: 18:23:25,461 INFO [DefaultPartition] Dead members: 0 ([]) 18:23:25,461 INFO [DefaultPartition] New Members : 1 ( [127.0.0.1:1199] ) 18:23:25,461 INFO [DefaultPartition] All Members : 2 ( [127.0.0.1:1099, 127.0.0.1:1199] ) 18:23:25,461 INFO [ODEJobClusterListener] The available nodes now are [127.0.0.1:1099, 127.0.0.1:1199] 18:23:25,993 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0.0.1:4692] 18:24:12,900 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0.0.1:4692] 18:24:36,479 INFO [GroupMember] org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@1b64fe4 got new view [127.0.0.1:4648|1] [127.0.0.1:4648, 127.0. 0.1:4692], old view is [127.0.0.1:4648|0] [127.0.0.1:4648] 18:24:36,495 INFO [GroupMember] I am (127.0.0.1:4648) 18:24:36,495 INFO [GroupMember] New Members : 1 ( [127.0.0.1:4692] ) 18:24:36,495 INFO [GroupMember] All Members : 2 ( [127.0.0.1:4648, 127.0.0.1:4692] ) 12. Stop the 'all2' server. 13. The 'all' server tries to fail over and the error below is logged to the server log for the 'all' server: 18:26:34,874 INFO [GroupMember] I am (127.0.0.1:4648) 18:26:36,733 WARN [GMS] 127.0.0.1:4648 failed to collect all ACKs (1) for mcasted view [127.0.0.1:4648|2] [127.0.0.1:4648] after 2000ms, missing ACKs from [127.0.0.1:4648] (r eceived=[]), local_addr=127.0.0.1:4648 18:26:37,077 INFO [GroupMember] Dead members: 1 ( [127.0.0.1:4692] ) 18:26:37,077 INFO [GroupMember] All Members : 1 ( [127.0.0.1:4648] ) 18:26:40,108 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|2] [127.0.0.1:4648] 18:26:40,296 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 2, delta: -1) : [127.0.0.1:1099] 18:26:40,296 INFO [DefaultPartition] I am (127.0.0.1:1099) received membershipChanged event: 18:26:40,296 INFO [DefaultPartition] Dead members: 1 ( [127.0.0.1:1199] ) 18:26:40,311 INFO [DefaultPartition] New Members : 0 ([]) 18:26:40,311 INFO [DefaultPartition] All Members : 1 ( [127.0.0.1:1099] ) 18:26:40,311 INFO [ODEJobClusterListener] The available nodes now are [127.0.0.1:1099] 18:26:40,327 ERROR [AbstractEntityManagerImpl] Unable to mark for rollback on PersistenceException: java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.nosuchtx] [com.arjuna.ats.internal.jta.transaction.arjunacore.nosuchtx] No such transactio n! at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.setRollbackOnly(BaseTransaction.java:191) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.setRollbackOnly(BaseTransactionManagerDelegate.java:123) at org.hibernate.ejb.AbstractEntityManagerImpl.markAsRollback(AbstractEntityManagerImpl.java:421) at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:576) at org.hibernate.ejb.QueryImpl.executeUpdate(QueryImpl.java:48) at org.apache.ode.dao.jpa.scheduler.SchedulerDAOConnectionImpl.updateReassign(SchedulerDAOConnectionImpl.java:177) at org.jboss.soa.bpel.clustering.ODEJobClusterListener.membershipChanged(ODEJobClusterListener.java:79) at org.jboss.ha.framework.server.ClusterPartition.notifyListeners(ClusterPartition.java:1589) at org.jboss.ha.framework.server.ClusterPartition.processEvent(ClusterPartition.java:1437) at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:108) at java.lang.Thread.run(Thread.java:619) 18:26:40,358 WARN [DefaultPartition] HAMembershipListener callback failure: org.jboss.soa.bpel.clustering.ODEJobClusterListener@1c0ef03 javax.persistence.TransactionRequiredException: Executing an update/delete query at org.hibernate.ejb.QueryImpl.executeUpdate(QueryImpl.java:48) at org.apache.ode.dao.jpa.scheduler.SchedulerDAOConnectionImpl.updateReassign(SchedulerDAOConnectionImpl.java:177) at org.jboss.soa.bpel.clustering.ODEJobClusterListener.membershipChanged(ODEJobClusterListener.java:79) at org.jboss.ha.framework.server.ClusterPartition.notifyListeners(ClusterPartition.java:1589) at org.jboss.ha.framework.server.ClusterPartition.processEvent(ClusterPartition.java:1437) at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:108) at java.lang.Thread.run(Thread.java:619) 18:26:40,561 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.1:4648|2] [127.0.0.1:4648]

      Description

      With RiftSaw deployed on a two node JBoss cluster, when one of the nodes is stopped, there is an TransactionRequiredException in the other surviving node. This exception then prevents successful failover of process instances running on the stopped node over to the surviving node.

      This error happens even with an 'empty' RiftSaw deployment that doesn't have any BPEL process deployed in it. (See Steps to Reproduce
      ). So it's not related to any specific BPEL process code.

        Gliffy Diagrams

          Attachments

          1. ant.properties
            0.7 kB
          2. deployment.properties
            0.8 kB
          3. failover_test_bpel.zip
            340 kB
          4. ibek_server2.log
            383 kB
          5. server.log
            111 kB
          6. server.log
            118 kB

            Issue Links

              Activity

                People

                • Assignee:
                  jeff.yuchang Jeff Yu
                  Reporter:
                  anujbhatia anuj bhatia
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: