Transactions Archive

Large Message Processing Issues In JMS Transactions

When there is a distributed transaction involved in message processing, it is very important that all the resources involved in the transaction are reachable without any latency. When this is not the case and there is an issue with the communication between the resources, there can be issues of message processing which might result in transaction timeouts, rollbacks and other runtime errors.

Take a scenario of the below architecture:
A SLSB client/Application sends a message of 2MB size to weblogic Queue, that message is received by the MDB deployed on the weblogic server. That message is then processed, retrieves some information from databases. After the processing is complete, the messages is then displayed by the MDB.

First of all we need to identify where is the message size causing an issue. Is it from the queue end, message bridge end (if used), or is it the MDB that is not able to consume the messages. Enable the debug flags as required.

When these messages are large in size and out of the size limit of the queue or the protocol by which the message is being transferred, below errors can occur:

1) weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: ‘10000080’ bytes exceeds the configured maximum of: ‘10000000’ bytes for protocol : ‘t3’.
Solution : Try to increase the message size from the admin console for the protocol by:
– Queue-> Configuration-> Threshholds and Quotas-> Maximum Message Size
– MAX message size in Server-> Admin/managed server-> Protocals-> General
– Setting the system property -Dweblogic.MaxMessageSize=20000000

Note : The message size would vary as per the application requirement.

2) <BEA-489003> <Caught Exception: com.bea.wli.variables.XmlObjectRuntimeException: Error Creating the XmlObject
com.bea.wli.variables.XmlObjectRuntimeException: Error Creating the XmlObject
at com.bea.wli.variables.ProcessXML.underlyingXmlObject(ProcessXML.java:759)
at com.bea.xml.FilterXmlObject.xmlText(FilterXmlObject.java:164)
at com.bea.wli.broker.Message.toString(Message.java:264)

– This is seen when the message processing includes retrieving some data from the database using SQL Queries but the messages size exceeds the maximum size of a document stored in the SQL Document Store.
Solution: – Increase the values of weblogic.wli.DocumentMaxInlineSize and weblogic.wli.DocumentMaxInMemorySize in wli-config.properties.
– Increase the WLI transaction timeout in the wlw-config.xml.
– Set the parameters XASetTransactionTimeout=”true” and XATransactionTimeout=”xxx” as perferred.

3)java.io.IOException: Could not connect to SQL Document Store
at com.bea.wli.variables.ProcessXML.getLoaderStream(ProcessXML.java:302)
at com.bea.wli.variables.ProcessXML.getTokenStreamData(ProcessXML.java:338)
at com.bea.wli.variables.XmlObjectVariableFactory.getXmlObject(XmlObjectVariableFactory.java:359)
Caused by: com.bea.wli.store.DocumentStoreException: Could not connect to SQL Document Store
at com.bea.wli.store.impl.SQLStore$SQLStoreConnection.<init>(SQLStore.java:3162)

– Check if there is a network issue and the SQL store is available. Take 5-6 thread dumps at the interval of 10 seconds each to check if there are any stuck threads while connecting to the SQL store.

4) java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: Transaction rolled
back: Transaction timed out after 600 seconds

– Now if the message size of 2mb is being processed successfully and if you are facing issue with a larger message say 3mb, you might see the above transaction timeout error in the log files. This means issue is with the message processing taking a long time.

– If the whole sending and receiving of messages from application is one single transaction, check where is the latency. Is the communication between all the resources happening properly or is there a network issue. Check if all the information is being retrieved from the database on time or is it taking long.

If you find any other issue or reason for the JMS transaction rollbacks or delay in message processing, feel free to comment and let us know about it, we will take a look and try to resolve the issue.

Best Regards.

WebLogic Server Transaction Manager Issues

When XA connection pools are used for the datasources which will involve transactions with the database and other resources, these are the most common issues faced:

1) Exception: : javax.ejb.EJBException: nested exception is: java.sql.SQLException: ORA-02049: timeout: distributed transaction waiting for lock

2) EJB Exception: : javax.ejb.EJBException: nested exception is: java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAER_NOTA : The XID is not valid start() failed on resource ‘CONNECTION_NAME’: XAER_NOTA : The XID is not valid
oracle.jdbc.xa.OracleXAException
at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:882)
at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:221)
at weblogic.jdbc.wrapper.VendorXAResource.start(VendorXAResource.java:50)
at weblogic.jdbc.jta.DataSource.start(DataSource.java:696)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1145)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1077)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:255)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:449)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1510)

The XAER_NOTA error raises due to, using two different JDBCConnection Pools for the same Database (Oracle) or two DataSources for the same pool.

XAER_NOTA errors are also seen during recovery.
These errors are thrown for transactions that have been committed before the server restart but still exist in the transaction log at the time the server was booted.
During recovery processing, for each transaction record in the transaction log, the transaction manager will inform the participating resources of the commit decision.
If the resource commit directive succeeded before the restart, the resource manager will respond to a subsequent commit with XAER_NOTA because it no longer has knowledge of the Xid.

The transaction manager ignores this error assuming that the commit succeeded before the crash.

The reason why there are transaction log records that exist for transactions that have already completed is because the transaction manager only removes entries during checkpoint operations.
A checkpoint occurs every five minutes by default and deletes transaction log files for which all records have been released.
The checkpoint interval can be configured via the JTAMBean.CheckpointIntervalSeconds attribute. You can set the Checkpoint Interval in the administration console on the Domain —> Configuration —> JTA tab.

To get more information about the issue, enable the below debug flags on the server:
weblogic.JTAXA
weblogic.JTA2PC
weblogic.JTAJDBC

The javax.transaction.xa.XAResource interface provides a method, setTransactionTimeout, which in some driver implementations sets the resource’s internal timeout interval.

The WebLogic transaction manager can be instructed to invoke this method with a value equal to the global transaction timeout prior to each resource enlistment.

For JDBC connection pool configurations, set the attribute “XASetTransactionTimeout” to “true” to enable this feature.
Note that this JDBCConnectionPool attribute is only applicable for XA-compliant drivers.

Also note that setting this attribute has no affect on XA drivers that do not implement the XAResource.setTransactionTimeout method.

The Oracle thin driver supports XAResource.setTransactionTimeout. The WebLogic jDriver for Oracle driver does not implement this method.

Below are the tunings that should help to resolve the issue:
– Use one connection pool for one datasource.
– Merge the 2 XA pools for one single database.
– Try to set autoCommit to true or call connection.commit() before returning the connection to the connection pool.

Transaction related tunings:
– KeepXAConnTillTxComplete=”true”
– TestConnectionsOnReserve=”true”
– XASetTransactionTimeout=”true”
– XATransactionTimeout=”300″ (This value must be greater than the JTA timeoutseconds value)
– DISTRIBUTED_LOCK_TIMEOUT on the database should be greater than XATransactionTimeout.
– Database Pool XATransactionTimeout should be greater than Global transaction timeout.

If these tunings do not solve the issue, do revert with the debugs enabled for the server so that the logs give more information about the transaction internals.

Best Regards.