Working with JMS and the Standard Issues in JMS

Java Messaging Service acronymised as JMS is popular in both development and administration world. Different vendors have got different implementations for this JMS API. Oracle Weblogic Server implements this JMS for communication between two different weblogic server domains. This implementation although looks simple but its usage is very high (especially if you are working for a telecom domain).

In the process of implementation of this we get many issues. This article mainly targets to the standard issues of JMS for which, most of us take the assistance from Oracle BEA customer support by raising tickets with them as Sev2 or Sev1.

 

Standard Issue -1: JMS Messages Redelivery

Why does a JMS message get redelivered?

The JMS Server could redeliver messages because of any of the following reasons:

• A java.lang.Error or java.lang.RuntimeException has been thrown from the Receiver/MDB’s onMessage method
• User has made a call to ejbcontext.setRollbackOnly() in his MDB’s onMessage method (this applies to Container Managed Transaction only)
• MDB participating in a Transaction failed for some reason. For example, if the MDB calls another EJB, which makes a JDBC call and the database operation fails. In this case the transaction in which MDB is participated will be rolled back and the message will be redelivered.
• MDB is transactional but takes a long time to process, the transaction monitor will timeout the transaction and rolls it back – consequently, JMS will redeliver the message.
• Slow down in the server process. This could happen because of a poison message.
• User has made a call to Session.recover() from a standalone receiver
• Session.acknowledgement() is not called from a standalone receiver with client acknowledgement

Application Coding Diagnostics

Add the following code snippet to the receiver’s onMessage method to find out if the message is being redelivered.

Public void onMessage (javax.jms.Messsage msg)

{

try{

System.out.println (“Mesg ID:”+msg.getJMSMessageID ());

System.out.println (“Is Message redelivered:”+msg.getJMSRedelivered ());

…

}

}

If the same message ID is getting printed multiple times and the JMSRedelivered() flag is returning true, then it confirms that the messages are getting redelivered.

Checklist for Troubleshooting JMS Redelivery Problems

• Check the receiver type – is it an MDB, Standalone Synchronous, Asynchronous Receiver?
• Check the acknowledgement options of the receiver:
  • Are Acknowledgement-modes, Transacted Sessions, orBean Managed being used?
  • Transactions or container managed transactions?
• In case of MDBs with CMT, BMT or standalone receivers, verify whether the onMethod () is returned successfully.
• If there’s a java.lang.Error or java.lang.RuntimeException thrown from the standalone receiver’s/MDB’s onMessage() method then the JMS server did not receive the acknowledgement and the message will be redelivered by the JMS Server.
• If the MDB (CMT only) calls ejbcontext.setRollbackOnly() this will force the JMS Server to redeliver the message, as the transaction has been marked for rollback.
• In the case of an MDB invoking an EJB (Session or Entity) that involves a JDBC call and if the DB operation fails for some unknown reasons the transaction in which MDB is enlisted will be rolled back and message gets redelivered. To avoid this problem, investigate the problems on the EJB side.
• In the case of long running processes in the MDB’s onMessage() method (CMT only) the transaction will be timed out and it will be rolled back. Consequently the JMS Server will redeliver the message.
• JMS message redelivery mostly happens due to the application coding errors. By enabling the following debug flags, the root cause of message redelivery can be found.
  • DebugJMSMessagePath=”true”  DebugJMSXA=”true”
  • In the config.xml, under the ServerDebug tag of the each Server tag, set the following debug flags:
    • <ServerDebug DebugJMSXA=”true” DebugJMSMessagePath=”true” />
    • Make sure the stdOutSeverity level of the server is INFO and StdoutDebugEnabled is set to “true”.
• If the session.recover() is invoked with CLIENT_ACKNOWLEDGE Mode (standalone receivers only), the JMS Server stops message delivery in this session, and restarts message delivery with the oldest unacknowledged message.
• The following parameters may be configured for a JMS destination to avoid poison messages.
  • Redelivery Delay Time
    When a message is rolled back or recovered, the redelivery delay is the amount of time a message is put aside before an attempt is made to redeliver the message.
  • Redelivery Limit
    Redelivery limit is the number of times JMS server will try to redeliver the message to the receiver/MDB
  • Error Destination
    If the JMS server tried to redeliver more than “Redelivery Limit”, then the JMS Server will forward these undelivered messages to the ‘Error Destination’.If the error destination is not configured, and a message exceeds its redelivery limit, that  message is simply deleted.
• Another way of debugging this problem is to print/log the exceptions in onMessage() method. The following code snippet will log an exception stack trace and also re-throws the exception from receiver’s onMessage() method.

public void onMessage(Message msg) {
try {
TextMessage tm = (TextMessage) msg;
String text = tm.getText();
System.out.println(“Msg: ” + text+” Mesg ID:”+msg.getJMSMessageID());
}
catch(JMSException ex) {
ex.printStackTrace();
}
catch(java.lang.RuntimeException ey) {
ey.printStackTrace();
throw ey;
}
catch(java.lang.Error ez) {
ez.printStackTrace();
throw ez;
}
}

Understanding JMS Message Redelivery Debug Info

The JMS debug flags DebugMessagePath and DebugJMSXA provide more details about the root cause of the message redelivery problem. DebugMessagePath will help identify if the message is getting redelivered or not.

<Feb 3, 2004 5:01:25 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Assigning to the backend consumer, message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:01:25 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Adding backend session’s unacked message list,
message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:01:25 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Dispatching to the frontend, message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:01:25 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
FRONTEND/FESession (id: <576180353183090550.27>) :
Pushing to the client, message ID:P<802808.1075856485894.0>>
>>Print From the onMessage method: I am the MESSAGE, MsgID: ID:P<802808.1075856485894.0>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<Feb 3, 2004 5:02:05 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Assigning to the backend consumer, message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:02:05 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Adding backend session’s unacked message list,
message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:02:05 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
BACKEND/BEQueue: Dispatching to the frontend, message ID:P<802808.1075856485894.0>>

<Feb 3, 2004 5:02:05 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging MSG_PATH!
FRONTEND/FESession (id: <576180353183090550.34>) :
Pushing to the client, message ID:P<802808.1075856485894.0>>
>>Print From the onMessage method: I am the MESSAGE, MsgID: ID:P<802808.1075856485894.0>From the above debug output, it can be seen that the same message is pushed multiple times to the receiver.

NOTE: This debug flag works in WLS 8.1 and later releases.

DebugJMSXA will help identify if the message is redelivered due to transaction related problems.

<Feb 3, 2004 5:16:10 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging XA ! XA(25421973,1007511,0000013BC2697F28EDB9) >RM-rollback() >

<Feb 3, 2004 5:16:10 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging XA ! XA(25421973,1007511,0000013BC2697F28EDB9) >TE-recv-startRollback() (TE-recv hash=31838215 xid=0000013BC2697F28EDB9 mId=<712671.1075857330638.0> queue=TestDest)>

<Feb 3, 2004 5:16:10 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging XA ! XA(25421973,1007511,0000013BC2697F28EDB9) <TE-recv-startRollback() (TE-recv hash=31838215 xid=0000013BC2697F28EDB9 mId=<712671.1075857330638.0> queue=TestDest)OK>

<Feb 3, 2004 5:16:10 PM PST> <Debug> <JMS> <BEA-040002> <JMS Debugging XA ! XA(25421973,27221567,0000013BC2697F28EDB9) >RM-rollback() >

Background

Once a JMS message is delivered to the consumer(s), JMS Server waits for acknowledgement from the consumer(s). There are four ways in which a consumer can send acknowledgement to the JMS Server:

1. Transacted Sessions
2. Container Managed Transactions
3. Bean Managed Transactions (or) User Transaction
4. Acknowledgement modes

 

Transacted Sessions

When JMS Transacted sessions are used for sending/receiving JMS messages, internally a local transaction is started. The scope of this transaction is limited to send/receive operations within this session. When the receiver/consumer uses Transacted session, message redelivery depends on commit/rollback call on the session. The receiver’s rollback call on the session will make the messages redelivered by the JMS server. The JMS Transacted Session’s transaction will not have any effect outside the scope of this session. Therefore, it cannot participate in the external transaction (BMT or CMT) (as far as it is not XA aware).

Container Managed Transaction (CMT)

Container Managed Transactions apply for Message Driven Beans. In order to run a MDB in container managed transaction, <transaction-type> tag in ejb-jar.xml and <transaction attribute> in weblogic-ejb-jar.xml should be set to “Container” and “Required” respectively. For example:

Before pulling the message from the destination, EJB container starts a transaction and delivers it to the MDB. Once the onMessage() method returns successfully without any exception, the transaction is committed by the container. EJB container is responsible for begin, commit and rollback for this transaction. If the TX gets rolled back for any reason, JMS server is required to immediately redeliver the message to the next available consumer.

Bean Managed Transaction (BMT) or User Transaction

Bean Managed Transactions apply for both MDBs and standalone consumers. For bean managed or User Transaction, the application code should take the responsibility for begin, commit, or rollback of the transaction. The following sample code snippet creates a User Transaction.

UserTransaction tx1 =(UserTransaction)context.lookup(“javax.transaction.UserTransaction”)
public void onMessage(Message msg)  {
try {
tx1.begin();
String msgText = ((TextMessage)msg).getText();
System.out.println(“Mesg ID:”+msg.getJMSMessageID()+”msg:”+msgText);
tx1.commit();
} catch (Exception jmse) {
jmse.printStackTrace();
}
}

In order to configure a MDB to run in Bean-Managed Transaction, the <transaction-type> tag value in ejb-jar.xml should be set to “Bean”.

In BMT, since the transaction started within MDB, pulling the message from the destination is not part of the transaction. Therefore, the outcome of the transaction, whether commit or rollback, has no impact on message redelivery. Even if the transaction is rolled back, the message is not redelivered.

Acknowledgement Modes

Acknowledgement Mode may be specified while creating a JMS session. Acknowledgement Mode may be set to any of the following values:

• AUTO_ACKNOWLEDGE
• DUPS_OK_ACKNOWLEDGE
• CLIENT_ACKNOWLEDGE

The following sample code snippet sets the Acknowledgement Mode to AUTO_ACKNOWLEDGE.

QueueSession qs = qconn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

For AUTO_ACKNOWLEDGE and DUPS_OK_ACKNOWLEDGE, the message will be automatically acknowledged if it is successfully returned from the onMessage() method.

For CLIENT_ACKNOWLEDGE mode, call message.acknowledge() explicitly for the message to be acknowledged.

If MDB is not running in a Tx, then it may run with one of the following modes:

• TO_ACKNOWLEDGE
• DUPS_OK_ACKNOWLEDGE

CLIENT_ACKNOWLEDGE mode is not supported for MDBs

For standalone asynchronous listener, if the message is not acknowledged for any reason, then the message is redelivered only once. If the redelivery fails, then the message is implicitly deleted from the JMS server. As a result, the redelivery count does not get updated, and the message does not get redirected to the error destination (even if one is set for this destination).

Standard Issue -2: JMS Pending Message

What is the meaning of pending messages?

A message present on a JMS destination can be either current or pending. A current message is available for immediate consumption by consumers, whereas a pending message is unavailable. The current message can be referred to as an available message.

The JMS message lifecycle can be summarized as below with respect to the following two states:

1. A message sent by a JMS producer
   • Without any associated transaction:
     It is immediately current.
   • Within a JTA transaction or transacted JMS session:
     It remains pending until the transaction is committed or rolled back. If the transaction is committed, the message becomes available, and if the transaction is rolled back, the message is removed from the destination.
   • With a specified TimeToDeliver property:
     It remains pending until the TimeToDeliver period expires. This is expected because the purpose of the TimeToDeliver property is to keep the message unavailable to consumers until the specified time period.

1. A message received by a JMS consumer
   • Without any associated JTA transaction and in a non-transacted JMS session with an acknowledgement mode of NO_ACKNOWLEDGE or MULTICAST_NO_ACKNOWLEDGE:
     It is immediately removed from the destination.
   • Within a JTA transaction or transacted JMS session:
     It becomes pending until the transaction or session is committed or rolled back. On commit, the message is removed from the destination, and on rollback, the message becomes available again, unless a Redelivery delay is specified, in which case it continues to remain pending until the Redelivery delay.
   • Within a non-transacted JMS session with the acknowledgement mode of AUTO_ACKNOWLEDGE, DUPS_OK_ACKNOWLEDGE, or CLIENT_ACKNOWLEDGE:
     It becomes pending until the acknowledgement is received.

To summarize, pending JMS messages means the messages that were:

• Sent in a transaction but not committed
• Received but not acknowledged
• Received but not committed
• Subject to a redelivery delay
• Subject to a delivery time

 

 

Are pending messages harmful?

Having pending messages in your JMS destinations does not by itself have any adverse effect on your JMS server or destination. However, if you see an unusual increase in the number of pending messages, it does point to something not working correctly in the application or the WebLogic Server, which will be discussed in this pattern.

Monitoring Pending Messages

Pending Messages can be monitored as follows:

• Using WebLogic Admin Console
  1. Browse to JMS -> Servers -> Monitoring -> Monitor Active JMS Destinations.
  2. “Messages pending” column shows the pending messages for each destination.
  3. “Bytes pending” column shows the pending bytes for each destination. This bytes count includes the size of only the properties and the body of the messages, and not that of the headers.

• Using JMX
  1. Use JMSDestinationRuntimeMBean.getMessagesPending() to get pending messages.
  2. Use JMSDestinationRuntimeMBean.getBytesPending() to get pending bytes.

• Set WebLogic Server environment in the command window. For example, you can run the setDomainEnv.cmd/sh script in your domain directory.
• Run “java weblogic.WLST”.
• Run the following WLST commands:
• Connect to your WebLogic Server instance:
• connect(‘[username]’,'[password]’,'[url]’)
• Access the server runtime MBean hierarchy:
• serverRuntime()
• Change directory to your JMS server instance:
• cd (‘JMSRuntime/examplesServer.jms/JMSServers/examplesJMSServer’)
• ‘examplesServer’ is the name of your WebLogic Server instance, and ‘examplesJMSServer’ is the name of the JMS server instance.
• Get the pending messages count on the JMS server:
• get(‘MessagesPendingCount’)
• Note: that “Bytes Current” and “Messages Current” numbers in the JMS -> Servers -> Monitoring -> Monitor Active JMS Destinations section of the Weblogic Server Admin Console do not include the pending bytes and pending messages respectively. In other words, “Messages Pending” and “Bytes Pending” count are over and above the “Messages Current” and “Bytes Current” count respectively.

Why does the problem occur and how do you troubleshoot it?

This section discusses the common causes for the increase in the number of pending messages, and the corresponding debug mechanisms.

• Hang in asynchronous client
  1. Description
     One of the most common causes of pending messages is that an asynchronous consumer is lagging behind the producer. In other words, there is a hang or slowdown of message processing by asynchronous consumers.

Weblogic JMS pipelines messages that are delivered to asynchronous consumers, otherwise known as message listeners. This action aids performance as messages are aggregated when they are internally pushed from the server to the client.

This message buffer or pipeline can be configured using the MessagesMaximum parameter in the Connection Factory settings the default value of this parameter is 10 per instance of the asynchronous client. In case of a MDB, each instance is a separate client. Hence, if there are 15 instances of a given MDB, there can be 10*15=150 pending messages buffered on the MDB side in the message pipeline.

These buffered messages will not be consumed by the client if the client happens to be hanging for any reason while processing a message. Other messages in the pipeline will wait and contribute to the number of pending messages. Each such hanging thread corresponds to a hanging asynchronous client and contributes to pending messages equal to the MessagesMaximum parameter. For instance, 5 threads hanging in onMessage() method will contribute to 50 pending messages, assuming the MessagesMaximum parameter has the default value of 10.

• 1. Debug steps
     the main problem is the hang in the client. The presence of pending messages is just an after effect. To find out if the client is hanging, take thread dumps as described in Generic Server Hang Pattern document. If you see threads hanging in the onMessage() method of the asynchronous consumer, it means that the client is hanging while processing a message in the queue.

You can always reboot the standalone client or redeploy the MDB. All the messages in the pipeline, including the message being currently processed, will again become available in the queue, ready to be processed by the receivers. This may or may not help resolve the hang depending upon the cause of the hang. For instance, if the hang occurs because the onMessage() code is waiting for an external resource that is currently unavailable, the client will continue to hang after the reboot/redeployment, until the resource becomes available.

You can also reduce the value of the MessagesMaximum property to 1 from the default value of 10. This means that there will be no messages in the pipeline. A second message will be delivered to the asynchronous consumer only after the first message processing has been completed. This can have some performance impact.

It is important to note that this MessagesMaximum property defined on the Connection Factory is different from the MessagesMaximum property for a JMS destination. The property for a JMS destination is used to configure the maximum number of messages that can be stored in the destination.

• Absence of one or more durable subscribers for topics
  1. Description
     By definition, a topic message remains on the topic until it is consumed by all the currently available non-durable subscribers, and all the registered durable subscribers, irrespective of whether they are currently available or not. Once a topic message is consumed by any given subscriber, it becomes pending and remains so, until it has been consumed by all of these subscribers. This is especially important to understand in case of durable subscribers. Messages can stay in a pending state forever, if one or more of the registered durable subscribers never come back up. In contrast, a queue message is consumed by just one consumer. The message transitions to a pending state when it is delivered to the consumer, and is removed from the queue on acknowledgement of the receipt or commit of the concerned transaction.

• 1. Debug steps
     Monitor if all the durable subscribers for the topic are “Active”.
     • On the WebLogic Admin Console, browse to:
       JMS -> Servers -> [JMS Server Name] -> Monitoring -> Monitor Active JMS Destinations -> Monitor all Durable Subscribers

• 1. • Check if the “Active” column has a value of “true” for all durable subscribers.

If any of the subscribers are not active, bring them up. For MDB clients acting as durable subscribers, make sure that the MDBs are deployed correctly on the server(s), and for standalone JMS clients, ensure that they are running.

• Long Redelivery Delay Period and frequent rollbacks on consumer side
  1. Description
     If the Connection Factory and/or the JMS Destination is configured with a long Redelivery Delay period, and messages are rolled back frequently by the consumers, it will lead to a high number of pending messages.

• 1. Debug steps
     Investigate the cause of the rollbacks on the consumer side by analyzing the exceptions thrown. You can also reduce the Redelivery Delay period.

• Infrequent acknowledgement when using CLIENT_ACKNOWLEDGE
  1. Description
     Another possible cause of pending messages is that clients using non-transacted sessions with CLIENT_ACKNOWLEDGE mode do not send acknowledgements frequently. This will cause the number of pending messages to increase as messages are delivered to and consumed by consumers, but there is no acknowledgement sent back to the JMS server.

Debug steps
Check the code which does the client acknowledgement to ensure that the acknowledgement is done frequently.

Additional information on pending messages

• There is no way to purge current or pending messages in a JMS destination.
• There is no mechanism to browse pending messages in a destination.
• The storage mechanism of a message is independent of whether a message is pending or available. In both states, if the message is non-persistent, it stays in memory, whereas a persistent message is persisted to the underlying JMS store.
• To avoid the server from running out of memory due to pending or current messages, configure ‘Paging’ for the connection factory. This ensures that the message properties and body may be persisted to the paging store depending upon the quotas specified for the destination. The message headers are always present in memory irrespective of paging.
• If a JMS sender or receiver having an associated JTA transaction does not explicitly commit or rollback the transaction, the messages remain pending, until one of the following conditions takes place:
  1. The transaction times out causing a rollback or
  2. The transaction is explicitly rolled back through the WebLogic Admin Console.

In case of a transacted JMS session, the message will remain pending until the client JVM is alive. In cases, JTA transaction and transacted JMS session, if the client JVM exits without committing or rolling back the transaction, the server receives a PeerGoneException, rolls back the transaction, and the messages are rolled back. In case of a producer the messages are removed from the destination, and in case of a consumer they are moved from the pending to the current state in the destination.

• Rolling back an associated JTA transaction or transacted JMS session multiple times does not cause double counting of pending messages, nor does an exception that set a JTA transaction as rollbackOnly followed by an actual rollback.
• JMS filestore does not shrink in size – The size of a JMS filestore increases as messages are sent to the destinations included in its target JMS server. This can be controlled by tuning quotas and flow control to restrict the number of messages sent to the destinations. Unfortunately, the filestore does not shrink in size on message retrieval. The workaround is to delete the filestore when there are no messages on the store. The exact steps to follow for this purpose are:
  1. Check that there are no available or pending messages on the destination.

• 1. Untarget the JMS server from the WebLogic Server instance. You can also just stop the WebLogic Server instance.
  2. Delete the filestore (*.dat file) from the filesystem.

Retarget the JMS server to the Weblogic Server instance. If you had stopped the Weblogic Server instance, restart it.

Debug information

The debug parameters that can be specified for diagnosis are: DebugJMSBackEnd, DebugJMSFrontEnd, DebugJMSCommon, DebugJMSDispatcher , and DebugJMSXA.

DebugJMSXA is required only when the pending messages are related to transacted JMS sessions and/or JTA transactions. You can add these parameters:

• In the config.xml file:

<ServerDebug DebugJMSBackEnd=”true” DebugJMSFrontEnd =”true” DebugJMSCommon=”true” DebugJMSDispatcher=”true” DebugJMSXA=”true”/>

• Or as startup parameters in your startWebLogic.cmd script:

-Dweblogic.Debug=weblogic.DebugJMSBackEnd,weblogic.DebugJMSFrontEnd, weblogic.DebugJMSCommon,weblogic.DebugJMSDispatcher,weblogic.DebugJMSXA

• Note that enabling this debug is recommended for a development type of environment due to the verbosity of the debug messages.

 
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at weblogic.messaging.kernel.internal.PersistenceImpl.readMessageBody(PersistenceImpl.java:848)
at weblogic.messaging.kernel.internal.MessageHandle.pin(MessageHandle.java:373)
at weblogic.messaging.kernel.internal.MessageElementImpl.(MessageElementImpl.java:83)

We get this exception on the server on which the queue is available when we try to open a large message from WLS Console.Issue is fixed in 12.2.1 and patches are available in other version. You can check this bug number Bug 19222117.