Weblogic Thread Dumps

There could be scenarios like Server Hang, Crashes, Performance delays where you might need to capture Weblogic Thread Dumps for further analysis.

Thread dumps provide a snapshot of the current active threads. It provides the stack trace of all the java threads in the JVM.

There are different ways to capture thread dumps; some are based on the operating systems.

On Windows:

  1. Go to the server standard output and press a Control + Break and copy the thread dump onto a file

On UNIX/ Linux

Find the process id for your server

  1. Ps –ef  | grep java
  2. Kill -3 <pid>

 

WebLogic utilities to capture thread dumps

1. webLogic.Admin utility

a. Open a command prompt , set the classpath running <DOMAIN_HOME>/bin/setDomain.env

b. Execute the below command

java weblogic.Admin -url t3://localhost:7001 -username weblogic -password weblogic1 THREAD_DUMP

Note: Thread Dump will be printed to the servers standard out (by default, the shell in which the server is running).

2. Using Admin Console

a. Log into the Admin Console , click on the server

b. Click onto the Server  –> Monitoring  –> Threads

c. Click on the Dump Thread Stack

 

 

3. Using WLST (WebLogic Scripting Tool)

a. Save and execute  the below snippet as ThreadDump.py

************************************

connect('weblogic','weblogic1','t3://localhost:7001')

cd('Servers')

cd('AdminServer')

threadDump()

disconnect()

exit()

************************************

Note: The thread dumps get stored in the location from where you run the WLST script

 

If your Server is running as windows service, then follow the below steps

1. Open a command prompt and execute the below command

WL_HOMEbinbeasvc -dump -svcname:service-name

There are other tools that can be used to capture thread dumps for example jrcmd (for JRockit) ,Samurai etc.

NOTE: It is recommended to capture a set of 6-7 thread dumps at an interval 8-10 seconds to find a pattern in the thread execution.

Cheers,

Wonders Team

22 comments

  1. Hi, i want to know one information, once we enter the command to take thread dump, that file where it will stored and what is the location of that file.

  3. @naresh

    When ever you execute the above mentioned process the out put of the thread dump will be stored in the log file of the particular server with .out extension you can view that file using the vi editor in which you can find one line state the start of the thread dump

    Thank you

  4. Hi ,

    This is a great place for finding details / knowledge about WLS .

    Just to add to the post here , thread dump can be created by using the Sun JDK – jstack command as well.

    I am seeing this to be a far more easier way to capture the thread dumps .

    syntax : jstack -l

    Thanks
    Srinath

  5. Hi All,

    Please do the needful.

    “main” waiting for lock weblogic.t3.srvr.T3Srvr@46a746a7
    WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:167)

    weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:981)

    weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:490)

    weblogic.Server.main(Server.java:71)

    “JIT Compilation Thread” RUNNABLE
    om.ibm.tools.attach.javaSE.CommonDirectory.waitSemaphore(CommonDirector
    y.java:220)

    com.ibm.tools.attach.javaSE.AttachHandler$WaitLoop.waitForNotification(A
    ttachHandler.java:329)

    com.ibm.tools.attach.javaSE.AttachHandler$WaitLoop.run(AttachHandler.jav
    a:396)

    “Thread-7” waiting for lock
    java.util.Timer$TimerImpl@47034703 WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:167)

    java.util.Timer$TimerImpl.run(Timer.java:227)

    “Thread-8” waiting for lock
    java.util.Timer$TimerImpl@21e521e5 TIMED_WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:196)

    java.util.Timer$TimerImpl.run(Timer.java:247)

    “[STUCK] ExecuteThread: ‘0’ for queue:
    ‘weblogic.kernel.Default (self-tuning)'” waiting for lock
    oracle.tip.adapter.db.InboundWork@5640564 TIMED_WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:196)

    oracle.tip.adapter.db.InboundWork.run(InboundWork.java:609)

    oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.
    java:43)

    weblogic.work.ContextWrap.run(ContextWrap.java:41)

    weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.
    java:117)

    java.lang.Thread.run(Thread.java:736)

    “weblogic.timers.TimerThread” waiting for lock
    weblogic.timers.internal.TimerThread@6150615 TIMED_WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:196)

    weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)

    “[STUCK] ExecuteThread: ‘1’ for queue:
    ‘weblogic.kernel.Default (self-tuning)'” waiting for lock
    oracle.tip.adapter.db.InboundWork@6cc36cc3 TIMED_WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:196)

    oracle.tip.adapter.db.InboundWork.run(InboundWork.java:609)

    oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.
    java:43)

    weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)

    weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

    “VDE Transaction Processor Thread” waiting for lock
    com.octetstring.vde.backend.standard.TransactionProcessor@791b791b
    WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:167)

    com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(Tra
    nsactionProcessor.java:367)

    com.octetstring.vde.backend.standard.TransactionProcessor.run(Transactio
    nProcessor.java:212)

    “DoSManager” TIMED_WAITING native

    java.lang.Thread.sleep(Native Method)

    java.lang.Thread.sleep(Thread.java:851)

    com.octetstring.vde.DoSManager.run(DoSManager.java:433)

    “JMX Framework document pooling thread” TIMED_WAITING native

    java.lang.Thread.sleep(Native Method)

    java.lang.Thread.sleep(Thread.java:851)

    oracle.as.config.notification.filesystem.WatchingDocumentChangeNotifier.
    run(WatchingDocumentChangeNotifier.java:250)

    java.lang.Thread.run(Thread.java:736)

    “Runtime Authorization Event Notifier” waiting for lock
    oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImp
    l@1be31be3 WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:167)

    oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImp
    l.run(RuntimeEventNotifierImpl.java:217)

    java.lang.Thread.run(Thread.java:736)

    “Thread-22” waiting for lock
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56
    6d566d WAITING native

    “[STANDBY] ExecuteThread: ’28’ for queue:
    ‘weblogic.kernel.Default (self-tuning)'” waiting for lock
    weblogic.work.ExecuteThread@3c6d3c6d WAITING native

    java.lang.Object.wait(Native Method)

    java.lang.Object.wait(Object.java:167)

    weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)

    weblogic.work.ExecuteThread.run(ExecuteThread.java:171)

    I have taken Thread dump. I don’t know how to analyses and how to delete stuck threads?

    Kindly help on the above Thread dump.

    Thanks,
    Venkat

  7. How to divert the thread dumps to a diff file instead of writing in stdouterr.log. I tried kill -3 pid >> dump1.log but no luck. Any one can help me here.

  8. Hi All,

    We are getting lot of blocked threads can any one help me.

    “ajp-3.239.144.86-42009-27” daemon prio=1 tid=0x000000004b39c0e0 nid=0x3165 waiting for monitor entry [0x000000005195c000..0x000000005195db10]      at com.waveset.server.SubjectAuthCache.subjectHasRight(SubjectAuthCache.java:1437)      – waiting to lock (a com.waveset.server.SubjectAuthCache)      at com.waveset.session.LocalSession.subjectHasRight(LocalSession.java:3880)      at com.waveset.ui.PageNavigation.hasPerm(PageNavigation.java:570)      at com.waveset.ui.PageNavigation.access$100(PageNavigation.java:60)      at com.waveset.ui.PageNavigation$DefaultAccessFilter.accept(PageNavigation.java:497)      at com.waveset.ui.PageNavigation.authorized(PageNavigation.java:440)      at com.waveset.ui.PageNavigation.authorized(PageNavigation.java:403)      at com.waveset.ui.PageInformation.authorized(PageInformation.java:3160)      at com.waveset.ui.PageInformation.getMainNavigationTabs(PageInformation.java:2802)      at com.waveset.ui.PageInformation.getMainNavigation(PageInformation.java:2736)      at org.apache.jsp.account.list_jsp._jspService(list_jsp.java:808)      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:387)      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)      at com.sun.idm.profiler.instrumentation.RequestTimingFilter.doFilter(RequestTimingFilter.java:76)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381)      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)      at java.lang.Thread.run(Thread.java:595)

    Thanks,
    Raghu

  9. Hi,

    i am trying to generate thread dump using kill -3 ,
    but unable see the thread dump where it is generating, as of above post thread will generate live server.out (or) will it create separete server.out, plz suggest for the same.

    Regards,
    Ravi

    1. Hi Ravi,

      It should go to server stdout.
      How are you starting the server? where are you redirecting the o/p?

  10. Thanks in advance to weblogic wonders!!!!!!!!!!

    Please can anyone help me how to redirect the Thread_dump from KILL -3 PID to specific directory.

    –shanth kumar

    1. You can write a jython script with below snippet.

      jvm = AdminControl.completeObjectName(‘type=JVM,process=server1,*’)
      AdminControl.invoke(jvm, ‘dumpThreads’)

      Else you can use jstack utility to generate a thread dump as below.

      #!/bin/bash

      if [ $# -eq 0 ]; then
      echo >&2 “Usage: jstackSeries [ [ ] ]”
      echo >&2 ” Defaults: count = 10, delay = 1 (seconds)”
      exit 1
      fi

      pid=$1 # required
      count=${2:-10} # defaults to 10 times
      delay=${3:-1} # defaults to 1 second

      while [ $count -gt 0 ]
      do
      jstack $pid >jstack.$pid.$(date +%H%M%S.%N)
      sleep $delay
      let count–
      echo -n “.”
      done

      Reference:

      http://howtodoinjava.com/2012/12/19/how-to-get-thread-dump-in-linux-using-jstack/

Comments are closed.