There could be scenarios like Server Hang, Crashes, Performance delays where you might need to capture Weblogic Thread Dumps for further analysis.
Thread dumps provide a snapshot of the current active threads. It provides the stack trace of all the java threads in the JVM.
There are different ways to capture thread dumps; some are based on the operating systems.
On Windows:
- Go to the server standard output and press a Control + Break and copy the thread dump onto a file
On UNIX/ Linux
Find the process id for your server
- Ps –ef | grep java
- Kill -3 <pid>
WebLogic utilities to capture thread dumps
1. webLogic.Admin utility
a. Open a command prompt , set the classpath running <DOMAIN_HOME>/bin/setDomain.env
b. Execute the below command
java weblogic.Admin -url t3://localhost:7001 -username weblogic -password weblogic1 THREAD_DUMP
Note: Thread Dump will be printed to the servers standard out (by default, the shell in which the server is running).
2. Using Admin Console
a. Log into the Admin Console , click on the server
b. Click onto the Server –> Monitoring –> Threads
c. Click on the Dump Thread Stack
3. Using WLST (WebLogic Scripting Tool)
a. Save and execute the below snippet as ThreadDump.py
************************************
connect('weblogic','weblogic1','t3://localhost:7001') cd('Servers') cd('AdminServer') threadDump() disconnect() exit()************************************
Note: The thread dumps get stored in the location from where you run the WLST script
If your Server is running as windows service, then follow the below steps
1. Open a command prompt and execute the below command
WL_HOMEbinbeasvc -dump -svcname:service-name
There are other tools that can be used to capture thread dumps for example jrcmd (for JRockit) ,Samurai etc.
NOTE: It is recommended to capture a set of 6-7 thread dumps at an interval 8-10 seconds to find a pattern in the thread execution.
Cheers,
Wonders Team
Hi, i want to know one information, once we enter the command to take thread dump, that file where it will stored and what is the location of that file.
Very nice efforts regarding weblogic material.
Many of the professional used to visit this site.
thanks Sushil…
@naresh
When ever you execute the above mentioned process the out put of the thread dump will be stored in the log file of the particular server with .out extension you can view that file using the vi editor in which you can find one line state the start of the thread dump
Thank you
Hi ,
This is a great place for finding details / knowledge about WLS .
Just to add to the post here , thread dump can be created by using the Sun JDK – jstack command as well.
I am seeing this to be a far more easier way to capture the thread dumps .
syntax : jstack -l
Thanks
Srinath
thanks for sharing with everyone srinath!
Hi All,
Please do the needful.
“main” waiting for lock weblogic.t3.srvr.T3Srvr@46a746a7
WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:167)
weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:981)
weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:490)
weblogic.Server.main(Server.java:71)
“JIT Compilation Thread” RUNNABLE
om.ibm.tools.attach.javaSE.CommonDirectory.waitSemaphore(CommonDirector
y.java:220)
com.ibm.tools.attach.javaSE.AttachHandler$WaitLoop.waitForNotification(A
ttachHandler.java:329)
com.ibm.tools.attach.javaSE.AttachHandler$WaitLoop.run(AttachHandler.jav
a:396)
“Thread-7” waiting for lock
java.util.Timer$TimerImpl@47034703 WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:167)
java.util.Timer$TimerImpl.run(Timer.java:227)
“Thread-8” waiting for lock
java.util.Timer$TimerImpl@21e521e5 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
java.util.Timer$TimerImpl.run(Timer.java:247)
“[STUCK] ExecuteThread: ‘0’ for queue:
‘weblogic.kernel.Default (self-tuning)'” waiting for lock
oracle.tip.adapter.db.InboundWork@5640564 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
oracle.tip.adapter.db.InboundWork.run(InboundWork.java:609)
oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.
java:43)
weblogic.work.ContextWrap.run(ContextWrap.java:41)
weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.
java:117)
java.lang.Thread.run(Thread.java:736)
“weblogic.timers.TimerThread” waiting for lock
weblogic.timers.internal.TimerThread@6150615 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)
“[STUCK] ExecuteThread: ‘1’ for queue:
‘weblogic.kernel.Default (self-tuning)'” waiting for lock
oracle.tip.adapter.db.InboundWork@6cc36cc3 TIMED_WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
oracle.tip.adapter.db.InboundWork.run(InboundWork.java:609)
oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.
java:43)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
“VDE Transaction Processor Thread” waiting for lock
com.octetstring.vde.backend.standard.TransactionProcessor@791b791b
WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:167)
com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(Tra
nsactionProcessor.java:367)
com.octetstring.vde.backend.standard.TransactionProcessor.run(Transactio
nProcessor.java:212)
“DoSManager” TIMED_WAITING native
java.lang.Thread.sleep(Native Method)
java.lang.Thread.sleep(Thread.java:851)
com.octetstring.vde.DoSManager.run(DoSManager.java:433)
“JMX Framework document pooling thread” TIMED_WAITING native
java.lang.Thread.sleep(Native Method)
java.lang.Thread.sleep(Thread.java:851)
oracle.as.config.notification.filesystem.WatchingDocumentChangeNotifier.
run(WatchingDocumentChangeNotifier.java:250)
java.lang.Thread.run(Thread.java:736)
“Runtime Authorization Event Notifier” waiting for lock
oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImp
l@1be31be3 WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:167)
oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImp
l.run(RuntimeEventNotifierImpl.java:217)
java.lang.Thread.run(Thread.java:736)
“Thread-22” waiting for lock
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56
6d566d WAITING native
“[STANDBY] ExecuteThread: ’28’ for queue:
‘weblogic.kernel.Default (self-tuning)'” waiting for lock
weblogic.work.ExecuteThread@3c6d3c6d WAITING native
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:167)
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
weblogic.work.ExecuteThread.run(ExecuteThread.java:171)
I have taken Thread dump. I don’t know how to analyses and how to delete stuck threads?
Kindly help on the above Thread dump.
Thanks,
Venkat
Hi Venkat,
Please send the log files at
khan.faysal06@gmail.com
When did the thread dump happen?
You can use samurai to analyse thread dumps…
Very effective ways for Thread-Dump.
Thanks for sharing all these here.
thanks Abhishek!
How to divert the thread dumps to a diff file instead of writing in stdouterr.log. I tried kill -3 pid >> dump1.log but no luck. Any one can help me here.
./startWeblogic.sh > wls.log
kill -3 pid
thread dump will go to wls.log
Hi All,
We are getting lot of blocked threads can any one help me.
“ajp-3.239.144.86-42009-27” daemon prio=1 tid=0x000000004b39c0e0 nid=0x3165 waiting for monitor entry [0x000000005195c000..0x000000005195db10] at com.waveset.server.SubjectAuthCache.subjectHasRight(SubjectAuthCache.java:1437) – waiting to lock (a com.waveset.server.SubjectAuthCache) at com.waveset.session.LocalSession.subjectHasRight(LocalSession.java:3880) at com.waveset.ui.PageNavigation.hasPerm(PageNavigation.java:570) at com.waveset.ui.PageNavigation.access$100(PageNavigation.java:60) at com.waveset.ui.PageNavigation$DefaultAccessFilter.accept(PageNavigation.java:497) at com.waveset.ui.PageNavigation.authorized(PageNavigation.java:440) at com.waveset.ui.PageNavigation.authorized(PageNavigation.java:403) at com.waveset.ui.PageInformation.authorized(PageInformation.java:3160) at com.waveset.ui.PageInformation.getMainNavigationTabs(PageInformation.java:2802) at com.waveset.ui.PageInformation.getMainNavigation(PageInformation.java:2736) at org.apache.jsp.account.list_jsp._jspService(list_jsp.java:808) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:387) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.sun.idm.profiler.instrumentation.RequestTimingFilter.doFilter(RequestTimingFilter.java:76) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:595)
Thanks,
Raghu
Hi,
i am trying to generate thread dump using kill -3 ,
but unable see the thread dump where it is generating, as of above post thread will generate live server.out (or) will it create separete server.out, plz suggest for the same.
Regards,
Ravi
Hi Ravi,
It should go to server stdout.
How are you starting the server? where are you redirecting the o/p?
Thanks in advance to weblogic wonders!!!!!!!!!!
Please can anyone help me how to redirect the Thread_dump from KILL -3 PID to specific directory.
–shanth kumar
i think this should do
kill -3 processid >> /abc/def/threaddump.txt
How to take thread dumps using IBM jvm??I want to schedule the dump using a script..
Please reply
You can write a jython script with below snippet.
jvm = AdminControl.completeObjectName(‘type=JVM,process=server1,*’)
AdminControl.invoke(jvm, ‘dumpThreads’)
Else you can use jstack utility to generate a thread dump as below.
#!/bin/bash
if [ $# -eq 0 ]; then
echo >&2 “Usage: jstackSeries [ [ ] ]”
echo >&2 ” Defaults: count = 10, delay = 1 (seconds)”
exit 1
fi
pid=$1 # required
count=${2:-10} # defaults to 10 times
delay=${3:-1} # defaults to 1 second
while [ $count -gt 0 ]
do
jstack $pid >jstack.$pid.$(date +%H%M%S.%N)
sleep $delay
let count–
echo -n “.”
done
Reference:
http://howtodoinjava.com/2012/12/19/how-to-get-thread-dump-in-linux-using-jstack/
Very nice work khan ji … This site helps us a lot
Thank You
Thanks Sudheer!!!