Troubleshooting Out Of Memory Errors in Weblogic using Eclipse Memory Ananlyser

Eclipse Memory Ananlyser is a very useful tool to analyze heap dumps. It has a lot of features such as Memory Leak detection where it runs an automated test to determine the suspected leaks.

Here is the Procedure

Step 1). Start the Weblogic Server, with the application in active state which causes memory leak.

Step 2). Get the process id of the server using jps


Step 3). Acess the application that causes memory leak

Step 4).Take heap dump at regular interval using jmap.

jmap -dump:format=b,file=dump1.bin 5004

Step 5).Open the Heap Dump in Eclipse Memory Ananlyzer (U can download it from http://www.eclipse.org/mat/downloads.php) Approximate size is 42 MB..Just u need to extract this Zip then u can directly start (no Installation needed)

Observe the heap usage of Objects in the heap dumps. If the object instance keeps on increasing in the subequent heap dumps, force a garbage collection from the Weblogic Server console


Take heap dumps again and open in the Eclipse Memory Analyzer. If the number of instances still don’t go down for those objects, you can expect to see this

<Jul 16, 2010 10:49:15 AM IST> <Critical> <Health> <BEA-310003> <Free memory in
the server is 47,856 bytes. There is danger of OutOfMemoryError>
Exception in thread “Thread-12” java.lang.OutOfMemoryError: Java heap space
at demo.MemoryLeakTest.runTest(MemoryLeakTest.java:14)
at jsp_servlet.__memoryleak$1.run(__memoryleak.java:86)
at java.lang.Thread.run(Thread.java:619)

.

If the leak is happening due to a Weblogic Class, it can be a known issue or an undiscovered BUG. You need to get in touch with Oracle Support. If it’s an Application Class, you need to contact the developers. Out of Memory can also happen dude to third party codes such as database drivers.

28 comments

  1. Hi Faisal, It is great work!!
    This blog is pretty much helpful for WebLogic Administrators. The way you wrote it is really cool!! one of the wonder!!

    Expecting more wonders from your team…
    Keep doing good work… I mean wonders

  4. Hi,

    After Analysing heap dump

    weblogic.JMS.frontend.FEConsumer class consuming more than 1GB.why this happens.mine is 9.2MP3

    1. Hi Smitha,

      This can be a bug in the application/application server code.

      Can you develop a simple test case to reproduce this issue?

      Thanks,
      Faisal

  5. Hi’

    I have downloaded the eclipse memory analyzer from the location provided however when I click on the MemoryAnalyzer.exe, its is not starting the eclipse. Can you advice me, I am using Win XP.

    Thanks
    Yatan

    1. Hi Yatan,

      What is the error you are getting?
      Can you share screenshots?

      Cheers!
      Faisal

  6. Hi,
    Good Article,was very very useful for me.

    I followed the above steps for leak Suspects,it showed me the below Problem Suspect 1

    One instance of “weblogic.jdbc.common.internal.GenericConnectionPool” loaded by “sun.misc.Launcher$AppClassLoader @ 0x866ba340” occupies 335,917,304 (90.26%) bytes. The memory is accumulated in one instance of “java.util.LinkedList$Entry” loaded by “”.

    Keywords
    weblogic.jdbc.common.internal.GenericConnectionPool
    sun.misc.Launcher$AppClassLoader @ 0x866ba340
    java.util.LinkedList$Entry

    Can you please tell how can i suspect which page is creating a page leak.

    Regards
    Fabian

    1. Take multiple heap dumps and see if the count of those objects are increasing or not.. also force the garbage collection from the admin console and take heap dump after that. If those objects are still in the heap then determine if the application is using those objects… hope this helps..

  7. Thanks for sharing the information may i know how to handle the performance related issues

  11. Hi All,
    I am facing an issue that one of my managed server in SOA domain was running but with an warning message of STUCK THREAD. when I investigated in logs I found as follows:
    LOGS:
    ####
    #### < ResourceAllocationException thrown by resource adapter on call to ManagedConnectionFactory.createManagedConnection(): “Internal error: Cannot obtain XAConnection weblogic.common.ResourceException:
    SOADataSource(SOADataSource-rac1): Pool SOADataSource-rac1 is Suspended, cannot allocate resources to applications..
    SOADataSource(SOADataSource-rac2): Pool SOADataSource-rac2 is Suspended, cannot allocate resources to applications..
    SOADataSource(SOADataSource-rac3): Pool SOADataSource-rac3 is Suspended, cannot allocate resources to applications..
    SOADataSource(SOADataSource-rac4): Pool SOADataSource-rac4 is Suspended, cannot allocate resources to applications..

    at weblogic.jdbc.common.internal.MultiPool.searchHighAvail(MultiPool.java:505)
    at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:178)
    at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:89)
    at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1677)
    at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:446)
    at weblogic.jdbc.jta.DataSource.connect(DataSource.java:403)

    But all my applications are working up & running.
    Please help me in resolving this issue ASAP.

    Thanks
    Lokesh

    1. you can try to increase the max_test_wait_secs

      -Dweblogic.resourcepool.max_test_wait_secs=30

  12. Hi Faisal, Recently in our new environment, we got out of memory and perm gen errors while doing deployment on weblogic server. I have taken heap dump using jmap command.

    After that incresed the heap size from 1GB to 2GB and perm gen size from 512m to 640m. And then taken restart of weblogic instance and after that deployed application succesfully.

    After analying the generated heap dump using MAT tool, found the following classloader errors in analyzer console.
    =================================================================
    20 instances of “weblogic.utils.classloaders.ChangeAwareClassLoader”, loaded by “sun.misc.Launcher$AppClassLoader @ 0xc0000f40” occupy 2,42,85,568 (27.16%) bytes.

    Biggest instances:

    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc14e0770 – 30,27,624 (3.39%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc149e270 – 21,77,352 (2.44%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc0dc9300 – 21,23,592 (2.38%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc4b299d8 – 21,19,592 (2.37%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d556c0 – 19,94,592 (2.23%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d51248 – 19,17,592 (2.14%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc1358b08 – 18,90,544 (2.11%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d51738 – 18,42,968 (2.06%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc1359358 – 18,30,656 (2.05%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc13662c8 – 18,24,680 (2.04%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc13f45b0 – 17,62,784 (1.97%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc4ed4ef8 – 17,07,752 (1.91%) bytes.

    Keywords
    sun.misc.Launcher$AppClassLoader @ 0xc0000f40
    weblogic.utils.classloaders.ChangeAwareClassLoader
    ==============================================================

    What may be the reason for this issue and kindly let me know how to prevent this kind of issues in future.

    Thanks
    Krishna

    1. Hi Faisal, Recently in our new environment, we got out of memory and perm gen errors while doing deployment on weblogic server. I have taken heap dump using jmap command.
      After that incresed the heap size from 1GB to 2GB and perm gen size from 512m to 640m. And then taken restart of weblogic instance and after that deployed application succesfully.
      After analying the generated heap dump using MAT tool, found the following classloader errors in analyzer console.
      =================================================================
      20 instances of “weblogic.utils.classloaders.ChangeAwareClassLoader”, loaded by “sun.misc.Launcher$AppClassLoader @ 0xc0000f40″ occupy 2,42,85,568 (27.16%) bytes.

      Biggest instances:
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc14e0770 – 30,27,624 (3.39%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc149e270 – 21,77,352 (2.44%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc0dc9300 – 21,23,592 (2.38%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc4b299d8 – 21,19,592 (2.37%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d556c0 – 19,94,592 (2.23%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d51248 – 19,17,592 (2.14%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc1358b08 – 18,90,544 (2.11%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc3d51738 – 18,42,968 (2.06%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc1359358 – 18,30,656 (2.05%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc13662c8 – 18,24,680 (2.04%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc13f45b0 – 17,62,784 (1.97%) bytes.
      •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc4ed4ef8 – 17,07,752 (1.91%) bytes.
      Keywords
      sun.misc.Launcher$AppClassLoader @ 0xc0000f40
      weblogic.utils.classloaders.ChangeAwareClassLoader
      ==============================================================
      What may be the reason for this issue and kindly let me know how to prevent this kind of issues in future.

      Thanks
      Krishna

  13. Hi Faisal, Thanks for the article. Recently in our new environment, we got out of memory and perm gen errors while doing deployment on weblogic server. I have taken heap dump using jmap command. After that incresed the heap size from 1GB to 2GB and perm gen size from 512m to 640m. And then taken restart of weblogic instance and after that deployed application succesfully.
    After analying the heap dump using IBM memory analyzer tool, foundt the following classloader errors in analyzer console.
    =================================================================
    20 instances of “weblogic.utils.classloaders.ChangeAwareClassLoader”, loaded by “sun.misc.Launcher$AppClassLoader @ 0xc0000f40” occupy 2,42,85,568 (27.16%) bytes.
    Biggest instances:
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc14e0770 – 30,27,624 (3.39%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc149e270 – 21,77,352 (2.44%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc0dc9300 – 21,23,592 (2.38%) bytes.
    •weblogic.utils.classloaders.ChangeAwareClassLoader @ 0xc4b299d8 – 21,19,592 (2.37%) bytes.

    Keywords
    sun.misc.Launcher$AppClassLoader @ 0xc0000f40
    weblogic.utils.classloaders.ChangeAwareClassLoader
    ==============================================================
    kindly suggest how to prevent this kind of issues in future.

    Thanks
    Krishna

    1. Hi Krishna,

      You should find out the memory requiremets of the application using memory profiling tool such as jprobe, yourkit.. then set the memory arguments accordingly inorder to avoid such issues in future..

      Thanks,
      Faisal

  14. Anybody can help me to identify de cause about this out of memory? I think tha real cause is an application….

    Caused By: weblogic.transaction.internal.TimedOutException: Transaction timed out after 30 seconds

    BEA1-7FE25B07C4DEF02662E1

    >

    MessageType: 2

    MessageNum: 161

    MessageText: Cannot open catalog; Message number = 161

    AppendedText: Exception in thread “[ACTIVE] ExecuteThread: ‘4’ for queue: ‘weblogic.kernel.Default (self-tuning)'” java.lang.OutOfMemoryError: (class: weblog

    ic/kernel/KernelLogger)

    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:267)

    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

    Exception in thread “[STANDBY] ExecuteThread: ’10’ for queue: ‘weblogic.kernel.Default (self-tuning)'” java.lang.OutOfMemoryError: (class: weblogic/kernel/Ke

    rnelLogger)

Comments are closed.