Different Out Of Memory Issues

* Exception in thread “CompilerThread1” java.lang.OutOfMemoryError: requested 793020 bytes for Chunk::new. Out of swap space?

Out of memory while reading in symbol table of /apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl
( 0)  0xc8461230     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 1)  0xc80a5fec     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 2)  0xc7f00420     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 3)  0xc7f00ca0     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 4)  0xc8368d08     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 5)  0xc005b2e4   __pthread_body + 0x44  [/usr/lib/libpthread.1]
( 6)  0xc0065574   __pthread_start + 0x14  [/usr/lib/libpthread.1]
Java out of memory messages are marked with pid: 13828 in /var/adm/syslog/syslog.log.

Possible causes:
– not enough swap space left, or
– kernel parameter MAXDSIZ is very small.

Solution:
Although it appears that an OutOfMemoryError is thrown this apparent exception is reported by the HotSpot VM code when an allocation from the native heap failed and the native heap may be close to exhaustion. The message indicates the size (in bytes) of the request that failed and also indicates what the memory is required for. In some cases the reason will be shown but in most cases the reason will be the name of a source module reporting the allocation failure. If an OutOfMemoryError with this error is thrown it may require using utilities on the operating system to diagnose the issue further. Examples of issues that may not be related to the application are when the operating system is configured with insufficient swap space, or when there is another process on the system that is consuming all memory resources. If neither of these issues is the cause then it is possible that the application is failed due to native leak; for example, application or library code is continuously allocating memory but is not releasing it to the operating system.
For more information: http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf

The recommendation for swap space size in the Solaris is that swap should be configured about 30% of physical RAM.

The following link has suggested a workaround to add ‘-XX:+UseDefaultStackSize -Xss256K’ parameter.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4916142

Add: -XX:CodeCacheMinimumFreeSpace=2M -XX:+ReservedCodeCacheSize=64M -XX:PermSize=128m -XX:MaxPermSize=384m (As per your other JVM settings)

Sun team needs to be involved for this issue. Usually such issues are solved by sun support.

=============================

* java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at weblogic.work.RequestManager.createThreadAndExecute(RequestManager.java:271)
at weblogic.work.RequestManager.executeIt(RequestManager.java:245)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)

Solution:
1) Set kernel parameter maxdsiz to a higher value
2) Reduce the current heap size.
3) Check the kernel values:  ulimit -a
4) If the NPROC soft limit is lower than the hard limit, increase it as needed: ulimit -u <new value>.
Check the Operating System documentation to make changes permanent at the OS configuration files.
5) Need to reduce the JVM stack size and the OS stack size both. Set the -xss in java options and set ulimit -s on the OS level.

=============================

* java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.lang.Class.getConstructor0(Unknown Source)
at java.lang.Class.getConstructor(Unknown Source)

Solution: Increase the max permgen space -XX:MaxPermSize=256m

There can be a leak in the permgen objects. If tuning parameters do not resolve the issue, we need to use the memory leak detector tools and find out which instances in the permgen space are not getting cleared.

=============================

* java.lang.OutOfMemoryError: allocLargeObjectOrArray – Object size: 372032, Num elements: 372012
* java.lang.outofmemoryerror: nativeGetNewTLA

Solution: -XXtlasize:128k -XXlargeobjectlimit:128k

If this does not solve the issue, we need to check in the application code for the large objects being created and not being destroyed. Take JRA Recording (Oracle JRockit) or use JConsole and memory leak detector tools (JMAP, JHAT) for analysis on the

=============================

* java.lang.OutOfMemoryError: Java Heap Space

Solution: First thing that needs to be checked is the gc logs. Need to check whether the garbage collection is happening properly. If the heap keeps gradually increasing even after full gc, tune the gc algorithms and check if the behavior is the same. Use memory leak detector tools for both sun jdk and JRockit to check which instances from the application are not getting destroyed.

If this is not the case and the application genuinely needs more memory, increase the heap size by using the parameters:
Example: -Xms2048m -Xmx2048m

=============================

* java.lang.StackOverflowError

Solution: Stack over flow error is usually generated due to a recursive call made by the application (infinite recursion), or its because of an attempt to allocate more memory on the stack than will fit. This is usually the result of creating local array variables that are far too large for the current stack.

For the first possibility, we need to check the application code as to where is the recursive call being made.
For the second possibility, we can increase the JVM stack size by the parameter : Example: -Xss512K

=============================

java.lang.OutOfMemoryError: GC overhead limit exceeded.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209) 

Try adding this jvm option
-XX:-UseGCOverheadLimit

 

14 comments

  1. I noticed that our Admin_start.cfg file on weblogic 10 on Solaris has no MEM_ARGS options (e.g., -Xms256m -Xmx512m -XX:MaxPermSize=192m -Xss128k), and it reports permgen out of memory messages lately. Is there a reason that we should or should not have a MEM_ARGS= option listed?

    1. How do you start your Admin Server?
      I am not aware of any Admin_start.cfg file, where do you use it?

      Can you please give more details?

      -Faisal

  2. Hi,

    Can we trace out application causing memory leak in weblogic cluster if we have heap dump(.hprof) at the time of issue.

    Thanks,
    Swaroop

  3. Hi

    why we need swapspace? where we can assign the swap space?how much space we need ?

    1. Hi Pavan,

      Swap space temporarily holds memory pages that are inactive. Swap space is used when your system decides that it needs physical memory for active processes and there is insufficient unused physical memory available. This helps in memory paging.

      The OS will list the recommended amount of swap space (virtual memory) for your computer. If using Mac OS X, it is not necessary to adjust swap space. By default, OS X creates and adjusts swap space as needed by the system.

      For further reading refer the link.

      https://help.ubuntu.com/community/SwapFaq#What is swap?

      Regards,
      Anandraj

  4. Occasionally the admin server lose JMX connection with a managed server and does not display the state and other detailed information about the server. The state infromation is not recovered unless the managed server is restarted.

    1. Hi Praveen,

      Does the managed server and the application deployed still work properly? Can you please elaborate more on the issue?

      -Divya

  5. Hi All,

    Can any one please explain how works GC and Heap size arguments functionality of application in weblogic server.

    Thanks in Advance

    1. which version of WLS you are using?

      you can check your setDomainEnv.cmd file for -Xmx and -Xms parameters for the default heap sizes.

  6. Hi am New to weblogic i have 3 doubts please help me
    1. In a domain only Admin server is runing reaming All managed servers are down how will you troubleshoot this issue.

    2.How will you troubleshoot if Application is down?
    3.what are the improtant parameters in nodemanager?

    1. Hi Rajesh,

      1. In a domain only Admin server is runing reaming All managed servers are down how will you troubleshoot this issue.

      I would first check the server status either from the Admin console or from the respective server logs in case you dont have access to the admin logs.

      Then I would do a root cause about the failure reason.

      2.How will you troubleshoot if Application is down?

      First thing I would check if the application state, if its started and running. If its giving 404 page not found exception

      Then I would check if the servers are hung , stopped or running out of memory. And troubleshoot further.

      3.what are the important parameters in nodemanager?

      There are quite a few like
      ListenerPort
      ListenerAddress etc.

      You can find them in the nodemanager.properties file under \wlserver_10.3\common\nodemanager folder.

      Cheers,
      Anandraj

Comments are closed.