In our live environment we observed very HIGH CPU utilisation on the boxes. Weblogic JAVA process was hogging the CPU.
To find the culprit thread, we followed the steps outlined here.
The below thread was the culprit thread
“C2 CompilerThread1″ #7 daemon prio=9 os_prio=0 tid=0x00002acb10d48000 nid=0x511 runnable [0x0000000000000000]
There were known code cache issues with C2 Compiler Thread so enabled Native Memory Tracking by adding the flag -XX:NativeMemoryTracking=detail to capture native memory usage ( especially code cache.
jcmd [JVM pid] VM.native_memory > VM.native_memory.txt
jcmd [JVM pid] GC.class_histogram > histogram.txt
pmap -x [JVM pid] > pmap.txt
| Native Memory | Before Size ( MB) | Intermediate Size ( MB) | Final |
| Class | 303 | 773 | 766 |
| Thread | 99 | 157 | 138 |
| Code Cache | 153 | 471 | 568 |
| GC | 391 | 433 | 407 |
We could see a gradual increase in code cache size.
We monitoring the peak usage using JMX and it was around 730 MB.
We turned off tiered compilation (-XX:-TieredCompilation) to reduce code cache usage and this helped stabilize the production environment by reducing CPU usage.