<code>import java.io.IOException;
import java.net.MalformedURLException;
import javax.management.openmbean.*;
import javax.naming.*;
import javax.management.*;
import javax.management.remote.*;
import java.util.*;
import javax.management.ObjectName.*;
import weblogic.management.mbeanservers.*;
import weblogic.management.configuration.*;
import weblogic.health.HealthState;
import java.net.*;
import java.io.InputStream;
import javax.xml.parsers.*;
import org.xml.sax.InputSource;
import org.w3c.dom.*;
import java.io.*;
public class HeapMonitor {
private static MBeanServerConnection connection;
private static JMXConnector connector;
public static void main(String args[]) {
try {
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://stage2cpp094.qa.paypal.com:8888/jmxrmi");
JMXConnector jmxc = JMXConnectorFactory.connect(url, null);
jmxc.connect();
System.out.println("Connection successfull");
Object memoryMbean = null;
CompositeData memoryMbeanCd = null;
Object codeCacheMbean = null;
CompositeData codeCacheMbeanCd = null;
jmxc.getMBeanServerConnection().invoke(new ObjectName("java.lang:type=Memory"), "gc", null, null);
memoryMbean = jmxc.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=Memory"), "NonHeapMemoryUsage");
memoryMbeanCd = (CompositeData) memoryMbean;
System.out.println("Max Native Memory Usage: " + " " + memoryMbeanCd.get("max"));
codeCacheMbean = jmxc.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=MemoryPool,name=Code Cache"), "PeakUsage");
codeCacheMbeanCd = (CompositeData) codeCacheMbean;
System.out.println("Peak Code Cache Usage: " + " " + codeCacheMbeanCd.get("max"));
} catch (Exception e) {
e.printStackTrace();
}
}
}</code>
High CPU Usage due to C2 CompilerThread
In our live environment we observed very HIGH CPU utilisation on the boxes. Weblogic JAVA process was hogging the CPU.
To find the culprit thread, we followed the steps outlined here.
The below thread was the culprit thread
“C2 CompilerThread1″ #7 daemon prio=9 os_prio=0 tid=0x00002acb10d48000 nid=0x511 runnable [0x0000000000000000]
There were known code cache issues with C2 Compiler Thread so enabled Native Memory Tracking by adding the flag -XX:NativeMemoryTracking=detail to capture native memory usage ( especially code cache.
jcmd [JVM pid] VM.native_memory > VM.native_memory.txt
jcmd [JVM pid] GC.class_histogram > histogram.txt
pmap -x [JVM pid] > pmap.txt
| Native Memory | Before Size ( MB) | Intermediate Size ( MB) | Final |
| Class | 303 | 773 | 766 |
| Thread | 99 | 157 | 138 |
| Code Cache | 153 | 471 | 568 |
| GC | 391 | 433 | 407 |
We could see a gradual increase in code cache size.
We monitoring the peak usage using JMX and it was around 730 MB.
We turned off tiered compilation (-XX:-TieredCompilation) to reduce code cache usage and this helped stabilize the production environment by reducing CPU usage.
Updated deployment descriptors of WebLogic Server
U
weblogic.xml.process.XMLProcessingException: Error: For EJB modules, deployment plan overrides are not supported with DTD based deployment descriptors. The EJB module prbeans.jar is attempting to apply a deployment plan to DTD based descriptors. Please update your EJB deployment descriptors to be schema based before attempting to apply the deployment plan override to this module. The weblogic.DDConverter utility can be used to automatically update your descriptors. Run the weblogic.DDConverter utility on your application to update the deployment descriptors. After the descriptors are updated, the deployment plan overrides can be applied to the EJB module properly.
Message icon – Error Error: For EJB modules, deployment plan overrides are not supported with DTD based deployment descriptors. The EJB module prbeans.jar is attempting to apply a deployment plan to DTD based descriptors. Please update your EJB deployment descriptors to be schema based before attempting to apply the deployment plan override to this module. The weblogic.DDConverter utility can be used to automatically update your descriptors. Run the weblogic.DDConverter utility on your application to update the deployment descriptors. After the descriptors are updated, the deployment plan overrides can be applied to the EJB module properly.
source /x/bea-wl1213/wlserver/server/bin/setWLSEnv.sh
java weblogic.DDConverter -d prpc_j2ee14.ear
Create and configure Extended Logger for Weblogic
- Create an extended logger by compiling the code below and creating a jar ( el.jar in this example)
import weblogic.servlet.logging.CustomELFLogger;
import weblogic.servlet.logging.FormatStringBuffer;
import weblogic.servlet.logging.HttpAccountingInfo;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.Map;
public class PaypalCustomField implements CustomELFLogger
{
public void logField(HttpAccountingInfo metrics,
FormatStringBuffer buff) {
buff.appendValueOrDash(metrics.getHeader(“Cookie”));
Map<String, String> map = new HashMap<String, String>();
StringBuffer sbuf = new StringBuffer();
Enumeration headerNames = metrics.getHeaderNames();
while (headerNames.hasMoreElements()) {
String key = (String) headerNames.nextElement();
String value = metrics.getHeader(key);
sbuf.append(key+”:”+value+”\n”);
}
int length = sbuf.toString().length();
System.out.println(“headers==”+sbuf.toString());
System.out.println(“length==”+length);
buff.appendValueOrDash(” Req-Header-Length:”+String.valueOf(length));
}
}
2. / Go to domain bin folder
/x/bea-wl1213/user_projects/domains/frontoffice/bin
3. Create a script with the name below.
setUserOverrides.sh
4. Set the PRE_CLASSPATH variable to the location of the extended logger JAR.
export PRE_CLASSPATH=/x/bea-wl1213/user_projects/domains/frontoffice/el.jar
el.jar is the extended logger jar.
5. Go to Home >Summary of Servers > servername > Logging > HTTP ( Advanced) and change the Format to extended and provide the format fields.
6. Restart the JVM and check the access log of the JVM.
7. Check Entry in logs
JSESSIONID=8OTYD-98RbnwDmChUGx61TW20AfC_DbjA6775PT7YewummUh0YYw!-1884316704 Req-Header-Length=415 2019-08-28 04:50:45 0.008 GET /prsysmgmt/js/expcollapse.js 200
Reference
ORA-01555: snapshot too old: rollback segment number with name “” too small
ORA-01555: snapshot too old: rollback segment number with name “” too small
ORA-22924: snapshot too old
DatabaseException caused by prior exception: java.sql.SQLException: ORA-01555: snapshot too old: rollback segment number with name “” too small
ORA-22924: snapshot too old
| SQL Code: 1555 | SQL State: 72000
The LOB column for this table had an Very less Retention set for fetching undo blocks,modified accordingly.
04:34:25 SQL> select PARTITIONED,SECUREFILE,RETENTION_TYPE,RETENTION_VALUE,PCTVERSION,RETENTION from dba_lobs where TABLE_NAME=’PR_DATA’ and owner=’PRPCDBA’;
PAR SEC RETENTI RETENTION_VALUE PCTVERSION RETENTION
NO NO YES 1800
04:34:57 SQL> alter table PRPCDBA.PR_DATA modify lob(PZPVSTREAM) (pctversion 5);
Table altered.
04:36:17 SQL> alter table PRPCDBA.PR_DATA modify lob(PZPVSTREAM) (retention);
Table altered.
04:36:35 SQL> select PARTITIONED,SECUREFILE,RETENTION_TYPE,RETENTION_VALUE,PCTVERSION,RETENTION from dba_lobs where TABLE_NAME=’PR_DATA’ and owner=’PRPCDBA’;
PAR SEC RETENTI RETENTION_VALUE PCTVERSION RETENTION
NO NO YES 50400
Oracle WebLogic zero-day discovered in the wild
Several application running Oracle Weblogic were detected by Nessus as having a vulnerable version (being exploited in the wild).
Impacted Versions: WebLogic 10.X, WebLogic 12.1.3
Resolution/ Workaround:
Oracle is working on the fix, below is the work around for the time being.
To prevent attacks, KnownSec 404 is recommending that companies either remove the vulnerable components and
restart their WebLogic servers, or put firewall rules in place to prevent requests being made to two URL
paths exploited by the attacks ( /_async/* and /wls-wsat/*).
Remove below 2 wars form installation and bounce the JVM’s with server tmp clean.
bea-wl1213/oracle_common/modules/com.oracle.webservices.wls.bea-wls9-async-response_12.1.3.war
bea-wl1213/oracle_common/modules/com.oracle.webservices.wls.wsat-endpoints-impl_12.1.3.war
Update – Oracle has released the Patches
10.3.6.0
Jan PSU 10.3.6.0.190115 Patch 28710912 + Overlay Patch 29694149 on 10.3.6.0.190115 for CVE-2019-2725
Apr PSU 10.3.6.0.190416 Patch 29204678 + Overlay Patch 29694149 on 10.3.6.0.190416 for CVE-2019-2725
12.1.3.0
Jan 2019 PSU 12.1.3.0.190115 Patch 28710923 + Overlay Patch 29694149 on 12.1.3.0.190115 for CVE-2019-2725
Apr 2019 PSU 12.1.3.0.190416 Patch 29204657 + Overlay Patch 29694149 on 12.1.3.0.190416 for CVE-2019-2725
INSTALLING OHS 12C BINARIES IN SILENT MODE + CREATING STAND ALONE INSTANCE USING PYTHON SCRIPT
ENVIRONMENT: LINUX Red Hat Enterprise Linux Server release 7.5 (Maipo)
WILL BE INSTALLING OHS 12.2.1.3.0
Make sure you take care of all the OS level prerequisite before moving forward with OHS installation.
Below screen shot is providing listing of each folder that will be use in this silent install.
software folder only contain ohs 12.2.1.3.0 installation binaries file
scripts folder contain the script which will start installer. Name of the script is weblogic_12.2.1.3.0_binaries_installation_script.sh
Below is the content of weblogic_12.2.1.3.0_binaries_installation_script.sh
[oracle@host01 stage]$ cat /scripts/weblogic_12.2.1.3.0_binaries_installation_script.sh
export JAVA_HOME=/u01/app/jdk1.8.0_171
/software/fmw_12.2.1.3.0_ohs_linux64.bin -silent -responseFile /stage/ohs.rsp -invPtrLoc /stage/oraInst.loc
stage folder contains two files oracle inventory location file that is oraInst.loc and ohs response file ohs.rsp as shown in below screen shot. I have posted the content of ohs.rsp below so you can copy and paste. OraInst.loc contain only two lines so you can simply type. Make sure you have correct ORACLE_HOME set in your response file. My is /u01/domains/OHS_Web
Response File Version=12.2.1.3.0
[GENERIC]
ORACLE_HOME=/u01/domains/OHS_Web
INSTALL_TYPE=Standalone HTTP Server (Managed independently of WebLogic server)
DECLINE_SECURITY_UPDATES=true
MYORACLESUPPORT_UNSERNAME=
MYORACLESUPPORT_PASSWORD=
DECLINE_SECURITY_UPDATES=true
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
PROXY_HOST=
PROXY_PORT=
PROXY_USER=
PROXY_PWD=
COLLECTOR_SUPPORTHUB_URL=
Now we have everything align to start the process to initiate installation of ohs binaries.
Let’s move forward with executing
/scripts/weblogic_12.2.1.3.0_binaries_installation_script.sh
As you can see in blow output that installation processor has stated and finish with successful message. That’s it… IN NEXT POST I WILL SHOW HOW TO CREATE A STANDALONE OHS 12C INSTANCE USING PYTHON SCRIPT AND WLST
[oracle@host01 scripts]$ sh weblogic_12.2.1.3.0_binaries_installation_script.sh
Launcher log file is /tmp/OraInstall2018-10-01_02-37-10PM/launcher2018-10-01_02-37-10PM.log.
Checking if CPU speed is above 300 MHz. Actual 2601.000 MHz Passed
Checking swap space: must be greater than 512 MB. Actual 3071 MB Passed
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)
Checking temp space: must be greater than 300 MB. Actual 6383 MB Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-10-01_02-37-10PM
Log: /tmp/OraInstall2018-10-01_02-37-10PM/install2018-10-01_02-37-10PM.log
Copyright (c) 1999, 2017, Oracle and/or its affiliates. All rights reserved.
Reading response file..
Skipping Software Updates
Starting check : CertifiedVersions
Expected result: One of redhat-7, redhat-6, oracle-6, oracle-7, SuSE-11, SuSE-12
Actual Result: redhat-7.5
Check complete. The overall result of this check is: Passed
CertifiedVersions Check: Success.
Starting check : Packages
Checking for binutils-2.23.52.0.1; found binutils-2.27-28.base.el7_5.1-x86_64. Passed
Checking for compat-libcap1-1.10; found compat-libcap1-1.10-7.el7-x86_64. Passed
Checking for compat-libstdc++-33-3.2.3-x86_64; found compat-libstdc++-33-3.2.3-72.el7-x86_64. Passed
Checking for libgcc-4.8.2-x86_64; found libgcc-4.8.5-28.el7_5.1-x86_64. Passed
Checking for libstdc++-4.8.2-x86_64; found libstdc++-4.8.5-28.el7_5.1-x86_64. Passed
Checking for libstdc++-devel-4.8.2-x86_64; found libstdc++-devel-4.8.5-28.el7_5.1-x86_64. Passed
Checking for sysstat-10.1.5; found sysstat-10.1.5-13.el7-x86_64. Passed
Checking for gcc-4.8.2; found gcc-4.8.5-28.el7_5.1-x86_64. Passed
Checking for gcc-c++-4.8.2; found gcc-c++-4.8.5-28.el7_5.1-x86_64. Passed
Checking for ksh-…; found ksh-20120801-137.el7-x86_64. Passed
Checking for make-3.82; found make-1:3.82-23.el7-x86_64. Passed
Checking for glibc-2.17-x86_64; found glibc-2.17-222.el7-x86_64. Passed
Checking for glibc-devel-2.17-x86_64; found glibc-devel-2.17-222.el7-x86_64. Passed
Checking for libaio-0.3.109-x86_64; found libaio-0.3.109-13.el7-x86_64. Passed
Checking for libaio-devel-0.3.109-x86_64; found libaio-devel-0.3.109-13.el7-x86_64. Passed
Check complete. The overall result of this check is: Passed
Packages Check: Success.
Starting check : Kernel
Checking for VERSION=3.10.0; found VERSION=3.10.0-862.9.1.el7.x86_64. Passed
Checking for hardnofiles=4096; found hardnofiles=4096. Passed
Checking for softnofiles=4096; found softnofiles=4096. Passed
Check complete. The overall result of this check is: Passed
Kernel Check: Success.
Starting check : TotalMemory
Expected result: 1024MB
Actual Result: 2784MB
Check complete. The overall result of this check is: Passed
TotalMemory Check: Success.
Starting check : CheckJDKVersion
Expected result: 1.8.0_131
Actual Result: 1.8.0_131
Check complete. The overall result of this check is: Passed
CheckJDKVersion Check: Success.
Validations are enabled for this session.
Verifying data
Copying Files
Percent Complete : 10
Percent Complete : 20
Percent Complete : 30
Percent Complete : 40
Percent Complete : 50
Percent Complete : 60
Percent Complete : 70
Percent Complete : 80
Percent Complete : 90
Percent Complete : 100
The installation of Oracle HTTP Server 12.2.1.3.0 completed successfully.
Logs successfully copied to /home/oracle/oraInventory/logs.
IN SECOND PART OF THIS DOCUMENT I WILL SHOW YOU PYTHON SCRIPT TO CREATE A OHS STAND ALONE INSTANCE. THIS INSTANCE WILL BE CREATED USING DEFAULT TEMPLATE THAT COMES WITH ORACLE BINARIES INSTALLATION. THE TEMPLATE NAME IS ohs_standalone_template.jar and will be place under $ORACLE_HOME/ohs/common/templates/wls/ohs/ohs_standalone_template.jar Below is the content of my python script which i named create_ohs_stand_alone.12.2.1.3.0.py
# Directories
domainName =’ohs1′
domainDir = ‘/u01/OHS_WEB1’
# Read OHS Standalone template
readTemplate(‘/u01/domains/OHS_Web/ohs/common/templates/wls/ohs_standalone_template.jar’)
# Configure Nodemanager
cd(‘/’)
create(domainName, ‘SecurityConfiguration’)
cd(‘SecurityConfiguration/’ + domainName)
set(‘NodeManagerUsername’,’weblogic’)
set(‘NodeManagerPasswordEncrypted’,’Welcome1′)
setOption(‘NodeManagerType’, ‘PerDomainNodeManager’)
cd(‘/’)
create(‘localmachine’, ‘UnixMachine’)
cd(‘UnixMachine/localmachine’)
create(‘localmachine’, ‘NodeManager’)
cd(‘NodeManager/localmachine’)
set(‘ListenAddress’, ‘localhost’)
cd(‘/OHS/ohs1’)
cmo.setAdminHost(‘localhost’)
cmo.setAdminPort(‘7001’)
cmo.setListenAddress(‘localhost’)
cmo.setListenPort(‘7777’)
cmo.setSSLListenPort(‘4443’)
writeDomain(domainDir)
closeTemplate()
NOW RUN THIS python script using wlst.sh script. Here is how you will run it. Navigate to $ORACLE_HOME/oracle_common/common/bin/wlst.sh and run the command as shown in below syntax, please note i have place my python script under /stage/create_ohs2.12.1.3.standalone.py
/u01/domains/OHS_Web/oracle_common/common/bin/wlst.sh /stage/create_ohs2.12.1.3.standalone.py
Once the script is complete you will see a folder by then name of OHS_WEB1 under /u01/
that’s All.
How to create Work Manager and assign to application.
Please follow the steps below to create a WORK MANAGER
Environment -> Work Manager -> New
Configure Maxmimum and Minimum Thread Contraint
Target them on the server.
Once the work manager is created, an application can be configured to use it.
weblogic-ejb-jar.xml
<weblogic-enterprise-bean>
<ejb-name>PRAsync</ejb-name>
<message-driven-descriptor>
<pool>
<max-beans-in-free-pool>1000</max-beans-in-free-pool>
<initial-beans-in-free-pool>50</initial-beans-in-free-pool>
</pool>
<destination-jndi-name>jms/PRAsyncTopic</destination-jndi-name>
</message-driven-descriptor>
<reference-descriptor>
<ejb-local-reference-description>
<ejb-ref-name>ejb/EngineBMTLocal</ejb-ref-name>
<jndi-name>ejb/PREngineBMTLocal</jndi-name>
</ejb-local-reference-description>
<ejb-local-reference-description>
<ejb-ref-name>ejb/EngineCMTLocal</ejb-ref-name>
<jndi-name>ejb/PREngineCMTLocal</jndi-name>
</ejb-local-reference-description>
</reference-descriptor>
<enable-call-by-reference>True</enable-call-by-reference>
<jndi-name>ejb/PRAsync</jndi-name>
<dispatch-policy>PegaWorkManager</dispatch-policy>
</weblogic-enterprise-bean>
weblogic.xml
<?xml version="1.0" encoding="UTF-8"?> <weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app"> <dispatch-policy>PegaWorkManager</dispatch-policy> </weblogic-web-app>
Steps to generate boot.properties and startup.properties for AdminServer
If we start the AdminServer( which has never been started before using script ) from NodeManager, boot.properties and startup.properties file don’t get created by default and AdminServer fails to start.
To create the boot.properties and startup.properties file we need to follow the steps below.
sed -i “s/StartScriptEnabled=true/StartScriptEnabled=false/g” nodemanager.properties
connect(‘user’,’pwd!’,’t3://host:port’)
nmGenBootStartupProps(‘AdminServer’)
STEPS OF INSTALLING OHS 12.2.1.3.0 ON REDHAT LINUX 7 64 BIT
- DOWNLOAD OHS 12.2.1.3.0 INSTALLATION FILE FROM ORACLE WEBSITE. INSTALLATION FILE EITHER WILL BE IN a .jar FILE OR IT WILL BE A ZIP FILE. THE ONE I DOWNLOAD IS A ZIP FILE.
- AFTER DOWNLOADING THE ZIP FILE, EXTRACT THE ZIP FILE WITH FOLLOWING COMMAND unzip fmw_12.2.1.3.0_ohs_linux64_Disk1_1.zip I have also show this in below screen shot. ONCE THE FILE FINISHING UNZIPPING IT EXTRACT A FILE WITH THE NAME OF fmw_12.2.1.3.0_ohs_linux64.bin .
NOW LET’S START INSTALLATION
Make sure you have Xwindows session configure.
Started the installation by executing fmw_12.2.1.3.0_ohs_linux64.bin file as shown below. I will take default and click ok to continue. If you want different location for inventory then you can specify here base on your directory structure.
click Next here
We will take default here and click on Next
provide Oracle Home here based on your directory structure. In my case it will be /u01/OHS_HOME. Next
Here I will take default. I want to run it independently without weblogic. Click Next
Here i have to specify location of my java path where java is install. -Click Next
Next screen is running prerequisite check in witch oracle software goes through and check if all the prerequisite has been configure for successful installation. In my case Prerequisite Check failed with following errors shown in screenshot. I am getting this error because i am missing few libraries on this OS that is require for this installation.
so now first thing first we will have to install these packages using yum repository to make this installation successful. To do this part you have to be register with Redhat. I am running community development red hat linux free source. Here is the link where you can download free community software and register with them to access yum repositories . https://developers.redhat.com/products/rhel/hello-world/#fndtn-windows No scree shot provided
I installed the failed libraries using yum install command. Here is the final looks you should see after installing all failed libraries. Notice no red x sign. click Next to proceed.
Click Install
Installation started. This will take approximately 30 minutes. Once done click Finish
Once binaries installation part completed. Now we create ohs instance using config.sh script. cd into $ORACLE_HOME/oracle_common/common/bin folder run script config.sh
i will create domain under /u01/OHS click Next
select same option as shown. oracle http server standalone and click Next
Java path looks correct, so i will click Next here without any changes.
ohs1 will be our first ohs http server instance. Click Next
provide IP address or host name for admin Host and Listen port and Click Next
create node manager username and password. click Next
click Create
completed click Next and Finish ohs1 instance is created now next we will start the instance now.
Step to start OHS
- First start Nodemanager using script startNodemanager.sh under /u01/OHS/bin
- start ohs1 instance using startComponents.sh script under /u01/OHS when prompted for password provide password that was created earlier. After enter password you will see below screen that ohs1 instance is running now
To access. http://coherence1:7777
That’s it.