Weblogic Archive

Issue while creating datasource on Weblogic Server

In Weblogic Sometimes when you try add datasource to cluster group it fails with following error, which might be due to process limit on database side.

WLS Console Error
==============

Caused by: java.lang.Throwable: Substituted for the exception oracle.net.ns.NetException which lacks a String contructor, original message – Got minus one from a read call
at oracle.net.ns.Packet.receive(Packet.java:314)
at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:160)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:264)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection0(ConnectionEnvFactory.java:286)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.access$000(ConnectionEnvFactory.java:20)
at weblogic.jdbc.common.internal.ConnectionEnvFactory$1.run(ConnectionEnvFactory.java:215)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:212)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.setConnection(ConnectionEnvFactory.java:143)
at weblogic.jdbc.common.internal.JDBCResourceFactoryImpl.createResource(JDBCResourceFactoryImpl.java:185)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1356)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1272)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:240)
at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1754)
at weblogic.jdbc.common.internal.ConnectionPool.start(ConnectionPool.java:239)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:614)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:475)
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:344)

DB Error
==================

oracle@host02:/u01/app/db11g/product/11.2.0/dbhome_1/dbs >sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 6 08:21:02 2017

Copyright (c) 1982, 2011, Oracle. All rights reserved.

ERROR:
ORA-00020: maximum number of processes (150) exceeded
Validation Process
======================

List number of process running for the DB instance

oracle@host02:/practices/part1/practice12-02 >ps -auxf |grep orcl|wc -l
149
oracle@host02:/practices/part1/practice12-02

Check on DB level for limit
==========================
SQL> show parameter process scope=both;

NAME TYPE VALUE
———————————— ———– ——————————
aq_tm_processes integer 1
cell_offload_processing boolean TRUE
db_writer_processes integer 1
gcs_server_processes integer 0
global_txn_processes integer 1
job_queue_processes integer 1000
log_archive_max_processes integer 4
processes integer 150
processor_group_name string
SQL>

WorkAround
================

Increase the number of process on db level

SQL> alter system set processes=500 scope=spfile;

SQL> show parameter process;

NAME TYPE VALUE
———————————— ———– ——————————
aq_tm_processes integer 1
cell_offload_processing boolean TRUE
db_writer_processes integer 1
gcs_server_processes integer 0
global_txn_processes integer 1
job_queue_processes integer 1000
log_archive_max_processes integer 4
processes integer 500
processor_group_name string
SQL>

Now try to re-enable the datasource on weblogic

MOS Article Reference
=================
“IO Error:Got minus one from a read call”: In the Diagnostic logs (Doc ID 1995125.1)

What is Kerberos?

Three parties are involved in Kerberos Based Authentication – Client, Server and a Kerberos Distribution Centre.

The diagram below clearly demonstrates how the interactions between the three parties happen.

1 – Client requests for a TGT (Ticket to Get Tickets) from the KDC (Key Distribution Centre). Client sends its username in plain text format to get the TGT.

2 – KDC returns a TGT to the client. The TGT has the Session Key (SK) which is encrypted with a Key derived from the Client’s password. Hence only the Client will be able to retrieve the Session Key. This Session Key is used for all furthur communications between the Client and the KDC.

3 – The Client requests for a Service Ticket (ST) to the KDC by providing the Service name along with the Session Key for its own identification. The KDC will be able to provide the ST only if the Service is already registered with it.

4 – The KDC creates the Service Ticket. The Service Ticket has the Client’s Authentication Data and a Sub Sesion Key (SSK). The Service Ticket is then encrypted with a Key Derived from the Server’s Key which is shared with the KDC. This ensures that the Service Ticket can only be decrypted by the Server. The KDC then authors a message containing the Service Ticket and a Sub Session Key. The whole message is then encrypted with the Session Key so that only the intended Client can decrypt it.

Pictorially the Message looks like this.

5 – The Client decrypts the Message with the Session Key, retrieves the Service Ticket and the Sub Session Key. It sends the Service Ticket to the Server. This Service Ticket is the SPNEGO Token. The Server decrypts the Service Ticket with its Key and authenticates the Client based on the Clients Authentication Data. Also it gets a copy of a Sub Session Key from the Service Ticket. Now both the Client and the Server have a common key, Sub Session Key, which they use for all further communication.

6 – A session is established and no further authentication is required.

References :-

http://technet.microsoft.com/en-us/library/bb742516.aspx

Silent Installation of WebLogic Server

The below post describes about the Silent Installation of Oracle WebLogic Server 11g.

Silent mode installation allows you to define an installation configuration file called silent.xml, based upon which the required components are installed. This does not provide any configuration options during the installation process. Silent installation allows using the configuration to duplicate the installation on many machines. Silent-mode installation works on both Windows and UNIX systems.

The silent-mode installation process has two primary steps:

1. Create a silent.xml file that defines the installation configuration that you would normally enter during an interactive installation process, such as graphical-mode or console-mode installation.

For example, in silent.xml, you can supply the values for the Middleware home directory, the product directory, and the components to be installed.

A sample silent.xml looks like below.

**************************************
<?xml version="1.0" encoding="UTF-8"?>
<bea-installer>
<input-fields>
<data-value value="D:\WLS11g"/>
<data-value name="WLS_INSTALL_DIR" value="D:\WLS11g\wlserver_10.3"/>
<data-value name="COMPONENT_PATHS"
value="WebLogic Server/Core Application Server|WebLogic Server/Administration Console|WebLogic Server/Configuration Wizard and Upgrade Framework|WebLogic Server/Web 2.0 HTTP Pub-Sub Server|WebLogic Server/WebLogic JDBC Drivers|WebLogic Server/Third Party JDBC Drivers|WebLogic Server/WebLogic Server Clients|WebLogic Server/WebLogic Web Server Plugins|WebLogic Server/UDDI and Xquery Support|WebLogic Server/Server Examples"/>
<!--<data-value name="INSTALL_NODE_MANAGER_SERVICE" value="yes"  />
<data-value name="NODEMGR_PORT" value="5559" />
<data-value name="INSTALL_SHORTCUT_IN_ALL_USERS_FOLDER" value="yes"/>
<data-valuemso-spacerun:yes">       "value="D:\jrockit_160_05|D:\jdk160_11"/>
-->
</input-fields>
</bea-installer>
**************************************

NOTE: Make sure there are no empty spaces with in the COMPONENT_PATHS values.

2. Open a command prompt and start the installation by executing the below command.

wls1031_win32.exe -mode=silent -silent_xml=<path_to_silent.xml>

NOTE: Optionally you can specify an installation log file so that you can keep a track of any issues that occur during installation of the WebLogic server as below.

wls1031_win32.exe -mode=silent -silent_xml=<path_to_silent.xml> -log=g:\silent-install.log

It opens up a new windows showing the installation progress.

3. Check the install progress window and a successful install would display message like below.

In the verbose install log, a successful install would read like below.

[WizardController] com.bea.plateng.wizard.silent.tasks.LogTask – The installation was successfull!

Note: A typical install in the windows would look like.
Note: If you are using a generic jar installer then you need to set the PATH before you run the installer.

java -jar file_name.jar -mode=silent -silent_xml=path_to_silent.xml

file_name.jar is the name of the installation file
path_to_silent.xml is the full path to the silent.xml file.

There are other ways of silent installation of the weblogic server according to your situation, refer the below links to have more insight into them.

Common errors during installation:

1:

An external Eclipse Home must be specified when using the generic installer.

Troubleshooting:

You can specify the Eclipse External Home by using the below argument in the silent.xml

USE_EXTERNAL_ECLIPSE=”true”

EXTERNAL_ECLIPSE_DIR=<Eclipse-Location-Directory>

2:

The directory “/tmp” does not have enough space for extraction.

Troubleshooting:

Silent-mode installation requires the same amount of temporary disk space and uses the same temporary storage directories as a standard installation. If you re running out of temporary space, you can change the default temporary directory by using the below parameter.

Specify the below java handler.

-Djava.io.tmpdir=<temp-dir-location>:
References:
Cheers,
Wonders Team. 🙂

Not able to configure Error Page?

Customers have often reported issues that they are not able to configure error page even though the do the configuration in the web.xml correctly

web.xml
<?xml version=’1.0′ encoding=’UTF-8′?>
<web-app xmlns=”http://java.sun.com/xml/ns/j2ee” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”>
<display-name>TestApplication</display-name>
<welcome-file-list>
<welcome-file>index.jsp</welcome-file>
</welcome-file-list>
<error-page>
<error-code>404</error-code>
<location>/error.jsp</location>
</error-page>
<error-page>
<error-code>500</error-code>
<location>/error.jsp</location>
</error-page>
<error-page>
<error-code>403</error-code>
<location>/error.jsp</location>
</error-page>
</web-app>

error.jsp

<%@page isErrorPage=”true” contentType=”text/html” %>
<html>
<body>
Request that failed: ${pageContext.errorData.requestURI}
<br />
Status code: ${pageContext.errorData.statusCode}
<br />
Exception: ${pageContext.errorData.throwable}
<br />
${pageContext.errorData.servletName}
</body>
</html>
index.jsp
<HTML>
<HEAD>
<TITLE>Test</TITLE>
</HEAD>
<BODY>
<%
String s = null;
out.println(s.trim());
%>
</BODY>
</HTML>

When I deployed this application on Weblogic Server and tried to acces the application. I got null pointer exception causing server to send HTTP 500 Response
with Exception

java.lang.NullPointerException
at jsp_servlet.__index._jspService(__index.java:76)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run
(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecuri
tyHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.jav
a:292)
Truncated. see log file for complete stacktrace

But since we had done the error page configuration, we expected to see our custom error page. This was not the case.
We saw the following. ( Note I was using IE 7)



I had to do the following configuration change on IE to get it working. ( Uncheck Show Friendly HTTP Error Messages)



After changing the IE Setting, I was able to get the custom error page.


Reference :-

http://www.java2s.com

Enabling and Disabling Weblogic Admin Console

Disabling and Enabling Admin Console

Many at times for most of our administration work including the changes (deployments, start/stop of servers, etc) or configurations (JMS, creation/deletion/editing of our servers, etc) we use our weblogic admin console.

But, for security reasons some of the banking companies for its core banking applications prefer to disable the admin console in its banking applications.

This short and sweet article mainly targets to present you on how to enable and disable your admin console:

Disabling your Admin Console:

We can disable our weblogic admin console in two different ways

1)      Admin console

2)      Weblogic Scripting Tool

From Admin console:-

To disable access to the Administration Console:

  1. After you log in to admin console click Lock & Edit.
  2. In the left pane of the Console, under Domain Structure, select the domain name.
  3. Select Configuration > General, and click Advanced at the bottom of the page.
  4. Deselect Console Enabled.
  5. Click Save.
  6. To activate these changes, click Activate Changes.

From WLST:-

connect(“weblogic“,”weblogic“,”t3://localhost:7001“)

edit()

startEdit()

cmo.setConsoleEnabled(false)

save()

activate()

disconnect()

exit()

Enabling the Admin Console:

After we disable the admin console we can enable it again by using WLST.

Following are the steps on the same:

connect(“weblogic“,”weblogic“,”t3://localhost:7001“)

edit()

startEdit()

cmo.setConsoleEnabled(true)

save()

activate()

disconnect()

exit()

Note: Here,

1)      weblogic and weblogic are the user id and password of admin console.

2)      t3://localhost:7001 is the admin console URL

3)      After we enable/disable the admin console RESTART your admin server

Different Out Of Memory Issues

* Exception in thread “CompilerThread1” java.lang.OutOfMemoryError: requested 793020 bytes for Chunk::new. Out of swap space?

Out of memory while reading in symbol table of /apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl
( 0)  0xc8461230     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 1)  0xc80a5fec     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 2)  0xc7f00420     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 3)  0xc7f00ca0     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 4)  0xc8368d08     [/apps/bea/weblogic92/jdk1.5.0.18/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 5)  0xc005b2e4   __pthread_body + 0x44  [/usr/lib/libpthread.1]
( 6)  0xc0065574   __pthread_start + 0x14  [/usr/lib/libpthread.1]
Java out of memory messages are marked with pid: 13828 in /var/adm/syslog/syslog.log.

Possible causes:
– not enough swap space left, or
– kernel parameter MAXDSIZ is very small.

Solution:
Although it appears that an OutOfMemoryError is thrown this apparent exception is reported by the HotSpot VM code when an allocation from the native heap failed and the native heap may be close to exhaustion. The message indicates the size (in bytes) of the request that failed and also indicates what the memory is required for. In some cases the reason will be shown but in most cases the reason will be the name of a source module reporting the allocation failure. If an OutOfMemoryError with this error is thrown it may require using utilities on the operating system to diagnose the issue further. Examples of issues that may not be related to the application are when the operating system is configured with insufficient swap space, or when there is another process on the system that is consuming all memory resources. If neither of these issues is the cause then it is possible that the application is failed due to native leak; for example, application or library code is continuously allocating memory but is not releasing it to the operating system.
For more information: http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf

The recommendation for swap space size in the Solaris is that swap should be configured about 30% of physical RAM.

The following link has suggested a workaround to add ‘-XX:+UseDefaultStackSize -Xss256K’ parameter.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4916142

Add: -XX:CodeCacheMinimumFreeSpace=2M -XX:+ReservedCodeCacheSize=64M -XX:PermSize=128m -XX:MaxPermSize=384m (As per your other JVM settings)

Sun team needs to be involved for this issue. Usually such issues are solved by sun support.

=============================

* java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at weblogic.work.RequestManager.createThreadAndExecute(RequestManager.java:271)
at weblogic.work.RequestManager.executeIt(RequestManager.java:245)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)

Solution:
1) Set kernel parameter maxdsiz to a higher value
2) Reduce the current heap size.
3) Check the kernel values:  ulimit -a
4) If the NPROC soft limit is lower than the hard limit, increase it as needed: ulimit -u <new value>.
Check the Operating System documentation to make changes permanent at the OS configuration files.
5) Need to reduce the JVM stack size and the OS stack size both. Set the -xss in java options and set ulimit -s on the OS level.

=============================

* java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
at java.lang.Class.getConstructor0(Unknown Source)
at java.lang.Class.getConstructor(Unknown Source)

Solution: Increase the max permgen space -XX:MaxPermSize=256m

There can be a leak in the permgen objects. If tuning parameters do not resolve the issue, we need to use the memory leak detector tools and find out which instances in the permgen space are not getting cleared.

=============================

* java.lang.OutOfMemoryError: allocLargeObjectOrArray – Object size: 372032, Num elements: 372012
* java.lang.outofmemoryerror: nativeGetNewTLA

Solution: -XXtlasize:128k -XXlargeobjectlimit:128k

If this does not solve the issue, we need to check in the application code for the large objects being created and not being destroyed. Take JRA Recording (Oracle JRockit) or use JConsole and memory leak detector tools (JMAP, JHAT) for analysis on the

=============================

* java.lang.OutOfMemoryError: Java Heap Space

Solution: First thing that needs to be checked is the gc logs. Need to check whether the garbage collection is happening properly. If the heap keeps gradually increasing even after full gc, tune the gc algorithms and check if the behavior is the same. Use memory leak detector tools for both sun jdk and JRockit to check which instances from the application are not getting destroyed.

If this is not the case and the application genuinely needs more memory, increase the heap size by using the parameters:
Example: -Xms2048m -Xmx2048m

=============================

* java.lang.StackOverflowError

Solution: Stack over flow error is usually generated due to a recursive call made by the application (infinite recursion), or its because of an attempt to allocate more memory on the stack than will fit. This is usually the result of creating local array variables that are far too large for the current stack.

For the first possibility, we need to check the application code as to where is the recursive call being made.
For the second possibility, we can increase the JVM stack size by the parameter : Example: -Xss512K

=============================

java.lang.OutOfMemoryError: GC overhead limit exceeded.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209) 

Try adding this jvm option
-XX:-UseGCOverheadLimit