Troubleshooting Archive

Unable to start managed server from nodemanager

When we try to start the managed server from nodemanager, we get the following error.

<BEA-090064> <The DeployableAuthorizer “myrealm_weblogic.security.providers.xacml.authorization.XACMLAuthorizationProviderImpl” returned an error: weblogic.security.spi.ResourceCreationException: [Security:090310]Failed to create resource.>

This errror comes because the managed server ldap server (slave) is not in sync with the Admin Server ldap server ( master).

To overcome this issue we can use two very important configurations.

 

Refresh Replica at Startup and Master First.

 

master first

 

Master First ensures managed server always connect to the ldap server of the admin server.

Refresh Replica at startup ensures that whenever managed server is restarted, it takes the latest ldap data from the admin server.

 

 

High CPU Usage of WebLogic Server on Linux

If you observe CPU being hogged by the Weblogic Server process on a linux machine,
you need to find out the Weblogic Server process id using

ps -ef | grep java

After you’ve found out the process id, find the thread id’s ( light weight process id’s) and their CPU usage using top command.

top -H -p 2389 (Weblogic Server Process Id)

HighCPU

2703 , 2704 & 2705 are the light weight process that are hogging up the CPU.

We take thread dump using kill -3 2389 (Weblogic Server Process Id)

We can map the light weight process to Weblogic Server threads by converting 2703 to hex 0xa8f.

We search for the thread (nid=0xa8f) in the thread dump to know what the thread is doing.
From the stack we can make out some complex mathematical operation is going on.
Inorder to avoid the High CPU we need to change the code to simplify the operation.


“Thread-19” daemon prio=10 tid=0x8a092000 nid=0xa8f runnable [0x887fe000]
java.lang.Thread.State: RUNNABLE
at java.lang.StrictMath.pow(Native Method)
at java.lang.Math.pow(Math.java:609)
at jsp_servlet.__index$1.run(__index.java:85)
at java.lang.Thread.run(Thread.java:662)

javax.xml.stream.XMLStreamException while validating xml

javax.xml.stream.XMLStreamException: ParseError at [row,col]:[2,134]
Message: Tried all: ‘1’ addresses, but could not connect over HTTP to server: ‘java.sun.com’, port: ’80’

at com.sun.xml.stream.XMLReaderImpl.next(XMLReaderImpl.java:545)
at weblogic.servlet.internal.TldCacheHelper$TldIOHelper.parseXML(TldCacheHelper.java:132)
at weblogic.descriptor.DescriptorCache.parseXML(DescriptorCache.java:380)
at weblogic.servlet.internal.TldCacheHelper.parseTagLibraries(TldCacheHelper.java:65)
at weblogic.servlet.internal.War.getTagInfo(War.java:889)
at weblogic.servlet.internal.WebAppServletContext$ServletContextWebAppHelper.getTldInfo(WebAppServletContext.java:3708)

You might encounter the following exception when the server on which Weblogic is hosted is not able to communicate to internet.The reason being that internet access is allowed only via proxy.
To allow access we need to pass on the proxy details as a JVM Option.

-Dhttp.proxyHost=< proxy host> -Dhttp.proxyPort=

This should resolve the issue.

You can test the connection by a simple jsp that makes a URL Connection

System.setProperty(“http.proxyHost”, “proxy host”);
System.setProperty(“http.proxyPort”, “proxy port”);
URL url = new URL(“http://java.sun.com:80”);

NodeManager issues during oracle weblogic server environment setup

I/O error while reading domain directory: java.io.FileNotFoundException: Domain directory ‘D:\bea\wlserver_10.3\common\nodemanager’ invalid (domain salt file not found

We need to to nmEnroll and restart the nodemanager to fix this error.


weblogic.nodemanager.common.ConfigException: Native version is enabled but node manager native library could not be loaded

You need to set the LD_LIBRARY_PATH for Solaris/Linux and SHLIB_PATH for HP-UX

Solaris/Linux

LD_LIBRARY_PATH:$WL_HOME/server/lib/solaris:$WL_HOME/server/lib/solaris/ocixxx_x
HP-UX
SHLIB_PATH=$SHLIB_PATH:$WL_HOME/server/lib/hpux11:$WL_HOME/server/lib/hpux11/ocixxx_x

Fatal error in node manager server
java.lang.NullPointerException
at weblogic.nodemanager.server.ServerManager.getStartCallbacks(ServerManager.java:187)
at weblogic.nodemanager.server.AbstractServerManager.initialize(AbstractServerManager.java:97)
at weblogic.nodemanager.server.AbstractServerManager.(AbstractServerManager.java:63)

This is a bug on WLS and there is a patch available for this.
Please contact Oracle Customer Support.

common database related issues with Weblogic Server

<BEA-001112> <Test “SELECT 1 FROM DUAL” set up for pool “Test” failed with exception: “java.sql.SQLException: Io exception: Connection reset by peer: socket write error”.>

Network or the DBMS is killing the connection, you need to check with the Database Admin and the network Admin.

 

java.lang.ClassCastException: oracle.jdbc.driver.LogicalConnection incompatible with oracle.jdbc.OracleCo
nnection

Check if you’ve got multiple versions of the driver in your environment, server\lib, domain\lib or added in the server classpath in the startWeblogic.cmd. Keep only one driver in your environment.

ORA-28267: Invalid NameSpace value

This is fixed by Patch 13923804 – 11.2.0.2.7 Patch Set Update (11G database patch)

weblogic.common.ResourceException: IO Error: Connection reset atweblogic.jdbc.common.internal.XAConnectionEnvFactory.makeConnection(XAConnectionEnvFactory.java:512)

Try to set the initial capacity of the datasource to zero.

Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource ‘Operational’: XAER_RMERR : A resource manager error has occured in the transaction branch

Issue can occur when you create tables using 10g client on 11g database. Recreate the tables with 11g client.

 

XAER_RMFAIL : Resource manager is unavailable

Keep your maximum and minimum connection as same.
Set this to true
Keep XA Connection Until Transaction Complete
Check with DBA on the number of process & sessions.


Error when getting datasource
weblogic.common.resourcepool.ResourceDeadException:
0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Locale not recognized

This is due to the locale configuration of the server.
You can start WLS with the following JVM Params and also make sure the client browser has the correct locale setting.

-Duser.language=
-Duser.country=
-Duser.timezone=

If it still doesnt solve the issue you can try to set the locale programmatically before getting connection.

Locale.setDefault(Locale.ENGLISH);
return DriverManager.getConnection(URL, username, password);

Caused by: java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAER_PROTO : Routine was invoked in an inproper context start() failed on resource ‘MSD_CONN_POOL’: XAER_PROTO : Routine was invoked in an inproper context
oracle.jdbc.xa.OracleXAException

This is a known issue and is fixed in 10.3.1
You can set Test Connection on Reserve to true.

java.sql.SQLException: Result set already closed

This is a coding isssue. Avoid calling a closed resultset.

EXCEPTION: (12021) Cannot obtain new connection.

Please make sure your ojdbc14.jar is there in the classpath.
weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool

Check application code if the connections, resultset and jdbc statements have been closed properly.

 

ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)'” RUNNABLE native
jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
java.net.SocketInputStream.socketRead0(SocketInputStream.java)
java.net.SocketInputStream.read(SocketInputStream.java:129)
weblogic.jdbc.sqlserver.SQLServerByteOrderedDataReader.makeMoreDataAvailable(Unknown Source)

This is a known issue with MS SQL Server and WLS 10.3 to 10.3.6. Please set the following connection properties.

SendStringParametersAsUnicode=false

 

 

 

Issues while starting Managed Server as a Window’s Service

[E] [ServiceStart] Error in JVM. Cause=Unable to find a JVM!

If we look at the complete log file we will observe the following

lpszCmdLine = -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -XX:+UseSpinning -Dweblogic.Stdout=C:\Oracle\Middleware\user_projects\domains\ClassicDomain\stdout.txt –

The service is starting Weblogic Server with -client option.

-server option should be used on a Windows 64 bit system.

We can either edit the registry and edit the CmdLine value data from -client to -server.
Or we can delete the existing service and create a new service again after modifying the installSvc.cmd file with
set JAVA_VM=-server or for JRockit set JAVA_VM=-jrockit.