Archive

Issue while creating datasource on Weblogic Server

In Weblogic Sometimes when you try add datasource to cluster group it fails with following error, which might be due to process limit on database side.

WLS Console Error
==============

Caused by: java.lang.Throwable: Substituted for the exception oracle.net.ns.NetException which lacks a String contructor, original message – Got minus one from a read call
at oracle.net.ns.Packet.receive(Packet.java:314)
at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:160)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:264)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection0(ConnectionEnvFactory.java:286)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.access$000(ConnectionEnvFactory.java:20)
at weblogic.jdbc.common.internal.ConnectionEnvFactory$1.run(ConnectionEnvFactory.java:215)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:212)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.setConnection(ConnectionEnvFactory.java:143)
at weblogic.jdbc.common.internal.JDBCResourceFactoryImpl.createResource(JDBCResourceFactoryImpl.java:185)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1356)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1272)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:240)
at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1754)
at weblogic.jdbc.common.internal.ConnectionPool.start(ConnectionPool.java:239)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:614)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:475)
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:344)

DB Error
==================

oracle@host02:/u01/app/db11g/product/11.2.0/dbhome_1/dbs >sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 6 08:21:02 2017

Copyright (c) 1982, 2011, Oracle. All rights reserved.

ERROR:
ORA-00020: maximum number of processes (150) exceeded
Validation Process
======================

List number of process running for the DB instance

oracle@host02:/practices/part1/practice12-02 >ps -auxf |grep orcl|wc -l
149
oracle@host02:/practices/part1/practice12-02

Check on DB level for limit
==========================
SQL> show parameter process scope=both;

NAME TYPE VALUE
———————————— ———– ——————————
aq_tm_processes integer 1
cell_offload_processing boolean TRUE
db_writer_processes integer 1
gcs_server_processes integer 0
global_txn_processes integer 1
job_queue_processes integer 1000
log_archive_max_processes integer 4
processes integer 150
processor_group_name string
SQL>

WorkAround
================

Increase the number of process on db level

SQL> alter system set processes=500 scope=spfile;

SQL> show parameter process;

NAME TYPE VALUE
———————————— ———– ——————————
aq_tm_processes integer 1
cell_offload_processing boolean TRUE
db_writer_processes integer 1
gcs_server_processes integer 0
global_txn_processes integer 1
job_queue_processes integer 1000
log_archive_max_processes integer 4
processes integer 500
processor_group_name string
SQL>

Now try to re-enable the datasource on weblogic

MOS Article Reference
=================
“IO Error:Got minus one from a read call”: In the Diagnostic logs (Doc ID 1995125.1)

Unable to Activate Changes in Enterprise Manager or WebLogic Console

This issue can be observed while updating a change from the console and saving and activating the changes or in this case from Enterprise manager console trying to edit a configuration and got the below error in the process and the same was seen in AdminServer logs too.

<Apr 4, 2017 9:19:34 AM PDT> <Warning> <DeploymentService> <BEA-290015> <Domain wide secret mismatch>

<Apr 4, 2017 9:20:34 AM PDT> <Warning> <DeploymentService> <BEA-290015> <Domain wide secret mismatch>

 

Failed-server: server2, Reason: java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading the input. : with response code ‘401’ : with response message ‘Unauthorized’

Edit-owner: principals=[weblogic, Administrators], Exclusive: false, Acquired: 1491322469714, Expire: 0

Possible Reason

1) Significant time delay between Admin and Managed Servers

Validation

1) Check the time on both admin servers, if there time difference is more then 2 mins / 120 sec then please proceed further.

# date —–→ Command Can be used to validate time on unix/linux systems

2) Check NTP configruations

# ntpq -pn —→ will list out the ntp servers if configured.

Sample

———

[root@host01 ~]# ntpq -pn

remote           refid      st t when poll reach   delay   offset  jitter

==============================================================================

10.232.144.1    .INIT.          16 u    – 1024    0    0.000    0.000   0.000

130.35.136.1    .INIT.          16 u    – 1024    0    0.000    0.000   0.000

*127.127.1.0     .LOCL.          10 l   18   64  377    0.000    0.000   0.000

3) Run a time check on ntp server to see if it is working, using ntpdate -dv command

# ntpdate -dv <ntp server ip>

Sample

——–

[root@host01 ~]# ntpdate -dv 10.232.144.1

4 Apr 15:12:08 ntpdate[1014]: ntpdate 4.2.4p8@1.1612-o Tue Jul  6 21:50:29 UTC 2010 (1)

Looking for host 10.232.144.1 and service ntp

host found : 10.232.144.1

transmit(10.232.144.1)

transmit(10.232.144.1)

transmit(10.232.144.1)

transmit(10.232.144.1)

transmit(10.232.144.1)

10.232.144.1: Server dropped: no data

server 10.232.144.1, port 123

stratum 0, precision 0, leap 00, trust 000

refid [10.232.144.1], delay 0.00000, dispersion 64.00000

transmitted 4, in filter 4

reference time:    00000000.00000000  Wed, Feb  6 2036 22:28:16.000

originate timestamp: 00000000.00000000  Wed, Feb  6 2036 22:28:16.000

transmit timestamp:  dc8e98c6.1b00c15f  Tue, Apr  4 2017 15:12:22.105

filter delay:  0.00000  0.00000  0.00000  0.00000

0.00000  0.00000  0.00000  0.00000

filter offset: 0.000000 0.000000 0.000000 0.000000

0.000000 0.000000 0.000000 0.000000

delay 0.00000, dispersion 64.00000

offset 0.000000

 

4 Apr 15:12:23 ntpdate[1014]: no server suitable for synchronization found

[root@host01 ~]#

 

In this case NTPServer isn’t responding so that’s the reason there was time delay between WLS servers

4) Use ntpdate -uv command to manually sync the server with active working ntp server.

# ntpdate -uv <ntpserver ip>

[root@host01 ~]# ntpdate -uv 192.0.2.1

4 Apr 09:40:37 ntpdate[31627]: ntpdate 4.2.4p8@1.1612-o Tue Jul  6 21:50:29 UTC 2010 (1)

4 Apr 09:42:34 ntpdate[31627]: step time server 192.0.2.1 offset 116.684819 sec

run the above command on all wls server to make sure time is sync, then run date command to verify.

5) Now try to update wls settings changes from admin console, it should work.

Oracle MOS References

Unable to Make Any Changes to Managed Server While It Is Running With Error “BEA-290015> Domain wide secret mismatch>”(Doc ID 2122342.1)

Unable To Lock/edit/Activate Changes from Weblogic Console with Response Message ‘Unauthorized’ (Doc ID 2240077.1)

The Activate Changes Operation for Request “xxxxxx” Could not be Completed Because the Activate Timed Out’ When Time On Managed Servers Is Out Of Sync (Doc ID 2107803.1)

How to force a NTP sync with the NTP server(s) on Oracle Linux or Oracle VM (Doc ID 2094959.1)