SSL Exceptions in Admin Server and Node Manager.

javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from oracle.test.com – xx.xxx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

 

<WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake>javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:849) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170)

The above exceptions are  the most common exceptions encountered during the setup of Weblogic Server in an environment. The stack does suggest what could be the reasons but the diagnostics are not mentioned.

To debug this issue, first we need to check the certificates used by Admin Server and the Node Manager. If we have Admin and the Node Manager using demo certificates, then the issue can be due to improper DNS mapping. We can use the nslookup to check the DNS entry. For testing purpose we can provide the ip address as the listen address for the admin server and the node manager and see if the issue is still occurring.

Also we will have to turn of host name verification and the basic validation check of the certificates. We can do it by specifying the following flag in startWeblogic.sh

-Dssl.debug=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.enforceConstraints=off

And the following flag in startNodeManager.sh

-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

If the Admin Server is using Custom Identity and Custom trust, then its better to configure the node manger with custom identity and custom trust as well.
By default the Node Manager is configured with Demo Identity and Demo Trust. To change it to custom identity and custom trust, we need to specify the following values in the nodemanager.properties file present in nodemanager home

Keystores=CustomIdentityandCustomTrust
CustomIdentityAlias=
CustomIdentityKeyStoreFileName=
CustomIdentityKeyStorePassPhrase = xxxxxx
CustomIdentityKeyStoreType = JKS
CustomIdentityPrivateKeyPassPhrase = xxxxxxx

Apply the same flags as above in the startup script of Admin Server and Node Manger.

Check from the console whether Node Manager is reachable or not.

Another option can be to use PLAIN communication between Admin Server and Node Manager.

We can change the Listen Type to PLAIN for the Node Manager from the console and set the secureListener=false in the nodemanager.properties file present in nodemanager home.

References:-

http://download.oracle.com/docs/cd/E15051_01/wls/docs103/nodemgr/nodemgr_config.html#wp1101097

19 comments

  1. Hi Faisal,

    Interesting and useful article about common SSL exceptions. It's crystal clear and easy to understand.
    I have just one observation, please correct the following error replacing "Keystores=CustomIdentityandCustomTrust" by "KeyStores=CustomIdentityAndCustomTrust" otherwise the NodeManager will load default DemoIdentity.jks keystore because java properties are case-sensitive.

    Thank you.
    Keep up the good work!

    Virgil Frum

  2. How to stop This message …what is the reason …pls help.

    javax.net.ssl.SSLProtocolException: [Security:090493]BAD_RECORD_MAC alert received from 24.1.2.79 – 24.1.2.79. The peer indicated it received a record with a
    n invalid MAC.
    javax.net.ssl.SSLProtocolException: [Security:090493]BAD_RECORD_MAC alert received from 24.1.2.79 – 24.1.2.79. The peer indicated it received a record with an
    invalid MAC.

    1. Hi Suresh,

      BAD_RECORD_MAC alerts are usually received when the record is received with an incorrect MAC Address.
      You may need to verify the mac address of the parties involved.

      It can also be an issue with the difference in the SSL Implementation. As a test u can use the same JDK version and JDK vendor for both the parties.

      What is running on 24.1.2.79?

      -Faisal

  3. Hi Faisal,

    I am facing the exact same error, when i use WLST to connect to nodemanager i am getting the below error:-

    javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from pdes.server.com . Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

    can u tel me if my nodemanager is running as windows service where to add

    -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

    can u specify the script and the line…..

    Appreciate if u could help

    1. Yes sure, connect to WLS from WLST using the command line below

      java -Dssl.debug=true -Dweblogic.security.TrustKeyStore=DemoTrust -Dweblogic.security.SSL.ignoreHostnameVerification=true weblogic.WLST

      Also if you need to pass the following java option to the nodemanager running as a Windows Service, you need to first un install the service (using uninstallNodeMgrSvc.cmd present at G:\bea103\wlserver_10.3\server)

      Then add the following paramaters as JAVA_OPTIONS

      -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

      in G:\bea103\wlserver_10.3\server\bin\installNodeMgrSvc.cmd

      Let me know if you have any doubts.

      Cheers!!
      Faisal

  4. Hi Faisal,

    I have tried

    java -Dssl.debug=true -Dweblogic.security.TrustKeyStore=DemoTrust -Dweblogic.security.SSL.ignoreHostnameVerification=true weblogic.WLST to connect and then when i use a nmConnect i am getting the error.

    Nodemanager logs says below:-

    [Security:090482]BAD_CERTIFICATE alert was received from AUSrui.aus.amer.sper.com – 10.15.4.16. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>
    javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from AUSrui.aus.amer.sper.com -. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertReceived(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handle(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)
    at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
    at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
    at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
    at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
    at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
    at com.certicom.tls.record.ReadHandler.read(Unknown Source)
    at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
    at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.BufferedReader.readLine(BufferedReader.java:362)
    at weblogic.nodemanager.server.Handler.run(Handler.java:66)
    at java.lang.Thread.run(Thread.java:595)

    I added the -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off in script installNodeMgrSvc.cmd in the below line and installed it again.

    set CMDLINE=%JAVA_VM% %MEM_ARGS% -classpath \”%CLASSPATH%\” -Djava.security.policy=\”%WL_HOME%\server\lib\weblogic.policy\” -Dweblogic.nodemanager.javaHome=\”%JAVA_HOME%\” -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
    -Dweblogic.security.SSL.enforceConstraints=off

    Am i adding it in the correct place as i couldnt see any java options in this.?? i also added it in comenv script as well.

    Please let me know your views..

  5. Thank You, Thank You!!! And also to Virgil. I had two properties spelled incorrectly (lower-case “s”) – thanks to an error in a Oracle document. but I finally have it working. Phew!

  6. Hi,
    I encountered the same error and cannot solve it yet. Please advise me what to do.

    My error message in nodemanager is:

    javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from 192.168.100.16 – 192.168.100.16. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertReceived(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handle(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)
    at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
    at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
    at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
    at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
    at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
    at com.certicom.tls.record.ReadHandler.read(Unknown Source)
    at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.BufferedReader.readLine(BufferedReader.java:362)
    at weblogic.nodemanager.server.Handler.run(Handler.java:71)
    at java.lang.Thread.run(Thread.java:662)

    Thanks.

  7. Hi Faisal,

    I am using custom identity and custom trust in my weblogic server, both are loading and I saw the below messag in admin server log file.
    <Demo trusted CA certificate is being used in production mode:

    I had added these parameters as well for Admin server. -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
    -Dweblogic.security.SSL.enforceConstraints=off

    Also added three parameters for the nodemanager which are mentioed in the above article. Still I saw the below exception is listing in node manager log file.
    Sep 13, 2011 7:38:17 AM weblogic.nodemanager.server.Handler run
    WARNING: Uncaught exception in server handlerjavax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from – . Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

    Please help on this. Thanks.

    Rajendra

  8. I m newbie to WebLogic, and I have the same problem

    javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from 192.168.100.16 – 192.168.100.16. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
    at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertReceived(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handle(Unknown Source)
    at com.certicom.tls.record.alert.AlertHandler.handleAlertMessages(Unknown Source)

    its one week that I m try to solve this problem, please , can you help me please ??? Thanks

  9. When run the ManagedServer in the cmd prompt.
    statrManagedWebLogic.cmd osb_server1 http://localhost:7001.
    It throws error like the below one
    <Server s
    ubsystem failed. Reason: java.lang.ExceptionInInitializerError
    java.lang.ExceptionInInitializerError
    at weblogic.management.provider.internal.BeanInfoAccessService.start(Bea
    nInfoAccessService.java:30)
    at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesMan
    ager.java:461)
    at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServ
    icesManager.java:166)
    at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
    at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
    at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
    at weblogic.Server.main(Server.java:71)
    Caused By: java.lang.NullPointerException
    at weblogic.utils.FileUtils.find(FileUtils.java:334)
    at weblogic.descriptor.DescriptorClassLoader.findJars(DescriptorClassLoa
    der.java:80)
    at weblogic.descriptor.DescriptorClassLoader.getExtendedClassLoader(Desc
    riptorClassLoader.java:154)
    at weblogic.descriptor.DescriptorClassLoader.getClassLoader(DescriptorCl
    assLoader.java:41)
    at weblogic.management.provider.internal.BeanInfoAccessSingleton$SINGLET
    ON.(BeanInfoAccessSingleton.java:34)
    at weblogic.management.provider.internal.BeanInfoAccessService.start(Bea
    nInfoAccessService.java:30)
    at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesMan
    ager.java:461)
    at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServ
    icesManager.java:166)
    at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
    at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
    at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
    at weblogic.Server.main(Server.java:71)
    >

    Provide me solution for this

  10. Thanks for the marvelous posting! I quite enjoyed reading it, you happen to
    be a great author. I will remember to bookmark your blog and will often come back
    later on. I want to encourage you to ultimately continue your great
    posts, have a nice morning!

  11. hello Faisal,

    I did follow these steps…but the node manager is still using DemoIdentity.jks

    Keystores=CustomIdentityAndCustomTrust
    CustomIdentityAlias=TBTTSOIM1D_identity
    CustomIdentityKeyStoreFileName=c:\apps\middleware\wlserver_10.3\server\lib\identit.jks
    CustomIdentityKeyStorePassPhrase=password
    CustomIdentityKeyStoreType=JKS
    CustomIdentityPrivateKeyPassPhrase=password

    one thing I am not sure about is how the path needs to be listed for ” CustomIdentityKeyStoreFileName=”

    any help is appreciated.

    Thanks
    Nitin

Leave a Reply