Troubleshooting SSL issues

Signature verification failed because RSA key public exponent [3] is too small

RSA Keys with Public Exponent results in faulty signature verification on WLS. Having so low exponent is considered as security vulnerability; hence keys with low exponents are not supported by WLS. However if we need to bypass this behavior, we can use the following flag

-Dweblogic.security.SSL.allowSmallRSAExponent=true

java.security.InvalidKeyException: Illegal key size or default parameters

This exception is encountered while using strong encryption such as AES256. We can overcome this by downloading the unrestricted jurisdiction policy files from the JVM vendor site and place it under jre/lib/security folder.

NEW ALERT with Severity: FATAL, Type: 70

We get this alert when the the party communication with Weblogic Server is using a different version of SSL. We need to check the Handshake Message for the version of SSL used.

Using this flag to specify the version of SSL at WLS can be helpful.

-Dweblogic.security.SSL.protocolVersion=SSL3

NEW ALERT=with Severity: FATAL, Type: 42

This alert means that the certificate presented to WLS is not trusted. It can be resolved by importing the certificate into the trust store of Weblogic Server.

HANDSHAKE_FAILURE alert received from localhost – 127.0.0.1.

Most of the time its because of HOST NAME VERIFICATION.
Ignore Host Name Verification by setting this flag for Admin & Managed Server
-Dweblogic.security.SSL.ignoreHostnameVerification=true

And this in the startNodeManager.cmd
-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false

Sometime when the root certificate does not meet the basic constraint, i.e. even when the issuer and the owner is the same, the criticality is not true

ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:0
]

To allow WLS to accept such certificates we need to pass on this flag

-Dweblogic.security.SSL.enforceConstraints=off

java.security.cert.CertificateParsingException: PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11

The root problem is the Certicom SSL does not support SHA256 algorithm, which is required with the trusted certificates of “ttelesecglobalrootclass2ca” and “ttelesecglobalrootclass3ca”

A fix is included in JDK 1.6.0_13 wherein WLS just ignores these certificates.

Trust failure (68): CERT_CHAIN_INCOMPLETE

We encounter this issue when the Weblogic Server is not able to verify the chain of certificates presented to it. From the debug message we can check the certificates and check their order in the chain. We can also check the trust store for the root and intermediate certificates on the signing authority of the certificates.
We can use this to validate the certificate chain using

java utils.ValidateCertChain -jks alias storefilename [storePass]

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

We need to specify the trustore as a JAVA OPTION
-Djavax.net.ssl.trustStore=samplecacerts
Or specify it as a System Property in the code
System.setProperty(“javax.net.ssl.trustStore”,”samplecacerts”);
System.setProperty(“javax.net.ssl.trustStorePassword”,”changeit”);

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Pass the keystore in the java options.

-Dssl.debug=true -Dweblogic.security.TrustKeyStore=CustomTrust -Dweblogic.security.CustomTrustKeyStoreFileName=faisal_trust.jks  -Djavax.net.ssl.keyStore=faisal.jks -Djavax.net.ssl.keyStorePassword=password -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Djava.protocol.handler.pkgs=weblogic.net

java.security.InvalidKeyException: Illegal key size

Try adding the following jvm option. This will make Weblogic Server FIPS 140-2 compliant.

-Dweblogic.security.SSL.nojce=true

javax.net.ssl.SSLException: Inbound closed before receiving peer’s close_notify: possible truncation attack?

This issue is fixed in 12.1.2
For 1035 and 1036 apply patch for BUG 13351178.

weblogic.wsee.jaxrpc.soapfault.WLSOAPFaultException: Failed to receive message javax.net.ssl.SSLKeyException: FATAL Alert:BAD_CERTIFICATE – A corrupt or unuseable certificate was received.

Add the following JVM Options to the server
-DUseSunHttpHandler=true
-Dssl.SocketFactory.provider=sun.security.ssl.SSLSocketFactoryImpl
-Dssl.ServerSocketFactory.provider=sun.security.ssl.SSLSocketFactoryImpl

Upgrade the client to use the same version of Java 7 as the webservices.
Ensure that both the client and the webservices were using unlimited strength encryption.

javax.net.ssl.SSLProtocolException: handshake alert:unrecognized_name

The issue is because of introduction of Server Name Indication in JAVA SE 7 update 2.
It can be disabled with the following flag.
-Djsse.enableSNIExtension=false

To resolve it make sure your webserver has virtual hostname set correctly.

javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big” ?

This looks to a jdk isssue, following jdk bug matches the description.
http://bugs.java.com/view_bug.do?bug_id=JDK-8149017

java.lang.RuntimeException: javax.net.ssl.SSLException: Received fatal alert: unexpected_message

This is due to a jdk1.7 bug. It can be remediated by applying the flag below.

-Dcom.sun.net.ssl.enableECC=false

javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

This happens when the application running on JDK still uses TLS1.0 As a temporary work around we can enable TLS protocol by commenting the line below in java.security file

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024,

EC keySize < 224, 3DES_EDE_CBC, anon, NULL