Hi Folks,
We have been facing a strange issue in SOA 11.1.1.4 Env.
The issue is DB Adapter sometimes failing to insert the data into One table.
I wanted to stress the word "strange" here, bcz this issue is not continuously happening -only sometimes.
Error Message :
Error: Exception occured when binding was invoked. Exception occured during invocation of JCA binding: "JCA Binding execute of Reference operation 'insert' failed due to: JCA Binding Component connection issue. JCA Binding Component is unable to create an outbound JCA (CCI) connection. The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/DB/ETCfrom JNDI and get a new Connection Handle. Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections. Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections. ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.
We have been struggled lot and finally figured out the root cause.
Reason:
There is a disconnection between WLS and JCA connection pool manager.
When we update JCA adapter deployment with new JNDI and redeploy the adapter instance then WLS invalidates all the existing connections and it seems WLS even doesn't know that JCA connection pool is the owner of these connections and it doesn't inform JCA pool manager that it has invalidated the connections.
Now JCA pool manager has no idea of what happened and it still maintains invalid connections in its pool.
Absolutely when client requests JCA connection then JCA manager try to return the dead connection which results above error.
Solution 1 :
1.Log on to your Admin console and navigate to Domain -> Deployments -> <adapter name> -> Configuration Tab -> Outbound Connection Pools Tab -> Under Groups and Instances expand the javax.resource.cci.ConnectionFactory
2.Locate and select the adapter JNDI and click on the Connection Pool Tab.
3.Please check to see if Shrinking is enabled. If not, please enable it.
4.Please check the value for Shrink Frequency Seconds. The default for some connection pools is as high as 900 seconds. Please reset this to 600 seconds or less.
5.Activate your changes.
We set connection shrink time to 600 sec that means 10 min.
What does "ShrinkFrequencySeconds" parameter mean?
"The number of seconds to wait before shrinking a connection pool that has incrementally increased to meet demand. ShrinkingEnabled must be set to true for a connection pool to shrink."
So, after 10 min, connections were returned to database. But just after that, probably a bunch of http requests came into play with the web servers. Most of the case, it needs to establish connection to the database to serve the request. Since these requests were more than the initial connection pool value which is 10 (back to the initial connection numbers after shrinking), jdbc establishes a bunch of new connections again to serve those requests. This is how we had a flood of new connection request after about 10 min on avg.
If this does not improve the situation, please also try increasing the max size of your connection pool.
Solution 2 :
Apply Oracle Patch "11812911"
Desc of Patch:-(11812911)
"SOLUTION
Solution 1
a. Download the patch 11812911 if it is available.
b. Apply the patch in a test environment.
c. Check if the issue is solved.
Solution 2
This solution updates the weblogic-ra.xml from the adapter without redeploying it
a. Modify the adapter from the weblogic administration console
b. Save the changes
c. Go to the Deployments page
d. Check the Adapter
e. Click on the Update button
f. Choose "Update this application in place with new deployment plan changes. (A deployment plan must be specified for this option)"
Do not use the default option: "Redeploy this application using the following deployment files"
g. Click finish
An adapter has the following structure:
AdapterName.rar
| - META-INF
| | - MANIFEST.MF
| | - weblogic-ra.xml
| | - ra.xml
| - AdapterName.jar
| - *.jar
Programming WebLogic Resource Adapters
weblogic-ra.xml Schema
- Table 9-13 default-connection-properties subelements
- Table 9-14 pool-params subelements "
I hope this should work.
Happy Learning .....!!!! Fun Sharing......!!!!!
We have been facing a strange issue in SOA 11.1.1.4 Env.
The issue is DB Adapter sometimes failing to insert the data into One table.
I wanted to stress the word "strange" here, bcz this issue is not continuously happening -only sometimes.
Error Message :
Error: Exception occured when binding was invoked. Exception occured during invocation of JCA binding: "JCA Binding execute of Reference operation 'insert' failed due to: JCA Binding Component connection issue. JCA Binding Component is unable to create an outbound JCA (CCI) connection. The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: javax.resource.spi.IllegalStateException: [Connector:199176]Unable to execute allocateConnection(...) on ConnectionManager. A stale Connection Factory or Connection Handle may be used. The connection pool associated with it has already been destroyed. Try to re-lookup Connection Factory eis/DB/ETCfrom JNDI and get a new Connection Handle. Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections. Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections. ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.
We have been struggled lot and finally figured out the root cause.
Reason:
There is a disconnection between WLS and JCA connection pool manager.
When we update JCA adapter deployment with new JNDI and redeploy the adapter instance then WLS invalidates all the existing connections and it seems WLS even doesn't know that JCA connection pool is the owner of these connections and it doesn't inform JCA pool manager that it has invalidated the connections.
Now JCA pool manager has no idea of what happened and it still maintains invalid connections in its pool.
Absolutely when client requests JCA connection then JCA manager try to return the dead connection which results above error.
Solution 1 :
1.Log on to your Admin console and navigate to Domain -> Deployments -> <adapter name> -> Configuration Tab -> Outbound Connection Pools Tab -> Under Groups and Instances expand the javax.resource.cci.ConnectionFactory
2.Locate and select the adapter JNDI and click on the Connection Pool Tab.
3.Please check to see if Shrinking is enabled. If not, please enable it.
4.Please check the value for Shrink Frequency Seconds. The default for some connection pools is as high as 900 seconds. Please reset this to 600 seconds or less.
5.Activate your changes.
We set connection shrink time to 600 sec that means 10 min.
What does "ShrinkFrequencySeconds" parameter mean?
"The number of seconds to wait before shrinking a connection pool that has incrementally increased to meet demand. ShrinkingEnabled must be set to true for a connection pool to shrink."
So, after 10 min, connections were returned to database. But just after that, probably a bunch of http requests came into play with the web servers. Most of the case, it needs to establish connection to the database to serve the request. Since these requests were more than the initial connection pool value which is 10 (back to the initial connection numbers after shrinking), jdbc establishes a bunch of new connections again to serve those requests. This is how we had a flood of new connection request after about 10 min on avg.
If this does not improve the situation, please also try increasing the max size of your connection pool.
Solution 2 :
Apply Oracle Patch "11812911"
Desc of Patch:-(11812911)
"SOLUTION
Solution 1
a. Download the patch 11812911 if it is available.
b. Apply the patch in a test environment.
c. Check if the issue is solved.
Solution 2
This solution updates the weblogic-ra.xml from the adapter without redeploying it
a. Modify the adapter from the weblogic administration console
b. Save the changes
c. Go to the Deployments page
d. Check the Adapter
e. Click on the Update button
f. Choose "Update this application in place with new deployment plan changes. (A deployment plan must be specified for this option)"
Do not use the default option: "Redeploy this application using the following deployment files"
g. Click finish
An adapter has the following structure:
AdapterName.rar
| - META-INF
| | - MANIFEST.MF
| | - weblogic-ra.xml
| | - ra.xml
| - AdapterName.jar
| - *.jar
Programming WebLogic Resource Adapters
weblogic-ra.xml Schema
- Table 9-13 default-connection-properties subelements
- Table 9-14 pool-params subelements "
I hope this should work.
Happy Learning .....!!!! Fun Sharing......!!!!!
wonderful information, I had come to know about your blog from my friend nandu , hyderaba.i have read atleast 7 posts of yours by now, and let me tell you,your website gives the best and the most interesting information.For more details about oracle fusion SCM training please check our website.
ReplyDeleteOracle Fusion SCM Online Training