1. 问题描述

本次现场主要处理上周和昨天出现的XXX商城weblogic server异常退出的故障。

2. 应用环境

中间件： WebLogic 10.3.3.0

操作系统： HP Unix B.11.31

JDK ： HP JDK 1.6.0.05

JVM启动参数：

/opt/java6/bin/IA64N/java -server -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+HeapDump -XX:+HeapDumpOnCtrlBreak -XX:+HeapDumpOnOutOfMemoryError -Xverbosegc:file=./log/jmetric.vgc -Dweblogic.Name=EBiz-Server-A01 -Dweblogic.management.username= -Dweblogic.management.password= -Dweblogic.management.server=t3://10.4.46.220:7001 -Djava.security.policy=/ecapp/soft/wlserver_10.3/server/lib/weblogic.policy weblogic.Server

3. 问题分析

Server进程异常退出后，重启后正常。查看问题出现前后的日志信息找到如下信息：

####<2012-7-11 上午11时28分35秒 GMT+08:00> <Info> <Common> <wecapp1> <EBiz-Server-A01> <[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1341977315557> <BEA-000627> <Reached maximum capacity of pool "EBiz Data Source", making "0" new resource instances instead of "1".>
####<2012-7-11 上午11时28分35秒 GMT+08:00> <Info> <Common> <wecapp1> <EBiz-Server-A01> <[ACTIVE] ExecuteThread: '202' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1341977315558> <BEA-000627> <Reached maximum capacity of pool "EBiz Data Source", making "0" new resource instances instead of "1".>

Server宕机前出现了大量的线程创建数据库连接不成功的info信息，当时线程池内的线程数有200左右，大概半数左右都在创建数据库连接，原因是数据库连接池内的连接数已经达到最大值，不能创建出新的连接供给线程使用。

####<2012-7-11 上午11时28分35秒 GMT+08:00> <Error> <HTTP> <wecapp1> <EBiz-Server-A01> <[ACTIVE] ExecuteThread: '41' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<WLS Kernel>> <> <> <1341977315542> <BEA-101020> <[ServletContext@27611459[app:app-web-portal-pro-A01 module:app-web-portal-pro-A01.war path: spec-version:2.5]]
Servlet failed with Exception
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Internal error: Cannot obtain
XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool EBiz Data Source to allocate to applications, please
increase the size of the pool and retry..
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:555)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:322)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:317)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:61)
at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1670)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1468)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:439)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:355)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:190)
at org.springframework.orm.ibatis.SqlMapClientTemplate.executeWithListResult(SqlMapClientTemplate.java:249)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:296)
at com.asiainfo.ebiz.common.dao.ibatis.SqlMapSysParameterDAO.querySysParametersByType(SqlMapSysParameterDAO.java:30)
at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy123.querySysParametersByType(Unknown Source)

由于不能创建出新的数据库连接导致sql异常。

####<2012-7-11 上午11时28分36秒 GMT+08:00> <Critical> <WorkManager> <wecapp1> <EBiz-Server-A01> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default
(self-tuning)'> <<WLS Kernel>> <> <> <1341977316568> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to java.lang.OutOfMemoryError:
unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at weblogic.work.RequestManager.createThreadAndExecute(RequestManager.java:272)
at weblogic.work.RequestManager.executeIt(RequestManager.java:245)
at weblogic.work.SelfTuningWorkManagerImpl.scheduleInternal(SelfTuningWorkManagerImpl.java:166)
at weblogic.work.SelfTuningWorkManagerImpl.schedule(SelfTuningWorkManagerImpl.java:149)
at weblogic.deploy.service.internal.RequestManager.serviceNextRequest(RequestManager.java:60)
at weblogic.deploy.service.internal.RequestManager.addRequest(RequestManager.java:40)
at weblogic.deploy.service.internal.transport.CommonMessageReceiver.receiveHeartbeatMsg(CommonMessageReceiver.java:625)
at weblogic.deploy.service.internal.transport.CommonMessageReceiver.dispatch(CommonMessageReceiver.java:751)
at weblogic.deploy.service.internal.transport.http.HTTPMessageReceiver.receiveMessage(HTTPMessageReceiver.java:31)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.handleDeploymentServiceMessage(DeploymentServiceServlet.java:670)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.internalDoPost(DeploymentServiceServlet.java:235)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.access$000(DeploymentServiceServlet.java:77)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet$1.run(DeploymentServiceServlet.java:210)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.doPost(DeploymentServiceServlet.java:207)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3686)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>

上面的错误是server进程异常退出的原因，由于本地内存溢出，不能创建出新的线程，导致JVM异常退出。由于在出现问题时没有及时做thread dump查看线程状态，但从日志中记录的信息可以看出当时的200多个线程大多处于等待创建新的数据库连接的状态，而当更多的新请求过来，需要线程处理时，已经没有新线程处理，同时JVM不能创建出更多的线程导致JVM异常退出。

4. 问题建议

4.1. 增加数据库连接池的最大容量

保证有足够的数据库连接可用。同时注意抓取数据库信息，确认是否有死锁等状况导致连接数被占用。同时，日志中记录了，当server重启后立刻就要创建新的数据库连接，说明设置的初始值过小，从性能优先的角度建议将初始值调大到100，步增长调整为10。

4.2. 避免JVM异常退出

为保证当连接数即使不够的情况下，也不创建出更多的线程导致本地内存溢出从而JVM崩溃Server异常退出，可用采用以下方法：

1) 调整线程栈大小，JDK1.6默认每个线程占用本地内存大小512K，为了使JDK能够创建出更多的线程，添加-Xss256K参数调整到每个线程占用本地内存256K，调整后参数：-Xms2048m -Xmx2048m –Xss256K -XX:MaxPermSize=512m

2) 限制线程池的最大个数

BEA官方建议主机1颗CPU处理的线程数最好不要超过50，根据经验，一个server的线程数控制在100-150效果最佳，同时为了保证JVM不被新增请求创建出更多的线程导致阻塞而异常退出，建议限制线程池内线程数最小值为150，最大值150

在config.xml中添加如下参数：(示例中修改的是AdminServer线程的个数)

<server>
<name>AdminServer</name>
<self-tuning-thread-pool-size-min>150</self-tuning-thread-pool-size-min>
<self-tuning-thread-pool-size-max>150</self-tuning-thread-pool-size-max>
<listen-port>7923</listen-port>
<listen-address></listen-address>
</server>

该贴被funny编辑于2014-3-4 14:45:41