环境
OS:AIX 6105
Weblogic:9.2MP3
JDK:IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423-20091104 (JIT enabled)
3 问题描述
1） admin server 挂起
admin server启动后，只要有被管服务启动，admin server就会挂起，管理控制台无法登录。
2）kill －9 sid后admin server进程消失。
3）启动管理服务，出现如下错误
<2011-1-26 下午06时54分10秒 CST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with IBM J9 VM Version.3 from IBM Corporation>
<2011-1-26 下午06时54分11秒 CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:75)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:374)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:125)
at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:630)
at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:402)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:361)
at weblogic.Server.main(Server.java:67)
>
<2011-1-26 下午06时54分11秒 CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<2011-1-26 下午06时54分11秒 CST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut
itself down>
<2011-1-26 下午06时54分11秒 CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
4）删除3中指定的文件AdminServer.lok后，重新启动admin server，出现如下错误：
<2011-1-26 下午06时57分38秒 CST> <Notice> <Log Management> <BEA-170019> <The server log file /home/weblogic/bea/user_pro
jects/domains/prpcar_8001/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written
to this file.>
<2011-1-26 下午06时57分39秒 CST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory
: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds a
nd then retrying in case existing WebLogic Server is still shutting down.>
<2011-1-26 下午06时57分49秒 CST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory
: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds a
nd then retrying in case existing WebLogic Server is still shutting down.>
<2011-1-26 下午06时57分59秒 CST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory
: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds a
nd then retrying in case existing WebLogic Server is still shutting down.>
<2011-1-26 下午06时58分09秒 CST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory
: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds a
nd then retrying in case existing WebLogic Server is still shutting down.>
<2011-1-26 下午06时58分19秒 CST> <Error> <EmbeddedLDAP> <BEA-171519> <Could not obtain an exclusive lock to the embedded
LDAP data files directory: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/AdminServer/data/ldap/ldapfiles
because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely sh
utdown and restart the server.>
<2011-1-26 下午06时58分19秒 CST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: Could not obtain an ex
clusive lock to the embedded LDAP data files directory: /home/weblogic/bea/user_projects/domains/prpcar_8001/servers/Adm
inServer/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebL
ogic Server is completely shutdown and restart the server.>
<2011-1-26 下午06时58分19秒 CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<2011-1-26 下午06时58分19秒 CST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut
itself down>
<2011-1-26 下午06时58分19秒 CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
5）重命名ldpfile文件夹后，重启admin server，错误提示为端口别其他进程占用
6） netstat -an看，端口7001被占用
$ netstat -an grep 7001
tcp 0 0 172.16.28.152.7001 *.* LISTEN
tcp4 0 0 172.16.28.152.7001 172.16.28.152.32824 CLOSE_WAIT
tcp4 0 0 172.16.28.152.7001 172.16.28.152.32825 CLOSE_WAIT
tcp4 0 0 172.16.28.152.7001 172.16.28.152.32946 CLOSE_WAIT
tcp4 1187 0 172.16.28.152.7001 172.16.28.152.32975 CLOSE_WAIT
tcp4 1184 0 172.16.28.152.7001 172.16.28.152.32976 CLOSE_WAIT
tcp4 1197 0 172.16.28.152.7001 172.16.28.152.32977 CLOSE_WAIT
tcp4 1213 0 172.16.28.152.7001 172.16.28.152.32978 CLOSE_WAIT
tcp4 1181 0 172.16.28.152.7001 172.16.28.152.32987 CLOSE_WAIT
tcp4 26 0 172.16.28.152.7001 172.16.28.152.33018 CLOSE_WAIT
tcp4 1198 0 172.16.28.152.7001 172.16.28.152.33034 CLOSE_WAIT
tcp4 1214 0 172.16.28.152.7001 172.16.28.152.33035 CLOSE_WAIT
tcp4 1182 0 172.16.28.152.7001 172.16.28.152.33036 CLOSE_WAIT
tcp 0 0 127.0.0.1.7001 *.* LISTEN
tcp4 1016 0 172.16.28.152.7001 172.16.21.143.1409 ESTABLISHED
tcp4 1146 0 172.16.28.152.7001 172.16.21.143.2312 ESTABLISHED
7）重启服务器后，能正常启动adminserver ，其他服务器上有被管服务而且能访问，不久admin server又被挂起。
8）admin server 挂起后，想查看admin server的进程信息，执行kill -3 sid，并没有生成thread dump，生成信息如下：
<2011-1-26 下午07时29分53秒 CST> <Warning> <netuix> <BEA-423420> <Redirect is executed in begin or refresh action. Redir
ect url is /console/console.portal?_nfpb=true&_pageLabel=AppDeploymentsControlPage.>
<2011-1-26 下午07时30分03秒 CST> <Warning> <netuix> <BEA-423420> <Redirect is executed in begin or refresh action. Redir
ect url is /console/console.portal?_nfpb=true&_pageLabel=AppDeploymentsControlPage.>
JVMDUMP006I Processing dump event "user", detail "" - please wait.
JVMDUMP032I JVM requested Java dump using '/home/weblogic/bea/user_projects/domains/prpcar_8001/javacore.20110126.193417
.8060944.0001.txt' in response to an event
JVMDUMP010I Java dump written to /home/weblogic/bea/user_projects/domains/prpcar_8001/javacore.20110126.193417.8060944.0
001.txt
JVMDUMP013I Processed dump event "user", detail "".
4 解决办法
从metalink上下载补丁GNLT，上传到weblogic服务器，打上补丁后，系统正常。操作步骤如下：
1）把补丁文件8173442_92mp3.jar上传到应用服务器“/home/weblogic/wlspk/”下。
2）修改文件WEBLOGIC_HOME/common/bin/commEnv.sh（当前环境下路径为：/home/weblogic/bea/weblogic92/common/bin/commEnv.sh）
在WEBLOGIC_CLASSPATH的值中，把8173442_92mp3.jar文件路径放在第一位。即把WEBLOGIC_CLASSPATH="${PATCH_CLASSPATH}${CLASSPATHSEP}${JAVA_HOME}/lib/tools.jar${CLASSPATHSEP}${WL_HOME}/server/lib/weblogic_sp.jar${CLASSPATHSEP}${WL_HOME}/server/lib/weblogic.jar${CLASSPATHSEP}${WL_HOME}/server/lib/webservices.jar"
修改为：
WEBLOGIC_CLASSPATH=/home/weblogic/wlspk/8173442_92mp3.jar:"${PATCH_CLASSPATH}${CLASSPATHSEP}${JAVA_HOME}/lib/tools.jar${CLASSPATHSEP}${WL_HOME}/server/lib/weblogic_sp.jar${CLASSPATHSEP}${WL_HOME}/server/lib/weblogic.jar${CLASSPATHSEP}${WL_HOME}/server/lib/webservices.jar"
3）停掉所有的weblogic服务，然后启动。
4）在启动日志中发现类似如下信息，即可表明补丁已经打上。
####<2011-1-27 下午04时09分37秒 CST> <Info> <Management> <> <> <main> <> <> <> <1296115777473> <BEA-141107> <Version: WebLogic Server Temporary Patch for 8173442, 8177658 Wed Aug 12 17:04:38 PDT 2009
5 备注
当前环境下：如果重建domain或者新建一个domain，weblogic不需要重新打补丁，如果把weblogic卸载后重装，则需要重新打同样的补丁
当前位置：
论坛首页 »
自由讨论区 »
Tomcat, WebLogic及J2EE讨论区
»
发起投票
技术讨论
加好友
发消息
2011-10-17 15:52:32 |
赞(
操作
顶一个～
