在一个的测试数据库上,创建表空间时出现了这个错误。
由于数据库环境比较复杂,简单描述一下。
这个测试环境安装的是Oracle 1106 for Solaris 10 sparc 64bit的RAC环境,搭建了ASM实例用于存放共享数据文件。
在RAC环境的其中一个节点上,又建立了一个单实例的数据库,并把这个数据库的数据文件也放到了ASM实例上。
结果尝试在这个实例上添加新的表空间时报错:
SQL> select file_name from dba_data_files;
FILE_NAME
------------------------------------------------------------------------------
+DATA/test/datafile/system.533.668281219
+DATA/test/datafile/sysaux.534.668281227
+DATA/test/datafile/undotbs1.535.668281229
+DATA/test/datafile/users.537.668281241
SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;
create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m
*第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/test/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/test/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.
这个错误似乎很少见,查看了一下Oracle的官方错误文档描述:
ORA-00569: Failed to acquire global enqueue.
Cause: A prior error occurred on one of the instances in the cluster. Typically errors are caused by shared pool resource contention.
Action: Check for and resolve prior errors on all instances in the cluster. If there is shared pool resource contention, increase the SHARED_POOL_SIZE, DML_ LOCKS, PROCESSES, TRANSACTIONS, CLUSTER_DATABASE_INSTANCES and PARALLEL_MAX_SERVERS initialization parameters.
虽然对问题进行了描述,不过从错误看不出导致问题的真正原因。
查询了一下METALINK,找到了一些错误说明,不过没有和当前错误相似度很高的,大部分出现这个错误的同时,都会伴随ORA-600错误和ORA-4031错误。
不过现在有一个简单的方法来确定到底是数据库产生的问题还是ASM实例导致的问题,现在只需要登陆RAC实例,执行类似的添加表空间的操作,检查是否会出现相同的问题就可以了:
bash-3.00$ export ORACLE_SID=ractest1
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:12:42 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
已连接到空闲例程。
SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1603887104 bytes
Fixed Size 2095208 bytes
Variable Size 741722008 bytes
Database Buffers 855638016 bytes
Redo Buffers 4431872 bytes数据库装载完毕。数据库已经打开。
SQL> CREATE TABLESPACE TEST DATAFILE '+DATA/ractest/datafile/test01.dbf' SIZE 4096M;
CREATE TABLESPACE TEST DATAFILE '+DATA/ractest/datafile/test01.dbf' SIZE 4096M
*第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/ractest/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/ractest/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.
可以看到,相同的错误产生了,看来问题可能和ASM实例的状态有关系,登陆ASM实例,进行简单的检查:
bash-3.00$ export ORACLE_SID=+ASM1
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:33:12 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> set pages 100 lines 120
SQL> select instance_name, status from v$instance;
INSTANCE_NAME STATUS
-------------------------------- ------------------------
+ASM1 STARTED
由于ASM实例可以用来检查的动态视图太少,从现有的视图也看不到特别的地方,看来只能重启数据库和ASM实例,再次检查问题:
bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:41:40 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> shutdown immediate数据库已经关闭。已经卸载数据库。
ORACLE 例程已经关闭。
SQL> exit
bash-3.00$ export ORACLE_SID=ractest1
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:43:21 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> shutdown immediate数据库已经关闭。已经卸载数据库。
ORACLE 例程已经关闭。
SQL> exit从 Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options 断开
bash-3.00$ export ORACLE_SID=+ASM1
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:44:05 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> shutdown immediate
^CORA-01013: user requested cancel of current operation
SQL> CONN / AS SYSDBA已连接。
SQL> shutdown abort
ASM 实例已关闭
SQL> startup
ASM 实例已启动
Total System Global Area 284008448 bytes
Fixed Size 2087944 bytes
Variable Size 256754680 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期三 2月 18 17:47:22 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
已连接到空闲例程。
SQL> startup
ORACLE 例程已经启动。
Total System Global Area 806133760 bytes
Fixed Size 2099064 bytes
Variable Size 455669896 bytes
Database Buffers 343932928 bytes
Redo Buffers 4431872 bytes数据库装载完毕。数据库已经打开。
SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;
create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m
*第 1 行出现错误:
ORA-01119: 创建数据库文件 '+DATA/test/datafile/test01.dbf' 时出错
ORA-17502: ksfdcre: 4 未能创建文件 +DATA/test/datafile/test01.dbf
ORA-00569: Failed to acquire global enqueue.
ORA-00569: Failed to acquire global enqueue.
可以看到,重启ASM实例,问题仍然出现。不过ASM实例也是在两个节点上同时运行的,莫非是另一个节点的ASM实例出现了问题:
bash-3.00$ export ORACLE_SID=+ASM2
bash-3.00$ sqlplus "/ as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期四 2月 19 16:38:38 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> set pages 100 lines 120
SQL> select instance_name, status from v$instance;
INSTANCE_NAME STATUS
-------------------------------- ------------------------
+ASM2 STARTED
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
检查ASM实例未发现异常,尝试重启ASM实例:
bash-3.00$ srvctl stop instance -d ractest -i ractest2
bash-3.00$ srvctl stop asm -n ser2
bash-3.00$ srvctl start asm -n ser2
再次登陆test数据库,执行CREATE TABLESPACE语句:
bash-3.00$ export ORACLE_SID=test
bash-3.00$ sqlplus "/as sysdba"
SQL*Plus: Release 11.1.0.6.0 - Production on 星期四 2月 19 16:41:09 2009
Copyright (c) 1982, 2007, Oracle. All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> set pages 100 lines 120
SQL> create tablespace test datafile '+DATA/test/datafile/test01.dbf' size 4096m;
表空间已创建。
看来问题果然和ASM实例状态不正常有关。
检查asm实例2的alert文件,发现在运行CREATE TABLESPACE语句对应的时间点,出现了ORA-4031错误:
Wed Feb 18 15:50:04 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2412):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2412/+ASM2_lmd0_3099_i2412.trc
Wed Feb 18 15:50:05 2009
Trace dumping is performing id=[cdmp_20090218155005]
WARNING: ran out of shared pool for GES enqueue object.
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2413):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2413/+ASM2_lmd0_3099_i2413.trc
Trace dumping is performing id=[cdmp_20090218155013]
Wed Feb 18 17:17:23 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2414):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2414/+ASM2_lmd0_3099_i2414.trc
WARNING: ran out of shared pool for GES enqueue object.
Wed Feb 18 17:17:24 2009
Trace dumping is performing id=[cdmp_20090218171724]
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2415):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2415/+ASM2_lmd0_3099_i2415.trc
Trace dumping is performing id=[cdmp_20090218171732]
Wed Feb 18 17:18:02 2009
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2416):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2416/+ASM2_lmd0_3099_i2416.trc
WARNING: ran out of shared pool for GES enqueue object.
Wed Feb 18 17:18:03 2009
Trace dumping is performing id=[cdmp_20090218171803]
Errors in file /data/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmd0_3099.trc (incident=2417):
ORA-04031: unable to allocate 3512 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","ges enqueues")
Incident details in: /data/oracle/diag/asm/+asm/+ASM2/incident/incdir_2417/+ASM2_lmd0_3099_i2417.trc
Trace dumping is performing id=[cdmp_20090218171811]
这个ORA-4031错误已经和Oracle的metalink文章相符了,和Oracle错误文档上对这个错误的描述也是一致的。
而且这个ORA-4031错误信息也很明显,在分配全局对了资源的时候出现的错误。
检查ASM实例的sga,发现:
SQL> show sga
Total System Global Area 284008448 bytes
Fixed Size 2087944 bytes
Variable Size 256754680 bytes
ASM Cache 25165824 bytes
对于跑了多个RAC环境的ASM实例而言,200M的SGA显然太小了,和大部分Oracle默认参数一样,默认的ASM实例参数也是偏小的。
以前也碰到过一次由于ASM实例PROCESS参数太小,导致ASM实例无法登陆的问题。因此,如果选择ASM作为产品库的存储方式,那么ASM实例要重新设置,默认的参数很可能无法满足需要。