CLSRSC-180: An error occurred while executing the command ‘/bin/rpm -qf /sbin/init’

Hello,

I recently encountered following error during Exadata GI home upgrade from 12.1.0.1 to 12.2.0.1, We encountered this error during the execution of rootupgrade.sh script on node 1 itself.

2018/06/10 02:59:18 CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init' 
Died at /u01/app/12.2.0.1/grid/crs/install/s_crsutils.pm line 2372. 
The command '/u01/app/12.2.0.1/grid/perl/bin/perl -I/u01/app/12.2.0.1/grid/perl/lib -I/u01/app/12.2.0.1/grid/crs/install /u01/app/12.2.0.1/grid/crs/install/rootcrs.pl -upgrade' execution failed 

I thought of furthur investigating this issue by running target command manually and i got following error. These errors were also logged in installation logfile.

[root@dm01dbadm01 ~]# /bin/rpm -qf /sbin/init 
rpmdb: Thread/process 261710/140405403039488 failed: Thread died in Berkeley DB library 
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery 
error: cannot open Packages index using db3 - (-30974) 
error: cannot open Packages database in /var/lib/rpm 
rpmdb: Thread/process 261710/140405403039488 failed: Thread died in Berkeley DB library 
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery 
error: cannot open Packages database in /var/lib/rpm 
rpmdb: Thread/process 261710/140405403039488 failed: Thread died in Berkeley DB library 
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery 
error: cannot open Packages database in /var/lib/rpm 
file /sbin/init is not owned by any package 
You have new mail in /var/spool/mail/root 

 

Issue was related corruption of OS level database RPM. We can validate this issue by running following command.

# /bin/rpm -qa | more

 

We had to fix RPM corruption issue by using following and so we can continue our Exadata upgrade.

As root OS user run the following: 
# rm -f /var/lib/rpm/__* 
# /bin/rpm --rebuilddb 
# echo $?

 

After rebuilding corrupted RPMs , using following command to validate them.

# /bin/rpm -qa | more