VMCD.ORG » Update exadata flash disk firmware due to poor performance on X2-2

ASR XD报警抛出如下错误

"
dm01cel02: Hardware Error has been detected in the Flash Logs on the Cell Node
dm01cel02: Please open up a Service Request with Oracle to resolve the Error
dm01cel02: View File /tmp/ASR-Flash-Fault-Check-Log for further Information of the Error
dm01cel02: This log File can be copied to the Oracle Service Request for Error Reference"

by running sundiag we got these informations:

	 name:              	 dm01cel02_FLASHCACHE
	 cellDisk:          	 FD_11_dm01cel02,FD_09_dm01cel02,FD_13_dm01cel02,FD_14_dm01cel02,FD_10_dm01cel02,FD_03_dm01cel02,FD_07_dm01cel02,FD_06_dm01cel02,FD_01_dm01cel02,FD_15_dm01cel02,FD_04_dm01cel02,FD_08_dm01cel02,FD_02_dm01cel02,FD_12_dm01cel02,FD_05_dm01cel02
	 creationTime:      	 2012-06-26T17:41:47+08:00
	 degradedCelldisks: 	 FD_00_dm01cel02
	 effectiveCacheSize:	 341.953125G
	 id:                	 8141998e-5451-4ee9-bf27-f61982297ade
	 size:              	 364.75G
	 status:            	 warning  --------------->error status


	 name:              	 FLASH_1_0
	 diskType:          	 FlashDisk
	 luns:              	 1_0
	 makeModel:         	 "MARVELL SD88SA02"
	 physicalFirmware:  	 D20Y
	 physicalInsertTime:	 2012-05-10T02:43:53+08:00
	 physicalSize:      	 22.8880615234375G
	 slotNumber:        	 "PCI Slot: 1; FDOM: 0"
	 status:            	 poor performance   ------->error status

Reference by note 1504776.1 and step by step updating firmware.

[root@dm02cel03 dev]# imageinfo 

Kernel version: 2.6.18-274.18.1.0.1.el5 #1 SMP Thu Feb 9 19:07:16 EST 2012 x86_64
Cell version: OSS_11.2.3.1.0_LINUX.X64_120304
Cell rpm version: cell-11.2.3.1.0_LINUX.X64_120304-1

Active image version: 11.2.3.1.0.120304
Active image activated: 2012-05-07 02:13:48 -0700
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

In partition rollback: Impossible

Cell boot usb partition: /dev/sdm1
Cell boot usb version: 11.2.3.1.0.120304

Inactive image version: undefined
Rollback to the inactive partitions: Impossible

update firmware on storage cell node

1 comfirm that griddisk status should return no output

[root@dm02cel02 ~]# cellcli -e 'list griddisk attributes name,asmmodestatus' | \
>        egrep -v 'UNUSED|ONLINE'

or check other nodes disk status 

cellcli -e 'list griddisk attributes name,asmmodestatus' | \
       egrep -v 'UNUSED|ONLINE'

2 apply path 14793859

[root@dm02cel02 14793859]# ./p14793859.sh
[INFO] Begin patch 14793859
[INFO] Resetting ILOM to ensure clean reboot. Wait 240 seconds.
[INFO] Patch has been staged.  Stop cell services, then run '/opt/oracle.cellos/CheckHWnFWProfile -U /opt/oracle.cellos/iso/cellbits' to activate new firmware and restart system.
[INFO] If restarting cells in a rolling manner, ensure ASMDEACTIVATIONOUTCOME=Yes for all griddisks before activating new firmware.
[root@dm02cel02 14793859]# 
[root@dm02cel02 14793859]# 
[root@dm02cel02 14793859]# 
[root@dm02cel02 14793859]# cellcli -e 'alter griddisk all inactive'
GridDisk DATA_DM02_CD_00_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_01_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_02_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_03_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_04_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_05_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_06_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_07_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_08_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_09_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_10_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_11_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_02_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_03_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_04_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_05_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_06_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_07_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_08_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_09_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_10_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_11_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_00_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_01_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_02_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_03_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_04_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_05_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_06_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_07_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_08_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_09_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_10_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_11_dm02cel02 successfully altered
[root@dm02cel02 14793859]# 
[root@dm02cel02 14793859]# 
[root@dm02cel02 14793859]# cellcli -e 'alter cell shutdown services all'

Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.

3 active new firmware

[root@dm02cel02 14793859]# /opt/oracle.cellos/CheckHWnFWProfile -U /opt/oracle.cellos/iso/cellbits
SUNFlashDOM: OK
[INFO] Reset the ILOM before trying ILOM update. Wait for 240 seconds as part of the reset for ILOM to be ready.
Sent cold reset command to MC
Now updating the ILOM and the BIOS ...
[INFO] Start ILOM firmware upgrade to version 3.0.16.10.d r74499. Attempt 1 of 2.
[INFO] Generated temporary ILOM user: iu_hhyll
[INFO] Generated temporary ILOM password: ********
[INFO] ipmitool user set name 3 iu_hhyll
[INFO] ipmitool user set password 3 ********
[INFO] ipmitool sunoem cli force "set /SP/users/iu_hhyll role=aucro"
Connected. Use ^D to exit.
-> set /SP/users/iu_hhyll role=aucro

Set 'role' to 'aucro'


-> Session closed
Disconnected
[INFO] export IPMI_PASSWORD=********
[INFO] ipmiflash -v -I lanplus -H 10.61.1.239 -U iu_hhyll -E write /tmp/firmware/SUNBIOS force script config delaybios warning=0
[INFO] unset IPMI_PASSWORD
[INFO] ipmitool user set name 3 ""
Set User Name command failed (user 3, name ): Invalid data field in request

[INFO] ILOM update, ipmiflash return code 0

[INFO] ILOM will be reloaded in case of successful firmware upgrade to version 3.0.16.10.d r74499
[INFO] In this case close Sun ILOM Remote Console and corresponding Internet Exporer windows.
[INFO] Re-open both windows in few minutes.

[INFO] Waiting for the service processor to come up for 180 seconds
[INFO] ILOM firmware upgrade completed with success
Update all DOM firmware for all hbas
flash_dom -h 1 -d /tmp/firmware/SUNFlashDOM -b All -s

Aura Firmware Update Utility, Version 1.2.7

Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved..

U.S. Government Rights - Commercial Software. Government users are subject
to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

Use is subject to license terms.

This distribution may include materials developed by third parties.

Sun, Sun Microsystems, the Sun logo, Sun StorageTek and ZFS are trademarks
or registered trademarks of Sun Microsystems, Inc. or its subsidiaries,
in the U.S. and other countries.



 1.  /proc/mpt/ioc0    LSI Logic SAS1068E C0     105      011b5c00     0

Using tag SD88SA02 to identify the DOMs
36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Found four DOMs.

36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 0

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 0
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 1

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 1
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 2

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 2
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 3

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 3
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded


Before the changes will take effect you must power down the system
After you have rebooted the OS you will need to reformat all of the 
updated DOMs.

flash_dom -h 2 -d /tmp/firmware/SUNFlashDOM -b All -s

Aura Firmware Update Utility, Version 1.2.7

Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved..

U.S. Government Rights - Commercial Software. Government users are subject
to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

Use is subject to license terms.

This distribution may include materials developed by third parties.

Sun, Sun Microsystems, the Sun logo, Sun StorageTek and ZFS are trademarks
or registered trademarks of Sun Microsystems, Inc. or its subsidiaries,
in the U.S. and other countries.



 2.  /proc/mpt/ioc1    LSI Logic SAS1068E C0     105      011b5c00     0

Using tag SD88SA02 to identify the DOMs
36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Found four DOMs.

36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 0

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 0
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 1

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 1
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 2

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 2
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 3

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 3
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded


Before the changes will take effect you must power down the system
After you have rebooted the OS you will need to reformat all of the 
updated DOMs.

flash_dom -h 3 -d /tmp/firmware/SUNFlashDOM -b All -s

Aura Firmware Update Utility, Version 1.2.7

Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved..

U.S. Government Rights - Commercial Software. Government users are subject
to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

Use is subject to license terms.

This distribution may include materials developed by third parties.

Sun, Sun Microsystems, the Sun logo, Sun StorageTek and ZFS are trademarks
or registered trademarks of Sun Microsystems, Inc. or its subsidiaries,
in the U.S. and other countries.



 3.  /proc/mpt/ioc2    LSI Logic SAS1068E C0     105      011b5c00     0

Using tag SD88SA02 to identify the DOMs
36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Found four DOMs.

36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 0

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 0
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 1

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 1
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 2

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 2
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 3

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 3
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded


Before the changes will take effect you must power down the system
After you have rebooted the OS you will need to reformat all of the 
updated DOMs.

flash_dom -h 4 -d /tmp/firmware/SUNFlashDOM -b All -s

Aura Firmware Update Utility, Version 1.2.7

Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved..

U.S. Government Rights - Commercial Software. Government users are subject
to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

Use is subject to license terms.

This distribution may include materials developed by third parties.

Sun, Sun Microsystems, the Sun logo, Sun StorageTek and ZFS are trademarks
or registered trademarks of Sun Microsystems, Inc. or its subsidiaries,
in the U.S. and other countries.



 4.  /proc/mpt/ioc3    LSI Logic SAS1068E C0     105      011b5c00     0

Using tag SD88SA02 to identify the DOMs
36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Found four DOMs.

36.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 0

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 0
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 1

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 1
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 2

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 2
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded
36.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y

Updating Marvell firmware on Bus: 0, Target: 3

SAS1068E's links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, off, off, off, off

     B___T  Type       Vendor   Product          Rev
 1.  0   0  Disk       ATA      MARVELL SD88SA02 D20Y
 2.  0   1  Disk       ATA      MARVELL SD88SA02 D20Y
 3.  0   2  Disk       ATA      MARVELL SD88SA02 D20Y
 4.  0   3  Disk       ATA      MARVELL SD88SA02 D20Y
Update to bus 0, target 3
Using mode 5
Using BufferID 0

Downloading image...
Download succeeded


Before the changes will take effect you must power down the system
After you have rebooted the OS you will need to reformat all of the 
updated DOMs.

[WARNING] The hardware and firmware are not supported. See details below

[BIOSVersion]
Requires:
 08120104 
Found:
 08080102

[BIOSDate]
Requires:
 05/08/2012 
Found:
 05/23/2011

[ILOMVersion]
Requires:
 3.0.16.10.d r74499 
Found:
 3.0.16.10 r65138

[PCISlot:HBA:LSIModel:LSIhw:MPThw:LSIfw:MPTBios:DOM:OSDevice:DOMMake:DOMModel:DOMfw:CountAuraCountDOM]
Requires:
 AllSlots AllHBAs SAS1068E B3orC0 105 011b5c00 06.26.00.00 AllDOMs NotApplicable MARVELL SD88SA02 D21Y  4_16
 
Found:
 AllSlots AllHBAs SAS1068E B3orC0 105 011b5c00 06.26.00.00 AllDOMs NotApplicable MARVELL SD88SA02 D20Y  4_16

[WARNING] The hardware and firmware are not supported. See details above
[INFO] Rebooting in 5 minutes for firmware updates to take effect ...

                                     
Cache Flush is successfully done on adapter 0.

Exit Code: 0x00
Starting ipmi drivers: [  OK  ]

Starting ipmi_watchdog driver: [  OK  ]

Starting ipmi_poweroff driver: [  OK  ]


                                     
Cache Flush is successfully done on adapter 0.

Exit Code: 0x00
[INFO] Power cycle using /tmp/firmware/SUNBIOSPowerCycle
Wait 180 seconds for the ILOM power cycle package to take effect. Then start the power down.


Broadcast message from root (pts/0) (Thu Mar 14 13:36:45 2013):




The system is going down for system halt NOW!


Connection closed by foreign host.

——–

after reboot

1 Confirm the firmware update is complete

[root@dm02cel02 ~]#  cellcli -e "list physicaldisk attributes name, physicalFirmware \
>        where diskType = 'FlashDisk'"
	 FLASH_1_0	 D21Y
	 FLASH_1_1	 D21Y
	 FLASH_1_2	 D21Y
	 FLASH_1_3	 D21Y
	 FLASH_2_0	 D21Y
	 FLASH_2_1	 D21Y
	 FLASH_2_2	 D21Y
	 FLASH_2_3	 D21Y
	 FLASH_4_0	 D21Y
	 FLASH_4_1	 D21Y
	 FLASH_4_2	 D21Y
	 FLASH_4_3	 D21Y
	 FLASH_5_0	 D21Y
	 FLASH_5_1	 D21Y
	 FLASH_5_2	 D21Y
	 FLASH_5_3	 D21Y
[root@dm02cel02 ~]# /opt/oracle.cellos/CheckHWnFWProfile -d | grep -A1 ILOMVersion
[ILOMVersion]
 3.0.16.10.d r74499
[root@dm02cel02 ~]# /opt/oracle.cellos/CheckHWnFWProfile -c loose
[SUCCESS] The hardware and firmware profile matches one of the supported profiles

2 active griddisks

[root@dm02cel02 ~]# cellcli -e 'alter griddisk all active'
GridDisk DATA_DM02_CD_00_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_01_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_02_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_03_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_04_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_05_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_06_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_07_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_08_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_09_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_10_dm02cel02 successfully altered
GridDisk DATA_DM02_CD_11_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_02_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_03_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_04_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_05_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_06_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_07_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_08_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_09_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_10_dm02cel02 successfully altered
GridDisk DBFS_DG_CD_11_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_00_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_01_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_02_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_03_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_04_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_05_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_06_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_07_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_08_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_09_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_10_dm02cel02 successfully altered
GridDisk RECO_DM02_CD_11_dm02cel02 successfully altered

3 Recreate the CELLBOOT USB

[root@dm02cel02 ~]# /opt/oracle.cellos/make_cellboot_usb.sh -execute -force
Candidate for the Oracle Exadata Cell start up boot device     : /dev/sdm
Partition on candidate device                                  : /dev/sdm1
The current product version                                    : 11.2.3.1.0.120304
Label of the current Oracle Exadata Cell start up boot device  : CELLBOOT
The current CELLBOOT USB product version                       : 11.2.3.1.0.120304
[DEBUG] set_cell_boot_usb: cell usb        : /dev/sdm
[DEBUG] set_cell_boot_usb: mnt sys         : /
[DEBUG] set_cell_boot_usb: preserve        : preserve
[DEBUG] set_cell_boot_usb: mnt usb         : /mnt/usb.make.cellboot
[DEBUG] set_cell_boot_usb: lock            : /tmp/usb.make.cellboot.lock
[DEBUG] set_cell_boot_usb: serial console  : 
[DEBUG] set_cell_boot_usb: kernel mode     : kernel
[DEBUG] set_cell_boot_usb: mnt iso save    : 
Create CELLBOOT USB on device /dev/sdm

The number of cylinders for this disk is set to 2825.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 2825.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): Command action
   e   extended
   p   primary partition (1-4)
Partition number (1-4): First cylinder (1-2825, default 1): Last cylinder or +size or +sizeM or +sizeK (1-2825, default 2825): 
Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
489600 inodes, 978513 blocks
48925 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1002438656
30 block groups
32768 blocks per group, 32768 fragments per group
16320 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
tune2fs 1.39 (29-May-2006)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds
//opt/oracle.cellos/iso ~
Copying ./imgboot.lst to /mnt/usb.make.cellboot/. ...
...

Copying ./imgboot.lst.14793859 to /mnt/usb.make.cellboot/. ...
Copying ./initrd.img.14793859 to /mnt/usb.make.cellboot/. ...
Running "tar -x -j -p -v -C /mnt/usb.make.cellboot -f //opt/oracle.cellos/iso/cellbits/cellboot.tbz initrd-*.img vmlinuz-* grub/" ...
grub/
grub/xfs_stage1_5
grub/vstafs_stage1_5
grub/menu.lst
grub/e2fs_stage1_5
grub/grub.conf
grub/oracle.xpm.gz
grub/device.map
grub/stage1
grub/jfs_stage1_5
grub/reiserfs_stage1_5
grub/minix_stage1_5
grub/ufs2_stage1_5
grub/stage2
grub/iso9660_stage1_5
grub/fat_stage1_5
grub/ffs_stage1_5
initrd-2.6.18-194.3.1.0.2.el5.img
vmlinuz-2.6.18-194.3.1.0.2.el5
Copying //opt/oracle.cellos/tmpl/oracle.xpm.gz to /mnt/usb.make.cellboot/grub/oracle.xpm.gz ...
[DEBUG] set_grub_conf_n_initrd: mnt sys        : /
[DEBUG] set_grub_conf_n_initrd: grub template  : USB_grub.in
[DEBUG] set_grub_conf_n_initrd: boot dir       : /mnt/usb.make.cellboot
[DEBUG] set_grub_conf_n_initrd: kernel param   : 2.6.18-274.18.1.0.1.el5
[DEBUG] set_grub_conf_n_initrd: marker         : I_am_CELLBOOT_usb
[DEBUG] set_grub_conf_n_initrd: mode           : 
[DEBUG] set_grub_conf_n_initrd: Image id file: //opt/oracle.cellos/image.id
[DEBUG] set_grub_conf_n_initrd: System device where image id exists: /dev/md5
[DEBUG] set_grub_conf_n_initrd: Kernel version: 2.6.18-274.18.1.0.1.el5
[DEBUG] set_grub_conf_n_initrd: System device with image_id (/dev/md5) and kernel version (2.6.18-274.18.1.0.1.el5) are in sync
[DEBUG] set_grub_conf_n_initrd: Full kernel version: 2.6.18-274.18.1.0.1.el5
[DEBUG] set_grub_conf_n_initrd: system device for the next boot: /dev/md5
[DEBUG] set_grub_conf_n_initrd: initrd for the next boot: /mnt/usb.make.cellboot/initrd-2.6.18-274.18.1.0.1.el5.img
[INFO] set_grub_conf_n_initrd: Set /dev/md5 in /mnt/usb.make.cellboot/I_am_CELLBOOT_usb
[INFO] Set kernel 2.6.18-274.18.1.0.1.el5 and system device /dev/md5 in generated /mnt/usb.make.cellboot/grub/grub.conf
[INFO] Set /dev/md5 in /mnt/usb.make.cellboot/initrd-2.6.18-274.18.1.0.1.el5.img
43450 blocks
log/
log/do_image.sh.log
log/cellos.11.2.3.1.0.120304.20120507.021428.PDT.tar.gz
log/cellos.11.2.3.1.0.120304.20120507.032629.EDT.tar.gz


    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub> root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.

4 checking ASM disk recover status

[root@dm02cel02 ~]# cellcli -e list griddisk attributes name, asmmodestatus
	 DATA_DM02_CD_00_dm02cel02	 SYNCING  ---> recovering
	 DATA_DM02_CD_01_dm02cel02	 SYNCING
	 DATA_DM02_CD_02_dm02cel02	 SYNCING
	 DATA_DM02_CD_03_dm02cel02	 SYNCING
	 DATA_DM02_CD_04_dm02cel02	 SYNCING
	 DATA_DM02_CD_05_dm02cel02	 SYNCING
	 DATA_DM02_CD_06_dm02cel02	 SYNCING
	 DATA_DM02_CD_07_dm02cel02	 SYNCING
	 DATA_DM02_CD_08_dm02cel02	 SYNCING
	 DATA_DM02_CD_09_dm02cel02	 SYNCING
	 DATA_DM02_CD_10_dm02cel02	 SYNCING
	 DATA_DM02_CD_11_dm02cel02	 SYNCING
	 DBFS_DG_CD_02_dm02cel02  	 ONLINE   ---> complete
	 DBFS_DG_CD_03_dm02cel02  	 ONLINE
	 DBFS_DG_CD_04_dm02cel02  	 ONLINE
	 DBFS_DG_CD_05_dm02cel02  	 ONLINE
	 DBFS_DG_CD_06_dm02cel02  	 ONLINE
	 DBFS_DG_CD_07_dm02cel02  	 ONLINE
	 DBFS_DG_CD_08_dm02cel02  	 ONLINE
	 DBFS_DG_CD_09_dm02cel02  	 ONLINE
	 DBFS_DG_CD_10_dm02cel02  	 ONLINE
	 DBFS_DG_CD_11_dm02cel02  	 ONLINE
	 RECO_DM02_CD_00_dm02cel02	 OFFLINE  ---> not start recovering
	 RECO_DM02_CD_01_dm02cel02	 OFFLINE
	 RECO_DM02_CD_02_dm02cel02	 OFFLINE
	 RECO_DM02_CD_03_dm02cel02	 OFFLINE
	 RECO_DM02_CD_04_dm02cel02	 OFFLINE
	 RECO_DM02_CD_05_dm02cel02	 OFFLINE
	 RECO_DM02_CD_06_dm02cel02	 OFFLINE
	 RECO_DM02_CD_07_dm02cel02	 OFFLINE
	 RECO_DM02_CD_08_dm02cel02	 OFFLINE
	 RECO_DM02_CD_09_dm02cel02	 OFFLINE
	 RECO_DM02_CD_10_dm02cel02	 OFFLINE
	 RECO_DM02_CD_11_dm02cel02	 OFFLINE

[root@dm02cel02 ~]# cellcli -e list griddisk attributes name, asmmodestatus
	 DATA_DM02_CD_00_dm02cel02	 SYNCING
	 DATA_DM02_CD_01_dm02cel02	 SYNCING
	 DATA_DM02_CD_02_dm02cel02	 SYNCING
	 DATA_DM02_CD_03_dm02cel02	 SYNCING
	 DATA_DM02_CD_04_dm02cel02	 SYNCING
	 DATA_DM02_CD_05_dm02cel02	 SYNCING
	 DATA_DM02_CD_06_dm02cel02	 SYNCING
	 DATA_DM02_CD_07_dm02cel02	 SYNCING
	 DATA_DM02_CD_08_dm02cel02	 SYNCING
	 DATA_DM02_CD_09_dm02cel02	 SYNCING
	 DATA_DM02_CD_10_dm02cel02	 SYNCING
	 DATA_DM02_CD_11_dm02cel02	 SYNCING
	 DBFS_DG_CD_02_dm02cel02  	 ONLINE
	 DBFS_DG_CD_03_dm02cel02  	 ONLINE
	 DBFS_DG_CD_04_dm02cel02  	 ONLINE
	 DBFS_DG_CD_05_dm02cel02  	 ONLINE
	 DBFS_DG_CD_06_dm02cel02  	 ONLINE
	 DBFS_DG_CD_07_dm02cel02  	 ONLINE
	 DBFS_DG_CD_08_dm02cel02  	 ONLINE
	 DBFS_DG_CD_09_dm02cel02  	 ONLINE
	 DBFS_DG_CD_10_dm02cel02  	 ONLINE
	 DBFS_DG_CD_11_dm02cel02  	 ONLINE
	 RECO_DM02_CD_00_dm02cel02	 SYNCING
	 RECO_DM02_CD_01_dm02cel02	 SYNCING
	 RECO_DM02_CD_02_dm02cel02	 SYNCING
	 RECO_DM02_CD_03_dm02cel02	 SYNCING
	 RECO_DM02_CD_04_dm02cel02	 SYNCING
	 RECO_DM02_CD_05_dm02cel02	 SYNCING
	 RECO_DM02_CD_06_dm02cel02	 SYNCING
	 RECO_DM02_CD_07_dm02cel02	 SYNCING
	 RECO_DM02_CD_08_dm02cel02	 SYNCING
	 RECO_DM02_CD_09_dm02cel02	 SYNCING
	 RECO_DM02_CD_10_dm02cel02	 SYNCING
	 RECO_DM02_CD_11_dm02cel02	 SYNCING

NOTE: Found o/192.168.10.4/DATA_DM02_CD_00_dm02cel02 for disk DATA_DM02_CD_00_DM02CEL02
SUCCESS: disk DATA_DM02_CD_00_DM02CEL02 (12.3746554323) replaced in diskgroup DATA_DM02
..

NOTE: Found o/192.168.10.4/DATA_DM02_CD_11_dm02cel02 for disk DATA_DM02_CD_11_DM02CEL02
SUCCESS: disk DATA_DM02_CD_11_DM02CEL02 (23.3746554331) replaced in diskgroup DATA_DM02

NOTE: disk 12 (DATA_DM02_CD_00_DM02CEL02) in group 1 (DATA_DM02) is online for writes
..
NOTE: disk 23 (DATA_DM02_CD_11_DM02CEL02) in group 1 (DATA_DM02) is online for writes

NOTE: Found o/192.168.10.4/RECO_DM02_CD_00_dm02cel02 for disk RECO_DM02_CD_00_DM02CEL02
SUCCESS: disk RECO_DM02_CD_00_DM02CEL02 (12.3746554389) replaced in diskgroup RECO_DM02
..

NOTE: Found o/192.168.10.4/RECO_DM02_CD_11_dm02cel02 for disk RECO_DM02_CD_11_DM02CEL02
SUCCESS: disk RECO_DM02_CD_11_DM02CEL02 (23.3746554395) replaced in diskgroup RECO_DM02



Thu Mar 14 13:48:47 2013
NOTE: disk 12 (RECO_DM02_CD_00_DM02CEL02) in group 3 (RECO_DM02) is online for reads
..
NOTE: disk 23 (RECO_DM02_CD_11_DM02CEL02) in group 3 (RECO_DM02) is online for reads


Thu Mar 14 13:55:58 2013
NOTE: disk 12 (DATA_DM02_CD_00_DM02CEL02) in group 1 (DATA_DM02) is online for reads
..
NOTE: disk 23 (DATA_DM02_CD_11_DM02CEL02) in group 1 (DATA_DM02) is online for reads

5 after all disks status are online and check flash disk work well.

CellCLI> LIST METRICCURRENT WHERE objectType = 'FLASHCACHE'
	 FC_BYKEEP_OVERWR       	 FLASHCACHE	 0.000 MB
	 FC_BYKEEP_OVERWR_SEC   	 FLASHCACHE	 0.000 MB/sec
	 FC_BYKEEP_USED         	 FLASHCACHE	 0.000 MB
	 FC_BY_USED             	 FLASHCACHE	 9,988 MB
	 FC_IO_BYKEEP_R         	 FLASHCACHE	 0.000 MB
	 FC_IO_BYKEEP_R_SEC     	 FLASHCACHE	 0.000 MB/sec
	 FC_IO_BYKEEP_W         	 FLASHCACHE	 0.000 MB
	 FC_IO_BYKEEP_W_SEC     	 FLASHCACHE	 0.000 MB/sec
	 FC_IO_BY_R             	 FLASHCACHE	 5,790 MB
	 FC_IO_BY_R_MISS        	 FLASHCACHE	 12,530 MB
	 FC_IO_BY_R_MISS_SEC    	 FLASHCACHE	 1.540 MB/sec
	 FC_IO_BY_R_SEC         	 FLASHCACHE	 5.707 MB/sec
	 FC_IO_BY_R_SKIP        	 FLASHCACHE	 120,480 MB
	 FC_IO_BY_R_SKIP_SEC    	 FLASHCACHE	 202 MB/sec
	 FC_IO_BY_W             	 FLASHCACHE	 12,110 MB
	 FC_IO_BY_W_SEC         	 FLASHCACHE	 1.415 MB/sec
	 FC_IO_ERRS             	 FLASHCACHE	 0
	 FC_IO_RQKEEP_R         	 FLASHCACHE	 0 IO requests
	 FC_IO_RQKEEP_R_MISS    	 FLASHCACHE	 0 IO requests
	 FC_IO_RQKEEP_R_MISS_SEC	 FLASHCACHE	 0.0 IO/sec
	 FC_IO_RQKEEP_R_SEC     	 FLASHCACHE	 0.0 IO/sec
	 FC_IO_RQKEEP_R_SKIP    	 FLASHCACHE	 0 IO requests
	 FC_IO_RQKEEP_R_SKIP_SEC	 FLASHCACHE	 0.0 IO/sec
	 FC_IO_RQKEEP_W         	 FLASHCACHE	 0 IO requests
	 FC_IO_RQKEEP_W_SEC     	 FLASHCACHE	 0.0 IO/sec
	 FC_IO_RQ_R             	 FLASHCACHE	 579,959 IO requests
	 FC_IO_RQ_R_MISS        	 FLASHCACHE	 386,847 IO requests
	 FC_IO_RQ_R_MISS_SEC    	 FLASHCACHE	 48.2 IO/sec
	 FC_IO_RQ_R_SEC         	 FLASHCACHE	 651 IO/sec
	 FC_IO_RQ_R_SKIP        	 FLASHCACHE	 220,180 IO requests
	 FC_IO_RQ_R_SKIP_SEC    	 FLASHCACHE	 263 IO/sec
	 FC_IO_RQ_W             	 FLASHCACHE	 455,710 IO requests
	 FC_IO_RQ_W_SEC         	 FLASHCACHE	 47.6 IO/sec

CellCLI> LIST METRICCURRENT WHERE objectType = 'FLASHLOG'  
	 FL_ACTUAL_OUTLIERS           	 FLASHLOG	 0 IO requests
	 FL_BY_KEEP                   	 FLASHLOG	 0
	 FL_DISK_FIRST                	 FLASHLOG	 155,818 IO requests
	 FL_DISK_IO_ERRS              	 FLASHLOG	 0 IO requests
	 FL_EFFICIENCY_PERCENTAGE     	 FLASHLOG	 100 %
	 FL_EFFICIENCY_PERCENTAGE_HOUR	 FLASHLOG	 100 %
	 FL_FLASH_FIRST               	 FLASHLOG	 11,176 IO requests
	 FL_FLASH_IO_ERRS             	 FLASHLOG	 0 IO requests
	 FL_FLASH_ONLY_OUTLIERS       	 FLASHLOG	 0 IO requests
	 FL_IO_DB_BY_W                	 FLASHLOG	 5,106 MB
	 FL_IO_DB_BY_W_SEC            	 FLASHLOG	 0.768 MB/sec
	 FL_IO_FL_BY_W                	 FLASHLOG	 5,605 MB
	 FL_IO_FL_BY_W_SEC            	 FLASHLOG	 1.955 MB/sec
	 FL_IO_W                      	 FLASHLOG	 166,994 IO requests
	 FL_IO_W_SKIP_BUSY            	 FLASHLOG	 0 IO requests
	 FL_IO_W_SKIP_BUSY_MIN        	 FLASHLOG	 0.0 IO/sec
	 FL_IO_W_SKIP_LARGE           	 FLASHLOG	 0 IO requests
	 FL_PREVENTED_OUTLIERS        	 FLASHLOG	 0 IO requests

6 When all these steps are completed proceed to the next storage server.

———————————–

Update database server node.

1 after all rolling update on cell nodes do database node update one by one.

2 Shutdown and disable Oracle Clusterware. Database instances and other
cluster resources will be stopped in this step.

# GRID_HOME/grid/bin/crsctl stop crs
# GRID_HOME/grid/bin/crsctl disable crs
3 apply patch 14793859

# cd /root/14793859
# ./p14793859.sh
4 Activate the new firmware
/opt/oracle.cellos/CheckHWnFWProfile -U /opt/oracle.cellos/iso/cellbits

5 Confirm the ILOM update is complete.

# /opt/oracle.cellos/CheckHWnFWProfile -d | grep -A1 ILOMVersion
[ILOMVersion]
3.0.16.10.d r74499

Run the CheckHWnFWProfile utility to verify firmware profile is supported

# /opt/oracle.cellos/CheckHWnFWProfile -c loose
[SUCCESS] The hardware and firmware profile matches one of the supported profiles

6 Enable and startup Oracle Clusterware.

# GRID_HOME/grid/bin/crsctl enable crs
# GRID_HOME/grid/bin/crsctl start crs

7 Proceed to the next database server and end of all of firmware update.