HW - FIRMWARE

FIRMWARE MANAGEMENT FROM HMC

Managed Systems Firmware Update

The FSP (Flexible Service Processor) is responsible to the communication between the HMC and the Power System. This FSP is using a special software called the firmware, which provides low-level control of the hardware. Firmware installation/update can be done through the HMC or through the OS, but if a system is managed by an HMC, then FW update is done through the HMC. (Sometimes IBM calls this firmware as LIC: Licensed Internal Code)

The service processor maintains two copies of the server firmware. One copy is considered the permanent copy and is stored on the permanent side. ("p" side). The other copy is considered the temporary copy and is stored on the temporary side  ("t" side). It is recommended that you start and run the server from the temporary side. When you install a server firmware update, it is installed on the temporary side. 

A FW update consists of 2 steps:

- installation: update what is in flash (read only memory).

- activation: making the new version to run on the system

Not all updates can be activated concurrently:

- Concurrent: Apply and activate with partitions running.

- Deferred: Concurrent activation but contains updates that affect the initial program load (IPL), these are not activated until the system is powerd off and on.

- Disruptive: System shutdown and restart is required; (none of the update contents are activated until the next time you shut down and restart the server).

If a system runs on the Permanent side, then a concurrent update is not possible.  Regardless of firmware update type, if you see this message during a firmware update, it means, it will be a disruptive update. The accept operation cannot be performed because all components are running on the permanent flash side.

When checking firmware level:

- Installed level: This level has been installed and will be activated (loaded into memory) after the managed system is powered off/on.

- Activated level: This is the level that is active and running in memory.

- Accepted level: This is the backup level. You can return to this level if you remove (updlic -o r ...) the current level.(this is the code on the permanent side.)

- Deferred level: It indicates the firmware level that contains unactivated deferred updates. (system restart is needed to activate these)

- Platform IPL level: This is the level the system was booted on. (After concurrent upd., activated level will change, but platf. IPL level remain unchanged)

# lslic -t sys -m <man. sys.>

activated_level=136,activated_spname=FW950.90,                  <-- this is active now   

installed_level=136,installed_spname=FW950.90,                  <-- during reboot this will be activated

accepted_level=131,accepted_spname=FW950.80,                    <-- backup level (permanent side),

deferred_level=None,deferred_spname=None,                       <--this level contains unactivated updates

platform_ipl_level=87,platform_ipl_spname=FW950.20,             <--this level was used during last reboot

curr_level_primary=136,curr_spname_primary=FW950.90,            <--current (activated) level, of the primary service processor

temp_level_primary=136,temp_spname_primary=FW950.90,            <-- temp side of the primary serv. proc.

perm_level_primary=131,perm_spname_primary=FW950.80,            <-- perm. side of the primary serv. proc.

curr_level_secondary=136,curr_spname_secondary=FW950.90,        <--curr. (activated) level of the secondary serv. proc. (P980/P1080... have redundant serv. processors: prim./sec.)

temp_level_secondary=136,temp_spname_secondary=FW950.90,        <-- temp side of the secondary serv. proc.

perm_level_secondary=131,perm_spname_secondary=FW950.80,        <-- perm. side of the secondary serv. proc.

---------------------------------------------------------

updlic command

The updlic command can be used to update (query) the system firmware from the HMC CLI

An example to do a concurrent update from IBM website:

updlic -m <man. sys.> -o a -t sys -l latestconcurrent -r ibmwebsite -v

-o  <action>

a: retreive,install and activate LIC update (previously activated updates will be automatically accepted, and accept means: copy T --> P)

i: retreive,install but not activate a LIC update

c: accept currently activated LIC update (copy T --> P)

d: disruptively activate a LIC update

u: upgrade LIC to a new release (it will as for a restart)

r: remove the recently installed update and activate the previouly acceted level (it brings back the level that is on the permanent side)

j: reject an installed LIC update (copy P to T) 

  (!!!this option (j) can be used ONLY if we boot the system first to the permanent side, so P side is active, then can we copy P to T.)

-t <type>

sys:   for managed system only, 

io:    for I/O updates only, 

all:   for managed system and I/O updates, 

sriov: for SR-IOV adapter updates

-l <level>

latest:           update to latest level, even if disruptive, 

latestconcurrent: update to latest concurrent level 

sss:              update to a specific level, even if disruptive (sss is 3 cahracter id, like level=136)

release_level:    update to a specific release and level, even if disruptive. (950_136 ???)

-r <repository> : ibmwebsite, ftp/sftp, disk/mountpoint for internal hard disk

-q query if an update is concurrent or disruptive. (The update is not performed with this option, and exit codes are documented in man page of updlic)

-----------------------------------------------------

SR-IOV Firmware Update

SR-IOV shared-mode adapters are updated during the system firmware update. If you are updating the system firmware concurrently, the new SR-IOV firmware is not automatically activated to prevent any unexpected outage.

Two types of firmware are required for adapters in SR-IOV shared-mode:

1. adapter driver firmware: it is used for configuring and managing the adapter. 

2. adapter firmware: it enables the adapter to interface with the adapter driver firmware. 

When you update the system firmware, this new system firmware might also contain adapter driver update or adapter firmware update, or both. The new SR-IOV level is not activated automatically while they are running because of a temporary I/O outage that occurs when the firmware is activated, so we can schedule a time for this outage. The outage lasts about 1 minute per adapter for the adapter driver firmware, and about 5 minutes per adapter  when you activate both the adapter driver and the adapter firmware. The best practice is to activate both simultaneously, you cannot activate only the adapter firmware. 

New SR-IOV firmware can be activated with system boot, or during maintenance (for example if adapter is replaced during maintenance), or with manual activation from HMC (GUI/CLI) (Firmware update for adapters that are not in SR-IOV shared mode (in dedicated mode) can be done through the OS that owns the adapter or by using HMC as an IO Adapter update.)

-----------------------------------------------------

Commands:

Man. Sys.:

lslic -m <man_sys> -t sys                          show man. sys. firmware details (installed/activated level....)

lslic -m <man_sys> -t sys -l update -r ibmwebsite  show if there are any fw updates available at ibm site, shows concurrency status too 

                                                   (-l upgrade will check for upgrades)

updlic -m <man. sys.> -o a -t all -l latest -r ibmwebsite -q  query whether the latest updates at IBM site are concurr. or disruptive

updlic -m <man_sys> -o k -v                                   readiness check (check before upd. for any issues (events/hw failures..)

updlic -m <man_sys> -o a -t sys -l latestconcurrent -r ibmwebsite -v    update man. sys. to the lates concurrent level from ibm site

updlic -m <man_sys> -o r -t sys -v                                      removing last update (go back to previous level)

lslic -c                                                                list FW levels on HMC local disk (repository)

updlic -o p --ecnumber 01VM920                                          remove all FW levels belonging to this release from HMC disk

SR-IOV:

lslic -m <man_sys> -t sriov     show sriov level and if there are any updates (update_available=0 means there are no new sriov updates)

                                (install_separate=1 means the adapter supports to only update the adapter driver firmware)

updlic -m <man_sys> -o f -t sriov --subtype adapterdriver,adapter -s adapt_id -v     update (activate) sriov driver+adapter fw (5 min outage, lslic lists adapter id)

updlic -m <man_sys> -o f -t sriov --subtype adapterdriver -s adapt_id1,adapt_id2 -v  update only adapter driver firmware for more adapters (1 min outage)

I/O Adapter:

lslic -m <man_sys> -t io -l update -r ibmwebsite                        check new levels for fc adapter, NVME devices... 

                                                                        (available_level will show new level)

updlic -m <man_sys> -o a -t io -l latestconcurrent -r ibmwebsite -v     update io adapters to new level from ibm site 

                                                                        (can be done online, just not in peak hours)

                                                                        (once a cluster was too sensitive for io latency, and it was reacted during fc adapter upd.)

-----------------------------------------------------

FIRMWARE MANAGEMENT FROM OS (AIX/VIO)

ADAPTER/SYSTEM FIRMWARE:

lsmcode -A                   displays microcode (same as firmware) level information for all supported devices 
                             (Firmware: Software that has been written onto read-only memory (ROM))
lsmcode -c                   shows firmware level for system, processor

invscout:
it helps to show which firmware (microcode) should be updated:

1. download: http://public.dhe.ibm.com/software/server/firmware/catalog.mic
2. copy on the server to:  /var/adm/invscout/microcode
3. run: invscout (it will collect data and creates: /var/adm/invscout/<hostname>.mup)
4. upload <hostanme>.mup to: http://www14.software.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

-----------------------------------------------------

SYSTEM FIRMWARE UPDATE:
(update is concurrent, upgrade is disruptive)

1. download from FLRT the files
2. copy to NIM server (NFS export) the files or put onto an FTP server (xml and rpm was in it for me)
3. makes sure there are no deconfigured cpu/ram in the server, or broken hardware devices
4. On HMC -> LIC (Licensed Internal Code) Maintenance -> LIC  Updates -> Channge LIC for the current release
(if you want to do an upgrade (not update) choose: Upgrade Licensed Internal Code)
5. Choose the machine -> Channge LIC (Licensed Internal Code) wizard
6. FTP site:
    FTP site:10.20.10.10
    User ID: root
    Passw: <root pw>
    Change Directory: set to the uploaded dir

7. follow the wizard (next, next..), about after 20-30 minutes will be done

"The repository does not contain any applicable upgrade updates HSCF0050" or ..."The selected repository does not contain any new updates."
It can happen if ftp user was not created by official AIX script: /usr/samples/tcpip/anon.ftp
Create ftp user with this and try again.

-----------------------------------------------------

ADAPTER FIRMWARE LEVEL:


For FC adapter:
root@aix1: /root # lscfg -vpl fcs0
  fcs0             P2-I2  FC Adapter

        Part Number.................09P5079
        EC Level....................A
        Serial Number...............1C2120A894
        Manufacturer................001C
        Customer Card ID Number.....2765            <--it shows the feature code (could be like this: Feature Code/Marketing ID...5704)
        FRU Number..................09P5080         <--identifies the adapter
        Network Address.............10000000C92BC1EF
        ROS Level and ID............02C039D0
        Device Specific.(Z0)........2002606D
        ...
        Device Specific.(Z9)........CS3.93A0        <--this is the same as ZB
        Device Specific.(ZA)........C1D3.93A0       
        Device Specific.(ZB)........C2D3.93A0       <--to verify the firmware level ignore the first 3 characters in the ZB field (3.93A0)
        Device Specific.(ZC)........00000000
        Hardware Location Code......P2-I2


For Network adapter:
lscfg -vpl entX --> FRU line

ROM Level.(alterable).......GOL021                <--it is showing the firmware level


For Internal Disks:
lscfg -vpl hdiskX

Manufacturer................IBM
Machine Type and Model......HUS153014VLS300



Firmware (Microcode) update should be started from FLRT (Fix Level Recommendatio Tool).
After selecting the system, it will be possible to choose devices to do an update, and read description, how to do it.
Check with Google the "FRU Number" to get the FC Code (this is needed to get the correct files on FLRT)

Basic steps for firmware upgrade:

0. check if /etc/microcode dir exists
1. download from FLRT to a temp dir the needed rpm package.
2. make the rpm package available to the system:
    cd to the copied temp dir
    rpm -ihv --ignoreos  --force SAS15K300-A428-AIX.rpm

3. check:
    rpm -qa
    cd /etc/microcode --> ls -l will show the new microcede
    compare filesize within the document
    compare checksum within document: sum <filename> (sum /usr/lib/microcode/df1000fd-0002.271304)

4. diag -d fcs0 -T download        <--it will install (download) the microcode

Description about: "M", "L", "C", "P" when choosing the microcode:

    "M" is the most recent level of microcode found on the source.
    It is later than the level of microcode currently installed on the adapter.  
                                             
    "L" identifies levels of microcode that are later than the level of microcode currently installed on the adapter.
    Multiple later images are listed in descending order.                

    "C" identifies the level of microcode currently installed on the adapter.                        
                                                 
    "P" identifies levels of microcode that are previous to the level of microcode currently installed on the adapter.
    Multiple previous images are listed in the descending order. 

5. lsmcode -A                <--for checking


To Back level the firmware
diag -d fcsX -T "download -f -l previous"