dropdown menu

Showing posts with label HMC. Show all posts
Showing posts with label HMC. Show all posts

HMC - INSTALL, CONFIG

HMC Install, Config

HMC network types:

HMC has 2 different networks:
- private network: for accessing/managing Power servers through the FSP
- open network: this network used for user logins, and each LPAR is doing DLPAR operations through this network

Private networks

The only devices on the HMC private network are the HMC itself and each of the managed systems to which that HMC is connected. The HMC is connected to each managed system's FSP (Flexible Service Processor). On most systems, the FSP provides two Ethernet ports labeled HMC1 and HMC2. This allows you to connect up to two HMCs.


This figure shows a redundant HMC environment with two managed systems. The "blue" HMC is connected to the first port on each FSP, and the "red" (redundant) HMC is connected to the second port. Each HMC is configured as a DHCP server, using a different range of IP addresses. The connections are on separate private networks. As such, it is important to ensure that no FSP port is connected to more than one HMC.

Each managed system's FSP port that is connected to an HMC requires a unique IP address. To ensure that each FSP has a unique IP address, use the HMC's built-in DHCP server capability. When the FSP detects the active network link, it issues a broadcast request to locate a DHCP server. When correctly configured, the HMC responds to that request by allocating one of a selected range of addresses.

--------------------------------------

Open networks

The open (public) network used for user logins and for DLPAR operations. Connecting to the Internet allows the HMC to "call home" when there are any hardware errors that need to be reported.


This figure shows two HMCs connected to a single managed server on the private network and to three logical partitions on the public network.

(If you decide to use a private network, use DHCP, and if you will use an open network to manage an FSP, you must set the FSP’s address manually through the Advanced System Management Interface menus, however a private, non-routable network is recommended.)

--------------------------------------

INSTALLING AND CONFIGURING A NEW HMC:

If this is a new installation, ensure that the managed system is not connected to a power source. If this is a second HMC that is connected to the same managed system, the managed system can be connected to a power source.

1.  Turn on the HMC and wait for the HMC to automatically select the default language and locale preference after 30 seconds.
2.  Accept the HMC license agreements and click "Log on and launch the Hardware Management Console web application."
3.  Login with hscroot und default-pw abc123
4.  Launch "Guided Setup Wizard"


5.  Change date/time/timezone
6.  Change default password for hscroot
7.  Configure HMC networks


    eth0 (private network):
    network type = private network
    Speed = default is "Autodetection" (if conn. to a switch, configure it manually: Change network sett. -> LAN Adapter)
    Enable DHCP = specify a DHCP address range
    (You can configure the HMC to be a DHCP server only on a private network)

    eth1 (public network: hmc-admin interface)
    network type = public network
    speed = default is "Autodetection" (if conn. to a switch, configure it manually: Change network sett. -> LAN Adapter)
    setup IP, netmask and gateway

8. Enable firewall-settings for eth1 (in the eth1 dialog-box)
    check or allow the following Services and Ports:
    ssh                         22:tcp
    secure remote web access    443:tcp,8443:tcp,9960:tcp
    RMC                         657:udp,657:tcp

9. Leave all other dialogs unchanged and finish changes made by the Wizard.

10. On the main view click on „HMC-Management“ and do the following:
    Enable „Remote Command execution“
    Enable „Remote Virtual Terminal“
    Enable „Remote Operation“

11. Reboot the HMC (so configuration changes will be available)
12. Login with the new hscroot password
13. Plug network cables into the HMC and check if HMC is answering DHCP request by plugging the cable to your laptop.

--------------------------------------

vHMC on Windows

(In this procedure I will use Cygwin, which I don't really like, because it is not easy to uninstall, but I could not find other working method. If you try below steps please keep in mind the complexity of uninstallation of Cygwin.)

Since some years IBM offered the possibility to run HMC on a VM (as an LPAR or in VMware on x86 ...), it is called vHMC (virtual HMC). Earlier the only option was to buy a physical HMC device, where the HMC application was running. With vHMC we download an image and use it on a VM.

For tests at home Ben Swinney wrote a great document how to run vHMC on Mac computers (http://gibsonnet.net/blog/cgaix/resource/vHMC_running_on_Apple_hardware.pdf) and based on that I tested vHMC on Windows:

1. install VirtualBox
2. download vHMC image from IBM ESS (KVM image has been downloaded, it is a 4GB tar.gz file)
3. unzip (file size was still 4GB) and tried to untar, but untar is not easy on Windows:
The downloaded tar file (beside some other files) contains a disk image file (disk1.img). This disk image is 500GB, but to save disk space it is a special file, called: sparse file. (If a file contains a lots of consecutive "0"s, and if that file is a sparse file, the actual "0"s will not be written on the disk, some method will remember for that, so we can save a lots of disk space.) IBM requirement for a disk in a vHMC is 500GB, but the HMC application is max. 10-20GB, so the rest in that image are a lots of "0"s. (The process to untar and convert that file may need temporarily more space, at first conversion failed with 100GB free space on local drive, so I decided to use an external drive with 1TB free space, but at the end file size will be about 10GB.)

Windows 10 supports sparse files, but it is not easy to untar a file on Windows and keep it as a sparse file. I used Cygwin:
- install cygwin, and start a cygwin terminal
- mount download dir as sparse: mount -o sparse D:/tmp /mnt (my image was in D:\tmp, please note in mount command "/" character is used)
- untar with sparse: tar -Sxvf /mnt/vHMC_KVM_image.tar (ls shows 500GB, but if free space is checked on Win., it shows max 20GB is taken)

4. convert disk1.img to virtual box compatible vdi file: "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" convertdd "D:\tmp\disk1.img" "D:\tmp\disk1.vdi"
This conversion is done by VBoxManage.exe, which should be installed by default during VirtualBox installation. After the conversion we should have a normal vdi file (not sparse file) with about 10GB:


5. create a VM in Virtualbox. I used these specifics:
    - type: Linux, version: Red Hat (64bit)
    - Memory size: 4096MB
    - using an existing virtual hard disk file (I copied disk1.vdi to VirtualBox VM directory)
  After creation (before starting the VM) in Settings:
   - in Display change Video Memory to 64MB
   - in Audio disable audio
   - in Networks enable Adapters (for me it was a test, I did not configure them for real usage):
      Adapter 1: NAT with Port Forwarding (in Advanced section): Host IP: 127.0.1.1, Host and Guest port:22, IP I used: 192.168.0.9
      Adapter 2: Bridged Adapter with default settings
      Adapter 3: Bridged Adapter with default settings

6. Start VM and follow Guided Setup Wizard to configure your vHMC :)))))
    Start up time will take longer, but after it should work normally. If network/IP configured correctly it can be accessed remotely:


--------------------------------------

HMC LDAP configuration

By default HMC is using local authentication, but it is possible to change it to LDAP. If LDAP authentication is configured, HMC users needs to be modified to use LDAP. It is also possible that some HMC users use LDAP and others use local authentication. (It is a user parameter (Auhtentication type) which decides how the user should login.)

Configuring LDAP
1. find out ldap server, port and protocol (ldap or ldaps)
We need to know the ldap server and ldap or ldaps protocol will be used, and which port can be used for LDAP requests:
Ports: 389 (LDAP) / 636 (LDAPS)   <--request information from local domain controller (search in the home domain of the global catalog)
Ports: 3268 (LDAP) / 3269 (LDAPS) <--queries specifically targeted for the Global Catalog (search in the entire forest)
(In below example we will use server myldap.mydomain.org, with ldaps protocol on port 3269)

2.AD account for LDAP communication with Active Directory
An AD user and password is needed, that is used by HMC for LDAP communication. This user does not need to exist on the HMC. When someone wants to login to the HMC, it will connect to AD and performs searches in AD. (In below example we will use HMCAD1 user with password abcd1234.)

3. Change HMC authentication from local to LDAP
For this an HMC user account is needed (like hscroot) with hmcsuperadmin role:
$ chhmcldap -o s --primary 'ldaps://myldap.mydomain.org:3269' --basedn 'DC=mydomain,DC=org' --binddn 'CN=HMCAD1,OU=Functional,OU=Accounts,DC=mydomain,DC=org' --auth ldap --loginattribute 'sAMAccountName'  --bindpw 'abcd1234' --scope sub --referrals 1 --automanage 0

4. Check config on HMC
$ lshmcldap -r config -v
primary=ldap://myldap.mydomain.org:3269,backup=,"basedn=DC=mydomain,DC=org",timelimit=30,bindtimelimit=30,referrals=1,ssl=0,loginattribute=sAMAccountName,hmcauthnameattribute=userPrincipalName,hmcuserpropsattribute=ibm-aixAdminPolicyEntry,"binddn=CN=HMCAD1,OU=Functional,OU=Accounts,DC=mydomain,DC=org",bindpwset=1,automanage=0,auth=ldap,searchfilter=,scope=sub,tlscacert=,hmcgroups=,authsearch=base

Details:

Get LDAP Config:
LdapSettings query primary[ldap://myldap.mydomain.org:3269]
LdapSettings query backup {}
LdapSettings query loginattribute[sAMAccountName]
LdapSettings query basedn[DC=mydomain,DC=org]
LdapSettings query timelimit[30]
LdapSettings query timelimit[30]
LdapSettings query referrals[1]
LdapSettings query bindtimelimit[30]
LdapSettings query tlscheckpeer {rc=1, stdOut=QueryTlsCheckPeer: tls_checkpeer not found!
, stdErr=}
LdapSettings query ssl[0]
LdapSettings query binddn[CN=HMCAD1,OU=Functional,OU=Accounts,DC=mydomain,DC=org ]
LdapSettings query bindpw [*************]
LdapSettings query searchfilter[]
LdapSettings query scope[sub]
LdapSettings query pam_authc_search {rc=1, stdOut=, stdErr=_QueryConfigAttr: pam_authc_search not found!
}
LdapSettings query tls_cacert[]
LdapSettings query authnameattribute[userPrincipalName]
LdapSettings query propsattribute[ibm-aixAdminPolicyEntry]
LdapSettings query hmcgroups[]


New user creation with LDAP:
$ mkhmcusr -i "name=gomaata3,taskrole=hmcsuperadmin,authentication_type=ldap,remote_webui_access=1,description=Super AIX Admin"
(Or use HMC GUI: Users and Roles --> Manage User Profiles and Access)


Modifying existing user from local authentication to LDAP:
$ chhmcusr -i "name=gomaata3,authentication_type=ldap"
(Or use HMC GUI: Users and Roles --> Manage User Profiles and Access)

--------------------------------------

Determining HMC adapter names:

If you set up a DHCP server, that server can operate only on eth0 and eth1.(You might need to determine which adapter to plug the ethernet cable into.)

To determine the name the HMC has assigned to an Ethernet adapter, do the following:
1. Open the restricted shell terminal. Select HMC Management -> Open Restricted Shell Terminal.
2. Type the following at the command line: tail -f /var/log/messages
3. Plug in your Ethernet cable. (If the cable was already plugged in, then unplug it, wait 5 seconds, and plug in the cable again.)
5. You will see a similar output on the console: Aug 28 12:41:20 termite kernel: e1000: eth0: e1000_watchdog: NIC Link is Up 100.
6. Repeat this procedure for all other Ethernet ports, and record your results.
7. Type Ctrl+C to stop thetailcommand.

--------------------------------------

Testing the connection between the HMC and the managed system:

This option enables you to verify that you are properly connected to the network:
(to test network connectivity, you must be a member of super administrator or service representative)
1. In the Navigation area, click HMC Management.
2. Click Test Network Connectivity.
3. In the Ping tab, type the host name or IP address of any system to which you want to connect. To test an open network, type the gateway.
4. Click Ping.

--------------------------------------

put ssh_key to HMC (passwordless login as hscroot)

version 1:
    - scp hscroot@hmc_name:.ssh/authorized_keys2 temp_hmc
    - cat id_rsa.pub >> temp_hmc
    - scp temp_hmc hscroot@hmc_name:.ssh/authorized_keys2
    - Test it: ssh hscroot@hmc_name date

version 2:
    scp id_rsa.pub hmc9:                        <--copy your public key to hmc
    ssh hmc9                                    <--login to hmc (it will ask for a password at this time)
    KEY=`cat id_rsa.pub`                        <--put in the cat command to a variable
    mkauthkeys -a "$KEY"                        <--puts the public key into authorized_keys file
    ssh hmc9                                    <--logint without a password

--------------------------------------

NTP configuration on a HMC

# lshmc -r -Fxntp,xntpserver                 <--check if ntp service is enabled: enable,<ntp_server_name> (/var/log/ntp logfile can be checked as well)
# chhmc -c xntp -s add -a <ntp_server_name>  <--configure ntp service and add ntp server to HMC
# chhmc -c xntp -s enable                    <--activate NTP service

--------------------------------------

HMC - API

HMC API:

API (Application Programming Interface) is an interface (a code or a program) which receives requests and sends responses. For example a weather application on a smartphone sends a request to a remote server and the API (on the remote server) sends back the current temperature. On HMC these things are managed by a web server and they are communicating via HTTP requests. The HMC API provides an additional tool for deploying, managing LPARs, VIOS etc.

It operates on HTTP port 12443 (it is using TLS). In the HTTP requests we need to specify on which element (Managed System, LPAR, VIO, SSP, vSCSI adapter...) which task (GET, PUT, POST and DELETE) is needed. To achieve this a complex URL is used with additional xml files.


Tasks that can be achieved:
GET: list details of a specific resource
PUT: usually used to create a new resource
POST: usually used to modify an already existing resource
DELETE: deletes a resource


The structure of the URL looks like this:
/rest/api/uom/{R}/operations                   <--get the defined job operations for {R}
/rest/api/uom/{R}/{UUID}/do/{OP}               <--get a template for job of type {OP}

/rest/api/uom/{R}/{UUID}/{C}/operations        <--get the defined job operations for {C}
/rest/api/uom/{R}/{UUID}/{C}/{UUID}/do/{OP}    <--get a template for job of type {OP}

{R} - is a Root object (like a Managed System)
{C} - is a Child object of a Root object (like an LPAR on a Managed System)
{UUID} - identifies a specific object (like a specific LPAR, it has this syntax: 31a90881-7523-3f1d-a4bf-6ddf3b8ab7c3)
{OP} - operation we want to do on the specific object (like Power On a Managed System)

for example to power off a Managed System:
/rest/api/uom/ManagedSystem/31a90881-7523-3f1d-a4bf-6ddf3b8ab7c3/do/PowerOff

Checking this pdf will show what should be the correct syntax:
http://ftpmirror.your.org/pub/misc/ftp.software.ibm.com/systems/power/docs/hw/p9/p9ehl.pdf

For every request API sends back a status code. Status codes show the results of the requests:
200 (OK): the request is succeeded and a response is provided
204 (No Content): the request is succeeded and no additional response is provided
400 (Bad Request): there is an error in the requests (probably a syntax error)
403 (Forbidden): not authorized for that specific request
404 (Not Found): the URL resource does not exists or use has no permission to access it
...
...

Steps needed to use HMC API:
1. Authenticate with HMC
2. Sending requests to HMC API

All actions can be done using Python, Java, PHP, C++ etc., but it also works with curl, which is good if someone doesn't want to use programming language environments. In below examples will use curl.

When referring to HMC in curl command it is possible to use:
https://<HMC_IP>/rest/api...
https://<HMC_NAME>/rest/api...
https://<HMC_NAME>:12443/rest/api...

----------------------------------

Authentication with HMC

Authenticating with HMC is the first step. A login.xml file needs to be created (with user and password), which will be used at the first login. If it is successful a trusted session is established and a cookies.txt (jar file) is generated which will be re-used for later requests. (the Logoff request disconnects or closes a session)

1.Create a login.xml file:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LogonRequest xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" schemaVersion="V1_0">
  <UserID>hscroot</UserID>
  <Password>abcd1234</Password>
</LogonRequest>

2.Authenticate with HMC
# curl -k -c cookies.txt -i -X PUT -H "Content-Type: application/vnd.ibm.powervm.web+xml; type=LogonRequest" -H "Accept: application/vnd.ibm.powervm.web+xml; type=LogonResponse" -H "X-Audit-Memento: hmc_test" -d @login.xml https://ls-hmc02.mycompany.vmta/rest/api/web/Logon

-k : “insecure” mode. It permits the use of the self signed certificate used by the HMC.
-c cookies.txt : setup the cookies jar where authentication session informations will be kept.
-X PUT : use PUT request method. Requested by the HMC API.
-d @login.xml : will use content of login.xml.
-H : http headers added to the request.
-i : show the http headers of the answer. Useful for debugging.

Content-Type: application/vnd.ibm.powervm.web+xml; type=LogonRequest    <---specifies the request content format
Accept: application/vnd.ibm.powervm.web+xml; type=LogonResponse :       <---specifies the expected response content format
“X-Audit-Memento: hmc_test”                                             <---a simple comment

Response should look like:
HTTP/1.1 200 OK
Date: Thu, 09 Apr 2020 09:26:12 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
X-Powered-By: Servlet/3.0
X-Transaction-ID: XT10000003
Content-Type: application/vnd.ibm.powervm.web+xml; type=LogonResponse
Content-Language: en-US
Content-Length: 576
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Cache-Control: no-cache="set-cookie, set-cookie2"
Set-Cookie: JSESSIONID=0000HWdO8vEpjtW6ctYAt1rDxlh:08d79c41-17cd-461b-bb48-079a931706b4; Path=/; Secure; HttpOnly
Set-Cookie: CCFWSESSION=0376B4771B26DE69F149E573A4E37751; Path=/; Secure; HttpOnly
Vary: User-Agent
X-XSS-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LogonResponse xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns:ns2="http://www.w3.org/XML/1998/namespace/k2" schemaVersion="V1_7_0">
    <Metadata>
        <Atom/>
    </Metadata>
    <X-API-Session kxe="false" kb="ROR">P-2YNulbgJ-N-Jk9kUKBdtEEcSmny-ah_YJ818uqby0-444kYl0-SD1wdy4TXDWCtR4ToGuZrqMvX3i1q4F_x3LplU5MQdYQFZhQNuTUylJ806tBmgNmHaFlFZkTs4wL-X3bUeywm8SVMkWb-2rZbxPx8vrMeuotorfK2rUMVCDCVbc-k83qlQeKM19p2azo4yNXb8GCG-uEQ1j0G0oazBIjNtFP6gKB1wz-D70sq5o=</X-API-Session>
</LogonResponse>


If authentication is successful we receive a 200 return code and cookies.txt is generated.

----------------------------------

Sending requests to HMC API

After successful authentication API requests can be sent.

List details of resources:
(These examples will list UUID of the spcified resource and beside that version,name etc. is also listed.)
# curl -k -c cookies.txt -b cookies.txt -H "Accept: application/atom+xml; charset=UTF-8" https://myhmc.com/rest/api/uom/ManagementConsole
# curl -k -c cookies.txt -b cookies.txt -H "Accept: application/atom+xml; charset=UTF-8" https://myhmc.com/rest/api/uom/ManagedSystem
# curl -k -c cookies.txt -b cookies.txt -H "Accept: application/atom+xml; charset=UTF-8" https://myhmc.com/rest/api/uom/LogicalPartition

-b cookies.txt: use the appropriate cookies during the session.
-H “Accept: application/atom+xml; charset=UTF-8” : the expected response should be a atom feed.


List possible operations for the specified object (here ManagementConsole):
# curl -k -c cookies.txt -b cookies.txt -H "Accept: application/atom+xml; charset=UTF-8" https://myhmc.com/rest/api/uom/ManagementConsole/operations


To get info from a specific LPAR:
# curl -k -c cookies.txt -b cookies.txt -i -H "Accept: application/atom+xml; charset=UTF-8" https://myhmc.com/rest/api/uom/LogicalPartition/search/\(PartitionName\=\=ld-aix01\)


----------------------------------

Modify a resources

Always perform GET on the resource, copy the output to an xml file, modify the content as desired and then perform POST operation.

For example to change the name of an LPAR:
1. Perform GET: # curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.uom+xml; Type=LogicalPartition" -H "Accept: application/atom+xml; charset=UTF-8" -X GET https://myhmc.com/rest/api/uom/LogicalPartition/{UUID}
2. Copy the output from <LogicalPartition:LogicalPartition> until to </LogicalPartition:LogicalPartition> and save it as an xml file (for example lpar.xml)
3. Edit the xml to modify the <PartitionName> attribute with desired name
4. Perform POST using that xml file: # curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.uom+xml; Type=LogicalPartition" -H "Accept: application/atom+xml; charset=UTF-8" -X POST -H "Expect:" -d @lpar.xml https://myhmc.com/rest/api/uom/LogicalPartition/{UUID}

----------------------------------

Create an LPAR

1. Perform a GET on an existing LPAR:
# curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.uom+xml; Type=LogicalPartition" -H "Accept: application/atom+xml; charset=UTF-8" -X GET https://myhmc.com/rest/api/uom/LogicalPartition/{UUID}

2. Save the output to an xml file (modify values as needed)
<LogicalPartition:LogicalPartition xmlns:LogicalPartition=" http://www.ibm.com/xmlns/systems/power/firmware/uom/mc/2012_10 "http://www.ibm.com/xmlns/systems/power/firmware/uom/mc/2012_10/" />" xmlns:ns2=" http://www.w3.org/XML/1998/namespace/k2 " schemaVersion="V1_8_0">

<PartitionID kb="COD" kxe="false">29</PartitionID>
<PartitionMemoryConfiguration kb="CUD" kxe="false" schemaVersion="V1_8_0">
<Metadata>
<Atom/>
</Metadata>
<DesiredMemory kb="CUD" kxe="false">512</DesiredMemory>
<MaximumMemory kb="CUD" kxe="false">1024</MaximumMemory>
<MinimumMemory kb="CUD" kxe="false">256</MinimumMemory>
</PartitionMemoryConfiguration>
<PartitionName kxe="false" kb="CUR">lpar5</PartitionName>
<PartitionProcessorConfiguration kb="CUD" kxe="false" schemaVersion="V1_8_0">
<Metadata>
<Atom/>
</Metadata>
<DedicatedProcessorConfiguration kb="CUD" kxe="false" schemaVersion="V1_8_0">
<Metadata>
<Atom/>
</Metadata>
<DesiredProcessors kb="CUD" kxe="false">2</DesiredProcessors>
<MaximumProcessors kxe="false" kb="CUD">2</MaximumProcessors>
<MinimumProcessors kb="CUD" kxe="false">2</MinimumProcessors>
</DedicatedProcessorConfiguration>
<HasDedicatedProcessors kxe="false" kb="CUD">true</HasDedicatedProcessors>
<SharingMode kxe="false" kb="CUD">sre idle proces</SharingMode>
</PartitionProcessorConfiguration>
<PartitionType kxe="false" kb="COD">AIX/Linux</PartitionType>
</LogicalPartition:LogicalPartition>


3. Perform PUT with xml file
# curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.uom+xml; Type=LogicalPartition" -H "Accept: application/atom+xml; charset=UTF-8" -X PUT -H "Expect:" -d @newlpar.xml https://myhmc.com/rest/api/uom/ManagedSystem/<ManagedSystem_UUID>/LogicalPartition


----------------------------------

Deleting an LPAR

# curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.web+xml; type=LogicalPartitionProfile" -H "Accept: application/atom+xml; charset=UTF-8" -X DELETE -H "Expect:" https://myhmc.com:12443/rest/api/uom/LogicalPartition/<LPAR_UUID>

----------------------------------

CLIRunner

CLIRunner is an API function which runs HMC commnads using API. Not all commands possible to use to use, some of them are: chhmcusr, lshmc, lshmcusr, hmcshutdown, migrlpar, viosvrcmd ... (viosvrcmd is very useful if we want to run VIOS commands from the HMC)


In below example I will change HMC user description from aaaaa to bbbbb
# lshmcusr --filter "names=sysadmin"
name=sysadmin,taskrole=hmcsuperadmin,description=aaaaa,...


1. create an xml file for CLIRunner (format templates can be found in HMC REST APIs pdf):
<?xml version="1.0" encoding="UTF-8"?>
<JobRequest:JobRequest xmlns:JobRequest="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/" xmlns:ns2="http://www.w3.org/XML/1998/namespace/k2" schemaVersion="V1_2_0">
   <Metadata>
      <Atom />
   </Metadata>
   <RequestedOperation kxe="false" kb="CUR" schemaVersion="V1_2_0">
      <Metadata>
         <Atom />
      </Metadata>
      <OperationName kxe="false" kb="ROR">CLIRunner</OperationName>
      <GroupName kxe="false" kb="ROR">ManagementConsole</GroupName>
   </RequestedOperation>
   <JobParameters kxe="false" kb="CUR" schemaVersion="V1_2_0">
      <Metadata>
         <Atom />
      </Metadata>
      <JobParameter schemaVersion="V1_0">
         <Metadata>
            <Atom />
         </Metadata>
         <ParameterName kxe="false" kb="ROR">cmd</ParameterName>
         <ParameterValue kxe="false" kb="CUR">chhmcusr -u sysadmin -t desc -v bbbbb</ParameterValue>
      </JobParameter>
      <JobParameter schemaVersion="V1_0">
         <Metadata>
            <Atom />
         </Metadata>
         <ParameterName kxe="false" kb="ROR">acknowledgeThisAPIMayGoAwayInTheFuture</ParameterName>
         <ParameterValue kxe="false" kb="CUR">true</ParameterValue>
      </JobParameter>
   </JobParameters>
</JobRequest:JobRequest>

(I had to change schemaversion from V1_2 to V1_2_0 otherwise I recived HTTP 400 Bad Request)

2. run CLIRunner:
curl -k -c cookies.txt -b cookies.txt -i -H "Content-Type: application/vnd.ibm.powervm.web+xml; Type=JobRequest" -H "Accept: application/atom+xml; charset=UTF-8" -X PUT -H "Expect:" -d @chhmcusr.xml https://192.168.0.9/rest/api/uom/ManagementConsole/31a90881-7523-3f1d-a4bf-6ddf3b8ab7c3/do/CLIRunner

HMC UUID: the unique identifier of HMC is shown if we list details of the ManagementConsole (curl ... https://myhmc.com/rest/api/uom/ManagementConsole)
@chhmcusr.xml: this is the file above we saved with the chhmcusr command


After successful run:
# lshmcusr --filter "names=sysadmin"
name=sysadmin,taskrole=hmcsuperadmin,description=bbbbb

----------------------------------

Additional reading: https://www.djouxtech.net/posts/hmc-v8-rest-api-part-1-curl/




HMC - UPDATE, UPGRADE

HMC UPDATE - HMC UPGRADE
(Remote HMC update/upgrade using network images.)



HMC UPDATE:
(apply fixes, service packs, for example MH01601)

# tail -f /tmp/HmcInstall.log                                    <--during upd. (updhmc) installation log entries are logged here
# ls -l /dump/efix                                               <--during upd. image from ftp server first will be copied here before install


1. Read Documentation (readme) of fix + download iso image
# csum -h SHA1 MH01601.iso                                       <--checksum on iso files, compare values with the value on above readme


2. Save config, backup and reboot
# lshmc -v; lshmc -V; lshmc -n; lshmc -r; monhmc -r disk         <--save config outputs and check fs for enough space (monhmc is like df)
# bkconsdata -r ftp -h <ftp host> -u <user> -d <directory>       <--do a backup (it can take longer, in GUI: Backup Management Console Data)
# hmcshutdown -t now -r                                          <--reboot HMC (to see if it comes back)


3. Update
# updhmc -t s -h <servername> -f <location of iso> -u <user> -i  <--do update (in GUI: Updates --> Update HMC)
                                                                (-t s: type to update from is server (s), -i: prompt for password)

4. create a new backup


-------------------------------


HMC UPGRADE:
(for example from 8.8.3 to 8.8.4)

# ls -l /mnt/upgrade                                             <--during saveupgdata (file "doRestore" triggers automat. upgr. after reboot)
# ls -l /hmcdump                                                 <--during getupgfiles (check for files and correct sizes)
                                                                 (from 8.8.3 to 8.8.4 only 1 file was copied, but usually all of them needed)

1. Read documentation (HMC models, fixes, sytem firmwares..) + download network images
download link: http://www14.software.ibm.com/webapp/set2/sas/f/netinstall/home.html
# sum bzImage initrd.gz disk1.img disk2.img disk3.img            <--do checksum on downloaded files, compare result: cat hmcnetworkfiles.sum
                                                                 (hmcnetworkfiles.sum contains checksum values)

2. Save profile data of each Man. Sys      
# BK_DATE="`date '+%Y%m%d'`"; export BK_DATE                     <--initialize variable of backup date variable (for correct file name)
# for MSYS in `lssyscfg -r sys -F name`; do bkprofdata -m ${MSYS} -f ${MSYS}_${BK_DATE}.prof --force; done <--saving profile data
                                                                 (in GUI: Sys. Man. --> Server --> Config. --> Man. Part. Data --> Backup)

3. Save config, backup HMC and reboot
# lshmc -v; lshmc -V; lshmc -n; lshmc -r                         <--save config outputs
# bkconsdata -r ftp -h <ftp host> -u <user> -d <directory>       <--do a backup (it can take longer, in GUI: Backup Management Console Data)                                                            
# chsvcevent -o closeall                                         <---close open service events
# chhmcfs -o f -d 0                                              <---clear old archived diagnostic logs
# hmcshutdown -t now -r                                          <--reboot HMC (to see if it comes back)


4. Save Upgrade Data
# saveupgdata -r disk                                            <--save config data (it mounts /mnt/upgrade, saves conf., then umounts it)  
(This data is used during next boot, do not reboot HMC after this has been done, or you have to do this step again. It should take few mins.)


5. FTP network images to HMC
# getupgfiles -h <ftp host> -u <user> -d <directory>             <--copy network images from ftp server to HMC (ls -l /hmcdump shows progress)
(After ftp shell gives back command prompt, /hmcdump will be empty again, as files were placed to a special location for alternate boot.)


6. Set HMC to boot from alternate disk
# chhmc -c altdiskboot -s enable --mode upgrade                  <--set alternate boot
# lshmc -r                                                       <--check if alternate boot is enabled
                                                                (..,netboot=disable,altdiskboot=enable,...)
# hmcshutdown -t now -r                                          <--reboot (it boots from alternate disk partition, upgr. can take an hour.)


7. Create a backup


-------------------------------

FTP connection between HMC and FTP server

FTP connection will be needed between HMC and FTP server. FTP server is where the install images are stored (it can be a NIM server.)

If you want to use remote FTP installs on HMC, an FTP user will be needed on the remote (AIX) server.
There is a default script on AIX which creates an ftp or anonymous user for this FTP connection: /usr/samples/tcpip/anon.ftp
IBM officially says this scripts should be used to create FTP user, otherwise connection will work but files may won’t be reachable by HMC, but actually I created in the past FTP users with other method and it worked well.

FTP check from HMC:
- in GUI try to do an update operation, and choose FTP, or do an HMC backup (just it could take long)
- try sendfile command on HMC, which transfer files with ftp to another server
  sendfile -f id_dsa.pub -h 10.10.10.1 -d /nim/hmcftp -u hmcftp


-------------------------------

The below post was my original description about HMC upgrades, and I keep it for reference, but the method described above is more up to date.

HMC UGRADE (v6.1.3 -> v7.3.5)
(this can be done as hmc user)

1. check prerequisites:
    - hmc device supports new hmc version
    - system firmware on supported levels (http://www-933.ibm.com/support/fixcentral/firmware/supportedCombinations)
    - Back up critical console data (to an FTP server is OK)

2. some fixes have to be installed prior to the upgrade:
MH01082, MH01110, MH01128 and after that MH01127 (this will enable upgrade to v7 in 1 step)

    - download zip files and store them on an FTP server
    - on HMC: Licensed Internal Code upd. -> HMC Code Update -> Inst. Corr. Serv. -> Download from remote ... (give absolute path to file)
    - hmcshutdown -t now -r            <--reboot HMC (it took for me 10-15 minutes)
    - lshmc -V                         <--verify update successfully installed


3. upgrade to v7.3.5:
    - download necessary install images to an FTP server
    - chsvcevent -o closeall           <--closes all serviceable events on the HMC to clean up before upgrade (for me did not work)
    - chhmcfs -o f -d 0                <--clearing out old logfiles to clean up before upgrade (for me did not work)
    - hmcshutdown -t now -r            <--make sure there are no problems with reboot (if we did this in step 2 we can omit here)

    - saveupgdata -r disk              <--it mounts /mnt/upgrade, save config. data then unmounts it (it took less than a minute)
                                       (this is only configuration data, which will be used during reboot after the upgrade)
    - getupgfiles -h <ftp server> -u <user> --passwd <paswd> -d <dir on ftp server>    <--mounts /hmcdump, copy install files there, umount
    - chhmc -c altdiskboot -s enable --mode upgrade        <--it sets HMC to boot from the install images and allow the upgrade to proceed
    - hmcshutdown -t now -r            <--it took 30-40 minutes for me
    - lshmc -V                         <--it will show new version

4. some fixes are available to 7.3.5 which can be installed:
    - download iso images to FTP server
    (first HMC_Update_V7R350_SP3.iso has to be installed after that MH01259.iso)

    - install:
        from command line:
        updhmc -t s -h <ftp server> -u <user> -p <passwd> -f /software/server/hmc/fixes/MH01195.iso -r
 
        from GUI:
        Updates -> Update HMC -> next, next ...

    - hmcshutdown -t now -r            <--reboot HMC for the changes to be available (after each fix install)

5. create backup

--------------------------------------------

HMC -DPO

DPO:

Managed Systems can become suboptimal in terms of processor and memory affinity when reassigning or moving resources. For example DLPAR operations, LPM  and LPAR creation/deletion can cause that some CPU and RAM assignemts of an LPAR are in different chips or CECs, so reaching out to these resources takes longer.

Memory locations can be:
- Local: Memory that is directly attached to the processor of the LPAR (same chip)
- Near: Memory that is within the same node/drawer (CEC) as the processor using it (other chip, but same CEC)
- Far: Memory in other nodes/drawers (CEC) (other CEC)

Local is the best and that is the aim of DPO.

In smaller systems (S922, S924, E950) there is no near, if it is not local memory access then it will go through a "far" path".  Near memory access has a meaning on servers with multiple CECs (like on E980 near means other chip but same CEC drawer)


The optimal situation is when an LPAR can fit into a CPU configuration where memory access is "local". However with bigger LPARs or after many LPMs  memory access can be degraded. Enhanced feature of the mpstat -d command can be used to display statistics on local, near and far memory accesses for every logical CPU in an LPAR, as presented here:


-----------------------------------

lssrad:

lssrad command displays how CPU/RAM assigned to an LPAR. (An SRAD is a group of CPU and its RAM which is directly attached to it.)

# lssrad -av
REF1   SRAD        MEM      CPU
0
          0   40050.38      0-71
          1   54281.25

(Here we have 2 SRADs (0,1) with RAM in both SRADs, but CPU is assgned only to SRAD 0. It means CPU has to reach sometimes SRAD1 for some memory, which is not optimal. DPO can rearrange this layaout.)

-----------------------------------

The Dynamic Platform Optimizer (DPO) is a PowerVM virtualization feature that improves partition memory and processor placement (affinity) on
Power servers. DPO determines an optimal RAM and CPU placement for the server based on the partition configuration and hardware topology. After that it performs memory and processor relocations to reach optimal layout. This process occurs dynamically while the partitions are running.

DPO related commands are issued in HMC command line and these  provides an affinity score to determine how optimal the system affinity is and what potential affinity can be achieved, after that optimization can occur.

Requirements:
- Power Systems Firmware level 760 or later
- HMC version 7.6.0 or later
- License code for DPO. (HMC GUI -> properties of the managed system)
- At least one free Logical Memory Block (LMB) on the syst
- A DPO aware AIX is preferable (but not a requirement), so no reboot is needed after optimization (AIX 6.1 TL8/7.1 TL2 or later versions)

-----------------------------------

Check affinity (lsmemopt):

lsmemopt can list the current affinity score, and can calculate the potential affinity score which could be attained after running DPO. A system's affinity score shows how good the processor-memory affinity on the system. 0 representing the worst affinity and 100 representing perfect affinity.

You can list explicitly which partition to optimize, or you can list which partition to exclude from optimization. If neither is used all partitions are considered to optimization.

lsmemopt -m <system_name> -o currscore                        <--shows affinity score of managed system
lsmemopt -m <system_name> -o calcscore                        <--shows potential affinity score after running DPO (this is a prediction)
                                                              (these commands are safe to use, these are doing no changes on the system)

-----------------------------------

Optimization with  DPO (optmem):

While an optimization is running, the performance of the partitions are degraded (memory of the partition has to be relocated.) A system with busy processors takes longer to optimize, because the background optimizer tries to minimize its effect on running partitions. (Partitions that are powered off can be optimized quickly because the contents of their memory do not need to be maintained.)

Impacted partitions can have up to 20% of performance degradation while partition memory is relocated. (However, partitions that are explicitly protected (listed as excluded partitions) from DPO are not affected.)

1. lshwres -m <system_name> -r mem --level sys -F curr_avail_sys_mem    <--shows available memory (at least 1 LMB size is needed)
2. optmem -m <system_name> -t affinity -o start                         <--starts optimization for all LPARs on the entire system
3. lsmemopt -m <system_name>                                            <--shows progress status as percentage completed (progress=2 means 2%)
4. lsmemopt -m <system_name> -o currscore                               <--check affinity score after DPO completes

If you want to specify/exclude some LPARS:
optmem -m sys -o start -t affinity -p lp1,lp3                           <--start DPO only on LPAR lp1 and lp3
optmem -m sys -o start -t affinity --id 2,4,6-9,11 --xid 1              <--start DPO with partition ID 2,4,6 - 9, and 11, while excluding ID 1

optmem -m sys -o stop                                                   <--stop a DPO operation

-----------------------------------

Some considerations: If (after 'Power On' of a Managed System), you start VIOSes first and then the high important, critical LPARs then these will get the best resource locations. These probably can be omitted from optimization. It can be a good idea to check 'calcscore' from crontab every week and then send the result via email. It is probably not a good idea to automatically run an optimization on this by crontab.

HMC - POWER INSTALL

POWER - Install, Config

NEW POWER SERVER INSTALL:
After the server arrived to the datacenter (or serverroom) and it has been put into the rack, the next step is to connect to the HMC, so we can do configuration tasks on this POWER machine from the HMC.

If the server is not powered on, plug it into the power source. Wait for the control panel to display 01 (a series of progress codes are shown before 01 appears). The system is powered on if the light on the control panel is green. (To view the control panel, press the blue switch to the left, then pull out the control panel all the way, and then pull it down.)

!!! If you will not use HMC, do not connect Ethernet cables to HMC1 or HMC2 ports as the default IP adress of the service processor could be overwirtten!!!
(more details about this in ASMI page)


1. Cabling the service processor to the HMC
If you will use this server with HMC then cabling is needed between HMC and the service processor (HMC1 or HMC2 ports on the server).


This is a private network between the HMC and the server (blue line), and a small non-routed VLAN should be enough.

The service processor's Ethernet ports (HMC1 and HMC2) are configured for DHCP by default. If the service processor is attached to a live Ethernet network equipped with a DHCP server and the service processor is turned on (the server is powered on), an IP address is assigned and the default IP address of the service processor is no longer valid. HMC has the capability to behave as a DHCP server. If we configure network ranges on HMC for DHCP, an IP address from that network will be assigned to the service processor, when a cable is plugged in to one of the HMC ports.

The HMC uses an SSL connection with port 30000 and 30001 of the service processor’s Ethernet port.

2. Adding server as a Managed System in HMC
If cabling is completed we can add this server as a Managed System in HMC. Choose "Connect Systems"


Add IP or an IP range and when the server is found it can be added to HMC.

If you do not choose password at that time, authentication will pending on pw change, so HMC connection pw, ASM general pw and ASM admin pw need to be set.

After that connection should be OK, while server is in Powered Off state


3. Configuring Managed System in HMC

By default there is a Service partition on the system with a preinstalled AIX, which can be used to check adapters, devices, WWPNs ... if needed. If Service Partition is not needed anymore,  we can setup server for real use.

These are being configured:
- Change server name
- Change Service partition to Unassigned (maybe server needs to be in standby state)
- Allow Migration with Inactive Source Storage VIOS
- Make sure Power-On side is on Temporary (otherwise concurrent firmware update is not possible later)
- Do Firmware upgrade if needed

Some tasks can be done while system is in Power Off state, some tasks can be done after Power On, while system is in standby state.


After that we can start with VIOS creation, installation ...

--------------------------------------

The open failed.
The session is reserved for physical serial port communication.

It is caused by the server is still in Manufacturing Default Configuration (MDC) mode.

Make sure server is in standby state, then when activating the default LPAR choose exit the MDC:
1. Power off managed system:
2. Partition start policy:  change from auto start --> User-Initiated
3. Power on:  it will go to standby state
4. Partition: Activate Profile --> choose "exit MDC"

--------------------------------------

HSCL3396 Failed to delete the partition with ID 1 since it is the service partition

Service partition cannot be deleted until it is configured to be as a Service Partition.
In the Properties of the Managed System, Service partition should be "Unassigned".

--------------------------------------

Finding FSP IP using the front panel:

The Operator panel has three blue buttons, the outer ones are up/down and the middle one is "enter".
To change from menu 01 to menu 02, press the up button and then the middle one.

Operator Panel menu 02, change N(normal) to M(Manual) - to do that, press the enter button until the cursor > moves next to the N character. Then press up, then press enter again until you get back to 02.

When you use the "up" button, you will see more menu items, keep pressing up until 30. Select option #30 and press enter. You should see "30**" on the screen, and if you press enter again it goes to “3000”.
(3000 represents eth0 and 3001 eth1 of the FSP) Press enter again, and the IP address of the ASMI should be presented on the Control Panel display.

Operator Panel menu 02, change M back to N(Normal)
WARNING It is very important to return this mode back to N(Normal)
Operator panel 01, press enter. To return the display to its normal condition.


--------------------------------------

Reconnect a managed system to the HMC:

If your managed system is connected to the HMC, and you disconnect it, before reconnecting it, clear the connection history
on the HMC:
# mksysconn -o auto

If you reconnect the managed system to the HMC without first using this command, HMC will not recognize the Managed System.

--------------------------------------

HMC - RMC


RMC (Resource Monitoring and Control):

RMC is a distributed framework and architecture that allows the HMC to communicate with a managed logical partition. RMC daemons should be running on AIX partition in order to be able to do DLPAR operations on HMC.

For example "Dynamic LPAR Resource Manager" is an RMC daemon that runs inside the AIX (and VIO server). The HMC uses this capability to remotely execute partition specific commands.

The daemons in the LPARs and the daemons on the HMC must be able to communicate to the AIX partition through an external network not through the Service Processor. An external network that the partition has access to and the HMC has acces to as well.

For example, if HMC has a connection to a 9.x.x.x network and I put my AIX partition to that 9.x.x.x network and as long there is network connectivity (HMC is allowed to communicate to that partition over that network) and RMC daemon is running on the partition, then DLPAR operations are available.

In order for RMC to work, port 657 upd/tcp must be open in both directions between the HMC public interface and the lpar.

The RMC daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC!

----------------------------------------

HMC and LPAR authentication (RSCT authentication)
(RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.)

Authentication is the process of ensuring that another party is who it claims to be.
Authorization is the process by which a cluster software component grants or denies resources based on certain criteria.
The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.


The RSCT authorization process in detail:
1. On the HMC: DMSRM pushes down a secret key and HMC IP address to NVRAM where AIX LPAR exists.

2. On the AIX LPAR: CSMAgentRM, reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect new HMCs.

3. On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource then it creates a ManagementServer resource on AIX.

4. On the AIX LPAR: After the creation of these resources on the HMC and AIX, CSMAgentRM grants HMC permission to access necessary resources on the LPAR and changes its ManagedNode Status to 1 on the HMC.

5. On the HMC: After the ManagedNode Status is changed to 1, a session is established with the LPAR to query operating system information and DLPAR capabilities, and then waits for the DLPAR commands from users.

----------------------------------------

RMC Domain Status

When partitions have active RMC connections, they become managed nodes in a Management Domain. The HMC is then the Management Control Point (MCP) of that Management Domain. You can then use the rmcdomainstatus command to check the status of those managed nodes (i.e. your partitions).

As root on the HMC or on the AIX LPAR you can execute the rmcdomainstatus command as follows:

# /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc

From HMC: You should get a list of all the partitions that the HMC server can reach on the public network on port 657.

Management Domain Status: Managed Nodes
  O a  0xc8bc2c9647c1cef3  0003  9.2.5.241
  I a  0x96586cb4b5fc641c  0002  9.2.5.33


From LPAR: You should get a list of all the Management Control Points

Management Domain Status: Management Control Points
   I A  0xef889c809d9617c7 0001  9.57.24.139


1. column:
-I: Indicates that the partition is "Up" as determined by the RMC heartbeat mechanism (i.e. an active RMC connection exists).
-O: Indicates that the RMC connection is "Down", as determined by the RMC heartbeat mechanism.

2. column:
-A: Indicates that there are no messages queued to the specified node
-a: Same as A, but the specified node is executing a version of the RMC daemon that is at a lower code level than the local RMC daemon.

more info: https://www-304.ibm.com/support/docview.wss?uid=isg3T1011508

----------------------------------------

If rmcdomainstatus shows "i" at (1st column):

Indicates that the partition is Pending Up. Communication has been established, but the initial handshake between two RMC daemons has not been completed (message authentication is most likely failing.)
Authentication problems will occur when the partition identity do not match each other's trusted host list:

# /usr/sbin/rsct/bin/ctsvhbal        <--list the current identities for the HMC and the logical partition (run this command on both)
# /usr/sbin/rsct/bin/ctsthl -l       <--list the trusted host list on the partition

On the HMC, there is an entry for the partition. On the partition, there is an entry for the HMC. The HOST_IDENTITY value must match one of the identities listed in the respective ctsvhbal command output.

----------------------------------------

Things to check at the HMC:

- checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc  (you must be root on the HMC)

- checking connection between HMC and LPAR:
hscroot@hmc10:~> lspartition -dlpar
<#0> Partition:<2*8204-E8A*0680E12 aix10.domain.com, 10.10.50.18>
       Active:<1>, OS:<AIX, 6.1, 6100-03-01-0921>, DCaps:<0x4f9f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<1452>
<#1> Partition:<4*8204-E8A*0680E32 aix20.domain.com, 10.10.50.71>
       Active:<0>, OS:<AIX, 6.1, 6100-04-05-1015>, DCaps:<0x0>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<656>

For correct DLPAR function:
- the partition must return with the correct IP of the lpar.
- the active value (Active:...) must be higher than zero,
- the decaps value (DCaps:...) must be higher 0x0

(The first line shows a DLPAR capable LPAR, the second line is anon-working LPAR)

- another way to check RMC connection: lssyscfg -r lpar -F lpar_id,name,state,rmc_state,rmc_ipaddr -m p750
(It should list "active" for the LPARs with active RMC connection.)


----------------------------------------

Things to check at the LPAR:

- checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc

- Checking RMC status:
# lssrc -a | grep rsct
 ctrmc            rsct             8847376      active          <--it is a RMC subsystem
 IBM.DRM          rsct_rm          6684802      active          <--it is for executing the DLPAR command on the partition
 IBM.DMSRM        rsct_rm          7929940      active          <--it is for tracking statuses of partitions
 IBM.ServiceRM    rsct_rm          10223780     active
 IBM.CSMAgentRM   rsct_rm          4915254      active          <--it is for  handshaking between the partition and HMC
 ctcas            rsct                          inoperative     <--it is for security verification
 IBM.ERRM         rsct_rm                       inoperative
 IBM.AuditRM      rsct_rm                       inoperative
 IBM.LPRM         rsct_rm                       inoperative
 IBM.HostRM       rsct_rm                       inoperative     <--it is for obtaining OS information

You will see some active and some missing (The key for DLPAR is the IBM.DRM)

- Stopping and starting RMC without erasing configuration:

# /usr/sbin/rsct/bin/rmcctrl -z    <--it stops the daemons
# /usr/sbin/rsct/bin/rmcctrl -A    <--adds entry to /etc/inittab and it starts the daemons
# /usr/sbin/rsct/bin/rmcctrl -p    <--enables the daemons for remote client connections

(This is the correct method to stop and start RMC without erasing the configuration.)
Do not use stopsrc and startsrc for these daemons; use the rmcctrl commands instead!

- recfgct: deletes the RMC database, does a discovery, and recreates the RMC configuration
# /usr/sbin/rsct/install/bin/recfgct
(Wait several minutes)
# lssrc -a | grep rsct

(If you see IBM.DRM active, then you have probably resolved the issue)

- lsrsrc "IBM.ManagementServer"    <--it shows HMCs via RMC

EXTRA - ISD

IBM Systems Director:



Systems Director enables monitoring and event management from a single browser-based user interface. IBM Systems Director provides broad platform coverage: Microsoft Windows, Linux, AIX, IBM i...

From one access point, users can monitor system environmentals, resources, inventory, events, task management, corrective actions, distributed commands, and hardware control for both servers and storage.

----------------------------------------

IBM Systems Director installation on AIX

1. Make sure X window (GUI) is working (don't forget to start an X server):
root@bb_lpar: / # xclock


2. Install a web browser (Firefox):
-Before Firefox install some rpms must be installed as well (check readme of firefox install), these can be installed like this:
root@bb_lpar: /bb # rpm -Uvh --force --nodeps *.rpm        <--in the directory where all needed rpms exist (I did not install dependencies)

root@bb_lpar: /bb # /usr/sbin/updtvpkg                     <--refreshes what is installed on the system (maybe it is not needed at all)
Please wait...

-After needed rpm have been installed, Firefox is installed:
root@bb_lpar: /bb # smitty install
3.5.13.1  Firefox Web Browser                              <--only this package has been chosen to be installed           

-Check if Firefox is running correctly:
root@bb_lpar: /bb # /usr/bin/firefox


3. Install Systems Director:

-After unzipping it:
root@bb_lpar: /bb # smitty install
DirectorServer6.2.0.0                                         <--my xlC, openssl was up to date, so I installed only this

-Installation has ben started and I received this output:
....
Starting server runtime...
The server runtime has been started.
Attempting to install features.    
Connection to localhost closed by remote host. Connection to bb_lpar closed.    <--during install, memory got full and I was logged out

-I increased lpar memory to 2.5GB and restarted the installation, when I received this:
...
Starting server runtime...
The server runtime is taking too long to start. Installer Exiting...
install:  Failed while executing the ./DirectorServer.post_i script.            <--this is becuase in DirectorServer.post_i is a syntax error


4. Workaround for the DirectorServer.post_i script problem:
(http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14304597)

-start installation again (smitty install) and check /usr/lpp/DirectorServer dir for the script: DirectorServer.post_i
root@bb_lpar: /usr/lpp/DirectorServer # ls -l
total 22264
-rw-r--r--    1 root     system           21 Nov 13 00:49 .cr.status
-rw-r--r--    1 root     system      3076263 Nov 13 00:49 DirectorServer.al
-rw-r--r--    1 root     system         9362 Nov 13 00:49 DirectorServer.config
-rw-r--r--    1 root     system      7969210 Nov 13 00:49 DirectorServer.inventory
-rw-r--r--    1 root     system        32258 Nov 13 00:49 DirectorServer.post_i

-when the file is there (maybe it will pop up later, not right away when you start installation), copy it to a temp. place
(You can only copy this file when an installation is occurring because when the install fails it removes the file. )

root@bb_lpar: /usr/lpp/DirectorServer # cp DirectorServer.post_i /tmp
root@bb_lpar: /usr/lpp/DirectorServer # cd /tmp
root@bb_lpar: /tmp # vi DirectorServer.post_i

-in the "CreateTWGServerFile" function a ";" is missing in the "if" statement and comment out the las line (twgserver.sh)
CreateTWGServerFile ( )
{
...
   if [ ! -d $TWGSERVER_ETC ]                               <--I added the ";" symbol here
   then
      mkdir -p $TWGSERVER_ETC
   fi
...

#   . $TWGSERVER_BIN/twgserver.sh                            <--I added "#" at the beginning of this line

-start installation again, and when DirectorServer.post_i script appears, overwrite it with our edited script.


5. Configuration before starting up IBM SystemDirector:

After installation is finished run this:
root@bb_lpar: / # /opt/ibm/director/bin/configAgtMgr.sh            <--you can give any ID and pw (not necessary an already existing user)

After that you can start up this:
root@bb_lpar: / # /opt/ibm/director/bin/smstart
Starting IBM Director...
The starting process may take a while. Please use smstatus to check if the server is active.

During startup you can check status:
root@bb_lpar: / # /opt/ibm/director/bin/smstatus -r
Starting                                                           <--first only this line was there           
Active                                                             <--later popped up this (I could log out from here with ctrl+c)


6. Logging in into Systems Director:

Start up Firefox and use any of the below commands for login:
http://bb_lpar:8421/ibm/console                                    <--for login I used the user "root" and his pw
https://bb_lpar:8422/ibm/console


----------------------------------------









HMC - CLI

HMC command line:

The 2 mostly used commands to gather info about managed systems and lpars:

lssyscfg: info about managed system, lpar (i.e. running or not) or profile configuration
lshwres: info about the resources (cpu/mem/vscsi..) of a managed system or lpar

------------------------------------------------------------
------------------------------------------------------------

lssyscfg

Lists managed-systems, partitions, partition profiles and attibutes of them. When you give resource type you can choose from: sys, lpar or prof.

If sys or lpar has been chosen it does not list resources like cpu/mem settings or vscsi/vfc adapter config, only general info about the system/lpar, like lpar is activated or not, what is the cpu mode, or rmc state...

If prof is chosen, it lists cpu,mem,vscsi... config from the lpar profile. (lshwres command shows these data from the running partition, not from the profile.)


lssyscfg -m <sys> -r <res> --filter "<name>=<value>,<value>" -F <attr>,<attr> --header

-m name of the managed system (if omitted all managed systems will be listed)

-r resource type
     sys gives general indo about managed system
     lpar gives general info about lpar
     prof gives specific infos (cpu, mem, vscsi..) from the lpar profile

--filter select values for a specific resource only (lpar_ids, lpar_names and profile_names)
   "lpar_ids=...,..."
   "lpar_names=...,."
   "profile_names=.."

-F display only the given attributes (name of the attr. will not be displayed and delimeter can be specified)

--header show header of the displayed values

------------------------------------------------------------

lssyscfg -r sys -F name lists name of managed systems (this name can be used in other commands)

lssyscfg -m <sys> -r lpar lists general info of lpars on the managed sys.
lssyscfg -m <sys> -r prof lists all settings in the lpar profiles on the managed sys.

other example:
lssyscfg -m <sys> -r lpar -F name,state --header                                  lists lpars and if it is activated or not
lssyscfg -m <sys> -r prof -F lpar_name,virtual_eth_adapters                        lists lpars with eth. adapters (vlan tags will be shown as well)
lssyscfg -m <sys> -r prof --filter "lpar_names=<lpar>" -F name,virtual_fc_adapters lists vfc adapters (with wwpn) of an lpar


------------------------------------------------------------
------------------------------------------------------------

lshwres:

Lists hardware resources of the managed-system and lpars, including physical I/O, virtual I/O, cpu/mem settings, (shared) processor pool, etc.

lshwres -m <sys> -r <res> --rsubtype <subtype> --level <lpar/sys> --filter "<name>=<value>,<value>" -F <attr>,<attr> --header

-m name of the managed system (lssyscfg -r sys -F name)

-r resource type
     io      physical io
     mem memory
     proc    processing unit
     virtualio virtual io

--rsubtype subtype of the resource given previously (it works for io and virualio)
     io       slot, slotchildresn (these can be used for the resource 'io')
     virtualio eth, fc, scsi, vswitch (these can be used for the resource 'virtualio')

--level      display info of managed system or lpar (it works for mem, proc and virtualio)
     lpar info is taken from lpar level (i.e memory values of the lpars)
     sys info is taken from system level (i.e memory value of the managed system)

--filter select values for a specific resource only (not all value can be used as filter)
   "lpar_ids=...,..."
   "lpar_names=...,."
   "vlans=...,...,.."

-F display given attributes (the name of the attribute will not be displayed and delimeter can be specified)

--header show header of the displayed values

------------------------------------------------------------

Physical I/O (io):
lshwres -m <sys> -r io --rsubtype slot/slotchildren shows io slot information (or children devices of slots)


MEM/PROC (mem/proc):
lshwres -m <sys> -r mem/proc --level lpar/sys shows mem/proc info of all lpars or the managed system


VIRTUAL I/O (virtualio):
lshwres -m <sys> -r virtualio --rsubtype vswitch shows virtual switches with configured vlan ids on them
lshwres -m <sys> -r virtualio --rsubtype eth --level lpar   shows virt. ethernet adapters with lpar_name,slot_id, vlan_id....
lshwres -m <sys> -r virtualio --rsubtype fc --level lpar    shows virt. fc adapters with lpar_name,slot_id, wwpns...
lshwres -m <sys> -r virtualio --rsubtype scsi --level lpar shows vscsi adapters with lpar_name,slot_id...


SR-IOV (sriov):
lshwres -m <sys> -r sriov --rsubtype physport --level ethc lists physical port config: location, speed, mtu size ...
lshwres -m <sys> -r sriov --rsubtype logport --level eth   lists logical port config: capacity, mac address, vlans ...
chhwres -m <sys> -r sriov --rsubtype physport -o s -a “adapter_id=1,phys_port_id=1,conn_speed=10000”  change phys. port config


other example:
lshwres -m <sys> -r virtualio --rsubtype eth --level lpar --filter "lpar_names=<lpar>" -F slot_num:addl_vlan_ids  shows lpar virt. adapters and vlans

------------------------------------------------------------
------------------------------------------------------------

Other commands:

TAB-TAB          shows all commands available for the user (commands have --help option and man pages)

chsysstate -m <sys> -o standby -r sys    power on a system to standby
chsysstate -m <sys> -r sys  -o off                                 normal power off the managed system
chsysstate -m <sys> -r sys -o off --immed                           fast power off the managed system

chsysstate -m <sys> -r lpar -n <lpar> -o shutdown --restart          it will reboot an lpar with dump
chsysstate -m <sys> -r lpar -n <lpar> -o shutdown --immed --restart  it will reboot an lpar immediately (without dump)
chsysstate -m <sys> -r lpar -n <lpar> -o shutdown --immed    it will shutdown the operating system
chsysstate -m <sys> -r lpar -n <lpar> -o on -f default               it will activate an lpar

migrlpar  perform partition migrations
lslparmigr  lists info about lpar migration
lslparmigr -m <sys> -r lpar                                       list lpars and migration state (not migrating or migrating..)
lslparmigr -r sys -m <managed system> | sed "s/,/\n/g"               shows how many concurrent migr. is possible

lsnportlogin -m sys> --filter "lpar_names=<lpar>"                    list all npiv (wwpn) info of an lpar
lsnportlogin -m sys> --filter "lpar_names=<lpar>" -F wwpn  lists only wwpn of given lpar

lspartition -dlpar  shows dlpar capable partitions
 (it will show if RMC connection is OK between the HMC and LPAR)

lssacfg -t email  shows email notification settings

lssysconn -r all  to see what IPs are assigned by the HMC
mksysconn -o auto  tells HMC to re-discover all servers' IP address

lssvcevents -t console -d 60  lists console events in the past 60 days
lssvcevents -t hardware -d 0  list serviceable events which occured today
lssvcevents -t console -d 300 | grep DLPAR  list DLPAR operations of last 300 days (you can grep more to mem or proc)
lssvcevents -t console -d 7 | grep Migration  list last 7 days LPMs

lshmc -V  hmc version
lshmv -v  hmc model
lshmc -n  lists the network settings of the hmc
hmcshutdown -t now -r  reboot the HMC (-t: timing in minutes, -r: reboot)

lslogon -r webui -u  lists users logged in gui (lslogon -r webui -t <--lists running tasks)
lslogon -r ssh -u  lists users logged in cli (lslogon -r ssh -t <--lists running tasks)
termtask -r { webgui | ssh } -s (session) -t (task)  terminate a running task (or all tasks)

monhmc -r mem -n 0  shows total, used and free mamory of HMC
monhmc -r disk -n 0  shows filesystems and usage info (same as "df -k")
monhmc -r proc -n 0  shows cpu usage of each processor
monhmc -r swap -n 0  shows paging space usage

sendfile  transfer files with ftp from HMC to another server

vtmenu  hmc menu options (console session can be opened as well)
logout: ~~. (2 tilde and a dot)


for i in `lssyscfg -r sys -F name`;do echo $i;lssyscfg -r lpar -m $i -F name|grep <LPARNAME>;echo;done    <--shows man. system of given LPAR
for i in `lssyscfg -r sys -F name`;do lssyscfg -r prof -m $i -F lpar_name,virtual_eth_adapters | grep vio | grep <VLAN>;echo; done <--shows spec. VLAN
for i in `lssyscfg -r sys -F name` ; do echo $i; lssyscfg -r prof -m $i -F name,lpar_name --header; done  <--show all lpar profiles

for sys in `lssyscfg -r sys -F name`; do for vio in `lssyscfg -r lpar -m $sys -F name,lpar_env | grep vioserver | cut -f 1 -d, | sort`; do echo $vio; viosvrcmd -m $sys -p $vio -c errlog; done; done                     <--runs a command (here errlog) on VIOS servers
------------------------------------

Default root password: passw0rd (try su -)

------------------------------------

/usr/samples/tcpip/anon.ftp

If you want to use remote FTP installs on HMC, an FTP user will be needed on the remote (AIX) server.
There is a default script on AIX which creates an ftp or anonymous user for this FTP connection: /usr/samples/tcpip/anon.ftp
This scripts should be used to create FTP user, otherwise connection will work but files won’t be reachable by HMC.

------------------------------------

opening/closing virtual terminal from hmc:

lssyscfg -r sys -F name                         <--get managed system name
lssyscfg -r lpar -m <managed system> -F name    <--get lpar name

mkvterm -m <managed system> -p <lpar>           <--opens a terminal window
rmvterm -m <managed system> -p <lpar>           <--closes a terminal window

~~.                                             <--logout from the session (not necessary to leave AIX)

------------------------------------

Adding/removing VLAN TAG dynamically:
(it will not add to profile)

adding:
chhwres -r virtualio --rsubtype eth -m <man.sys.> -o s -p <LPAR> -s <adapter id> -a "addl_vlan_ids+=<VLAN TAG>"

removing:
chhwres -r virtualio --rsubtype eth -m <man.sys.> -o s -p <LPAR> -s <adapter id> -a "addl_vlan_ids-=<VLAN TAG>"

------------------------------------

Changing LPAR weight:

lshwres -r proc -m <MAN_SYS> --level lpar --filter lpar_names=<LPAR_NAME>                            list actual running config
chhwres -r proc -m <MAN_SYS> -p <LPAR_NAME> -o s -a "uncap_weight=32"                                changes weight of actual running config

lssyscfg -r prof -m <MAN_SYS> --filter lpar_names=<LPAR_NAME>                                        list profile
chsyscfg -r prof -m <MAN_SYS> -i "name=<PROFILE_NAME>,lpar_name=<LPAR_NAME>,uncap_weight=32"         changes weight in given profile