DEVOPS - QEMU

QEMU - AIX on x86

QEMU (Quick EMUlator) is a machine emulator program, which is capable to create VMs with different processor architectures. For example if we want to test a Solaris (with RISC processor) or an HP-UX (with HPPA) or an AIX (with Power processor) QEMU can emulate that processor type and run the specified OS on that. So on Windows or Linux (on x86) with QEMU we can run AIX.

After installing QEMU, we can use a command to create an empty virtual disk and after a VM (with specified RAM and additional devices). If we use an AIX DVD during the boot, AIX will be installed there. (After installation not all AIX commands are available but many things still possible to do.)

Without any prior experience in QEMU, I used these 2 sources:
https://astr0baby.wordpress.com/2018/11/04/running-aix-7-2-tl3sp1-on-x86_64-via-qemu-system-ppc64/
http://gibsonnet.net/blog/cgaix/resource/AIX_QEMU_blog.pdf

--------------------------------------------------------------

AIX on Linux

Very important: be patient during this process, things can be (very) slow sometimes.

On Windows I installed VirtualBox and I created a Linux VM (CentOS 8). The plan was that on this linux VM in VirtualBox I will create an AIX VM using QEMU. After CentOS 8 installation completed, I configured network, so I could ssh to my linux server and do these steps:

0. Download AIX DVD 
From IBM ESS site download AIX DVD. I used 7.1 TL5 SP5, DVD1 is enough (710505.iso). Copy  to any directory, I used /root/aix.

1. Install Qemu
I could not find latest qemu package on linux (only version 2 was available, but that did not support latest Power systems), so it had to be compiled from source code:
(I had to install git and other missing things like glib-devel ...)
# git clone git://git.qemu.org/qemu.git   <--it downloads latest source code (automatically creates a qemu dir where I start command)
# cd qemu                                 <--go to qemu dir which was created by git
# mkdir build; cd build                   <--create a build directory and go there
# ../configure                            <--checks if all requirements are OK
# make                                    <--this compiles the source code (it can take long)
# make install                            <--it installs compiled software
# qemu-system-ppc64 --version             <--check if it works correctly
QEMU emulator version 4.2.50 (v4.2.0-2665-g3d0ac34603)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

2. Create a disk file
Go to the AIX DVD dir and create a disk which will be used by AIX
# qemu-img create -f qcow2 hdisk0.qcow2 20G
It will create an empty 20GB qcow2 disk image file, but its initial size is very small, as it is a sparse file. (So we have AIX DVD and this image file in our dir)

3. Create an AIX VM
This command will create an AIX VM with specified settings and boot it from DVD:
# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot cdrom:"

Parameters used:
-cpu: processor type, here POWER8
-machine: machine type, here pseries
-m: memory size, here 2GB
-serial: redirects serial output to specified device
-drive: the disk file we created before
-device: virtio-scsi-pci (not fully sure, but maybe this will create a virt. eth. adapter device)
-device: scs-hd (probably this will create hdisk device)
-cdrom: this is our downloaded AIX DVD which will be used during boot
-prom-env: sets NVRAM variable for PPC and SPARC servers, here a boot-command is used, which points to cdrom (DVD)

AIX boot will take several minutes, sometimes output can hang for 3-5 minutes. If everything is fine, the usual AIX install menu will apper (choose console and english language and default settings during install). The install will take about 2 hours, and disk file size will grow to 2.3GB. The installation process tries to restart AIX automatically after the install completed, but it was hanging, so after some time I did CTRL-C.

4. Fix fsck64 issue in Maintenance mode
fsck fails during boot and boot will hang. A workaround to this issue is to overwrite fsck64 file with exit 0. Boot our new AIX to maintenance moe:
# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot cdrom:"

When the "Welcome to Base Operating System" menu appears:  choose 3 (Start Maintenance Mode) --> 1 (Access a Root VG) -->  0 Continue  --> 1 (VG on hdisk0) --> 1 (Access this VG and start a shell). 

After that we will get a prompt with an environment where filesystems are already mounted:
# cd /sbin/helpers/jfs2    <--go to this location
# > fsck64                 <--delete content of that file
# vi fsck64                <--vi and insert these 2 lines. File is readonly, so save and quit with :w! and :q
#!/bin/ksh
exit 0

# cat fsck64               <--it should contain 2 lines (#!/bin/ksh and exit 0)
# sync ; sync              <--write cached memory to disk
# halt                     <--halt AIX


5. Boot from disk
This is the first time AIX will be booted from disk:
# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot disk:"

This boot will take much longer (about 5-10 minutes) as some daemons will fail and it will hang at NFS part until timeout, but we will get a prompt. After that some services will be disabled (so next boots will be faster) and installed ssh from DVD.
Disabling services in inittab:
# rmitab rcnfs
# rmitab cron
# rmitab piobe
# rmitab qdaemon
# rmitab writesrv
# rmitab naudio2
# rmitab naudio
# rmitab aso
# rmitab clcomd
# chrctcp -S -d tftpd

Install ssh and change root pw:
# chfs -a size=+200M /home
# mount -v cdrfs -o ro /dev/cd0 /mnt
# mkdir /home/ssh; cd /mnt/installp/ppc; cp openssh.base openssh.license openssh.man.en_US openssh.msg.en_US /home/ssh; umount /mnt
# cd /home/ssh; installp -acXY -d . all
# passwd root
# halt

After that booting up from disk should be faster.

6. Network setup
I wanted to use ssh on AIX, so network setup was needed. (When I worked in qemu terminal and I did ctrl-c, the whole qemu session (process) was stopped, so I had to boot again the VM, that was another reason to use ssh sessions.) Without knowing linux network virtualization techniques, it was not easy, but found this site, which worked: http://hostrepo.com/article.php?id=193. Basically a network bridge was needed, so network communication could be possible between my Linux and AIX VM. On Linux this network bridge function can be achieved by a TAP (Terminal Access Point) device and with ARP proxy. The following steps are needed each time after Linux is restarted, so these can be put in a script as well. (There are lots of internet sites which shows different other methods.)

current IP on Linux: 10.0.2.15 (enp0s3)
planned IP on AIX: 10.0.2.16 (en0)

On the Linux VM:
# ip tuntap add tap0 mode tap                      <--create tap0 interface ('ifconfig' will not show it, 'ip a' shows state is DOWN)
# ip link set tap0 up                              <--bring up tap0 ineterface ('ifconfig' will show tap0 state is UP)
# echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp  <--enable proxy arp on tap0 (empty file was already there)
# ip route add 10.0.2.16 dev tap0                  <--setup routing for 10.0.2.16 (AIX IP) on tap0 device ('ip route' shows new route)
# arp -Ds 10.0.2.16 enp0s3 pub                     <--broadcast ARP for AIX IP ('yum install net-tools' was needed for arp, netstat commands)

Boot up AIX using tap0 device:
# cd /root/aix
# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot disk:" -net nic,macaddr=56:44:45:30:31:32 -net tap,script=no,ifname=tap0

Then on AIX:
# chdev -l en0 -a netaddr=10.0.2.16 -a netmask=255.255.255.0 -a state=up

After that ping and ssh is possible to AIX. AIX IP configuration will survive restarts, but Linux steps are needed each time after Linux VM is rebooted. In future if we want to start AIX in the background without a console this can be used:
# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 2048 -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot disk:" -net nic,macaddr=56:44:45:30:31:32 -net tap,script=no,ifname=tap0 -daemonize

(logout from session, after AIX shutdown is possible using "~~.", same as in HMC console)

--------------------------------------------------------------

AIX on Windows

It is possible to do the same steps (as above) directly on Windows without VirtualBox and a Linux VM. The only difference is that we need to give qemu commands in command prompt (cmd) or in a terminal emulator (like MobaXterm). With these terminals on Windows there could be character problems.

We need to install latest Qemu on Windows: https://www.qemu.org/download/#windows. After install completed, above steps can be  followed, with slight modification in qemu commands:

Create a disk file: 
"C:\Program Files\qemu\qemu-img.exe" create -f qcow2 hdisk0.qcow2 20G

Boot an AIX VM and install AIX from DVD:
"C:\Program Files\qemu\qemu-system-ppc64.exe" -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=dev / 0 0 s\" ibm,aix-diagnostics\" property boot cdrom:\ppc\chrp\bootfile.exe -s verbose" -display vnc=:1

Boot our newly installed VM into Maintenance Mode and fix fsck64:
"C:\Program Files\qemu\qemu-system-ppc64.exe" -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot cdrom:" -display vnc=:1

Boot AIX from disk:
"C:\Program Files\qemu\qemu-system-ppc64.exe" -cpu POWER8 -machine pseries -m 2048 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom 710505.iso -prom-env "boot-command=boot disk:" -display vnc=:1

--------------------------------------------------------------