POST
Creating a PV RancherOS instance on Xen on Alpine (whew!)
Since I’m a bit of a masochist, I decided that I would try running RancherOS on the nodes of my in-house compute cluster.
Actually, let me back up. Since I’m a masochist, I decided to run my own compute cluster.
And since I’m really a masochist, I decided to try running RancherOS as a guest OS on some sort of hypervisor. I decided to start with Xen because Open Source, although I did consider the free tier of VSphere.
In everything that follows, my test machine is a real (physical) PC, an i7-3770K, 16GB of RAM, and an SSD as /dev/sda
. My goal was to put the Dom0 OS on a USB drive and dedicate the SSD to “workers”.
Alpine as Dom0
I decided to use Alpine Linux as the top-level (so-called “Dom0”) operating system. It’s designed to run from a thumb drive and even has a Xen-enabled version right on the downloads page. I grabbed the ISO and burned it to a thumb drive using Etcher.
And low and behold, the system booted right up.
[At this point, some mucking around with Alpine ensued. It’s counterintuitive that you need to burn a thumb drive (or CDROM) to install to a thumbdrive. I’m probably doing this wrong!]
At this point I thought I had a live USB system, but I quickly discovered Alpine was running entirely from RAM. It was as if I had booted from a CDROM.
Instead, I had to follow the section “Add Alpine Linux to the USB stick” to install Alpine from one USB drive to another. Weird!
So, follow the instructions to create a new VFAT partition on the target thumb drive, then run the alpine command:
setup-bootable /media/usb /dev/sdb1
(where, as always, replace sdb1
with your install drive.)
to install the Alpine distribution from one thumbdrive to the other.
At this point the thumb drive /dev/sdb1
is bootable, Go ahead and reboot, pull out the
first thumb drive, then run alpine-setup
again. This time, store configs on “usb”, which will let you
use lbu
to store configurations on the thumb drive. In the standard mode of operation
the thumb drive is mounted read-only. Alpine is smart and will re-mount the
drive writeable when it needs to (like when you run lbu
). If you want to store
application data on the thumb drive, you can do this manually with:
mount -o remount,rw /dev/sdb1 /media/usb
Neat. It take a little getting used to, but using lbu
to persist system changes
to the USB works well.
The Xen alpine image has all of the Xen modules and packages by default!
RancherOS as para-virtualized guest
Now we can get onto creating a RancherOS guest. In contrast to the Alpine install, my goal is to provide
RancherOS with space on the system SSD for saving it’s state (like downloaded Docker images, as well as configuration) between reboots. We do this by providing a persistent state partition, a hard drive partition with the label RANCHER_STATE
.
Interestingly, I think we can create the guest OS without ever running an installer … mostly because RancherOS is also mostly state-less. Whether booting from a thumb drive or a HDD, it runs a pre-packaged RAM disk, and only stores state (to the persistent state partition) as needed.
First, download a recent RancherOS ISO. I stored it to my USB drive (after remounting the USB drive read-write, as above):
mount -o remount,rw /dev/sdb1 /media/usb
wget -O /media/usb/isos/rancheros-v1.4.0.iso https://github.com/rancher/os/releases/download/v1.4.0/rancheros.iso
Then mount the ISO:
mount -t iso9660 -o loop,ro /media/usb/isos/rancheros-v1.4.0.iso /media/cdrom
Now we need to prepare partitions on the SSD (/dev/sda
) for the guest VM. I’m following the suggestion from Xen
and using LVM and have already created my physical volumes and volume groups
pvcreate /dev/sda1
vgcreate vg0 /dev/sda1
Then create the logical volumes. I’ll create one for the root partition and one for swap:
lvcreate -n rancher_root -L 32G vg0
lvcreate -n rancher_swap -L 4G vg0
Now, this is where things get interesting. Format the root partition as ext4 making sure to include the RANCHER_STATE label:
mkfs.ext4 -L RANCHER_STATE /dev/vg0/rancher_root
Also make a copy of the boot/
partition on from the Rancher ISO, and store it on the USB drive:
mkdir -p /media/usb/pvs/rancher
cp -r /media/cdrom/boot /media/usb/pvs/rancher/
This is part of the wierdness of PV guests. Their kernels live outside their bodies!
Now we can create the Xen specification for the guest. Make the file /etc/xen/rancher.cfg
:
# =====================================================================
# PV RancherOS Guest
# =====================================================================
# Guest name
name = "rancher"
# 128-bit UUID for the domain as a hexadecimal number.
# Use "uuidgen" to generate one if required.
# The default behavior is to generate a new UUID each time the guest is started.
#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
# Kernel image to boot
kernel = "/media/usb/pvs/rancheros/boot/vmlinuz-4.14.32-rancher2"
# Ramdisk (optional)
ramdisk = "/media/usb/pvs/rancheros/boot/initrd-v1.4.0"
# Kernel command line options
# rancher.debug=true
# printk.devkmsg=on
# rancher.password=mypassword
extra = "console=hvc0 rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait panic=10"
# Initial memory allocation (MB)
memory = 4096
# Maximum memory (MB)
# If this is greater than 'memory' then the slack will start ballooned
# (this assumes guest kernel support for ballooning)
#maxmem = 512
# Number of VCPUS
vcpus = 1
maxvcpus = 4
# Network devices
vif = [ 'bridge=br0' ]
# Disk Devices
disk = [ '/dev/vg0/rancher_root,raw,xvda1,w',
'/dev/vg0/rancher_swap,raw,xvda2,w' ]
Your VCPUS and memory configuration may vary. It’s important to recognize that the Linux kernel command line
args are specified in this file, not in the guest OS’s /boot
partition where it would be normally.
This should make a complete but not-that-usable RancherOS distribution. Not that usable because it doesn’t have any user-supplied configuration. However, when RancherOS boots and attaches the persistent state drive, if will create all of the necessary stub files.
Now, in a separate window, run:
xl create -c /etc/xen/rancher.cfg -c
You should see the console roll of the guest booting. It will stop (should!) at:
[ ] ros-sysinit:info: RancherOS v1.4.0 started
It won’t show a login prompt! Why not? RancherOS looks for the console at /dev/tty1
- /dev/tty6
. Xen likes to put the console at /dev/xcd0
or maybe /dev/tty0
. We can fix this manually later!
Now kill the guest vm.
xl shutdown rancher
And mount the root partition:
mkdir /mnt/rancheros
mount -t ext4 /dev/vg0/rancher_root /mnt/rancheros
If we now look on the root partition, it contains a whole bunch of RancherOS state:
# ls /mnt/rancheros
dev home lost+found mnt proc run sys usr var
etc lib media opt root sbin tmp usr-v1.4.0
You can go ahead and modify this directly, or be good and put everything into a cloud-config
file.
Specifically, clout-init files can go in /var/lib/rancher/conf/cloud-config.d
on the Rancher partition.
For example, added the following user_config.yml
file to that folder:
#cloud-config
ssh_authorized_keys:
- ssh-rsa AAAA......AAA mykey@hostname
write_files:
- owner: root
path: /etc/rc.local
permissions: "0755"
content: |
#!/bin/bash
# Enable swap
swapon /dev/xvda2
#
nohup /sbin/agetty --noclear hvc0 linux &
This does two things. First, it adds my SSH public key to the authorized keys for the user rancher
. Second, it adds a short /etc/rc.local
which enables swap and starts an agetty service on /dev/hvc0.
Now restart the guest again:
xl create /etc/xen/rancher.cfg -c
This time, you do get a login console (of course, you don’t know the password for rancher…)
[ ] ros-sysinit:info: RancherOS v1.4.0 started
, , ______ _ _____ _____TM
,------------|'------'| | ___ \ | | / _ / ___|
/ . '-' |- | |_/ /__ _ _ __ ___| |__ ___ _ __ | | | \ '--.
\/| | | | // _' | '_ \ / __| '_ \ / _ \ '__' | | | |'--. \
| .________.'----' | |\ \ (_| | | | | (__| | | | __/ | | \_/ /\__/ /
| | | | \_| \_\__,_|_| |_|\___|_| |_|\___|_| \___/\____/
\___/ \___/ Linux 4.14.32-rancher2
RancherOS #1 SMP Fri May 11 11:30:31 UTC 2018 rancher hvc0
docker-sys: 172.18.42.1 eth0: 192.168.88.54 lo: 127.0.0.1
rancher login:
and from that you can SSH into the machine using the key provided previously:
# ssh rancher@192.168.88.54
The authenticity of host '192.168.88.54 (192.168.88.54)' can't be established.
ECDSA key fingerprint is SHA256:0AICFBR8On9i2YjWs7k6TpDV4wAnz33HCGIggfNtgbk.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.88.54' (ECDSA) to the list of known hosts.
[rancher@rancher ~]$
From here, you can make all sorts of changes, either within the OS, or by changing the cloud-config!
Nice!