Administrators: Rasmus, Erki A, Lauri
4-node blade cluster, storage is backed by nas.k-space.ee.
¶ For user
- Appliances not providing services to K-SPACE itself will not get a (xyz.)k-space.ee domain name!
- While this has historically happened as once-in-a-two-to-three-years, no responsability is taken for the VMs acting out and rebooted. Reminder: K-SPACE does not advertise an SLA!
¶ How to get an Debian LTS VM:
- Have an eligable membership
- Sign in to pve.k-space.ee
- Send an e-mail to info@k-space.ee, include:
- what are you planning to do with the VM
- SSH public keys
- is public IPv4 needed
- do you need extra CPU, memory (4GiB) or storage (100GB)
- Access the hypervisor at pve.k-space.ee. Select 'Pool view':
Default SSH username is ubuntudebian.
Default IPv6 is currently a single-IP abnormality, fitting for the Zoo network. As long as it doesn't conflict (machines site is broken as of 2024-02, so you can't tell! 
If it works, it's probably fine) with existing IPs, you are welcome to change it. See also: prefix delegation
If not already done after confirmation (or IP change), claim ownership of your machine at members site
¶ qemu-guest-agent
Used to:
- Improve migrations of VM between hosts
- Check and notify if an absurd number of security updates are available.
- In theory sync to improve backups, but in reality not used.
If you remove qemu-guest-agent:
- Disable it under VM → Options.
- Shut down the machine fully. Reboot doesn't help.
- Start the machine.
¶ Using ISOs
nas ISO storage is shared and accessible to all PVE users. Integerity of images is not guranteed!
- Select Folder or Server View.
- Upload an ISO to
nasstorage →ISO Images.
- Your VM → Hardware → Double click CD/DVD drive → Select storage.
- Booting: If Your VM → Options → Boot Order is locked, you can still press Esc in Console while booting (similar to F12 on physical hardware).
- Cleanup: Hardware → CD/DVD: Do not use any media and Shutdown/Stop the VM. It'd be nice to clean up unused ISOs yourself. On periodic ISO cleanups, if your VM still had it attached, migrations will fail and your VM will be stopped out of your control.
¶ For cluster administrator
Storage comes from nas.k-space.ee.
IPv4 193.40.103.100 .. 193.40.103.199 and 2001:bb8:4008:20::100/64 .. 2001:bb8:4008:20::199/64 is reserved for proxmox VMs with IDs 100..199.
Creating VM for use:
- Full clone from template.
- Name the VM with the name containing the user's name.
- Cloud-init: Change public IPv4 and IPv6 address from …99 to VM ID. ID
155gets193.40.103.155. - Normal user permissions: use
K-SPACE_VM_USER
¶ Updating major versions
- Coordinate with others to not mess with their 5-figure projects
- Check patch notes as usual
- After update, comment the enterprise repo in
/etc/apt/sources.list.d/, then adddeb http://download.proxmox.com/debian/pve bullseye pve-no-subscription, make sure to updatebullseyeto whatever is current.
¶ Recovering from migration problems.
-
Failed.
2021-02-21 17:44:11 migration status: completed 2021-02-21 17:44:14 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve1' root@2001:bb8:4008:21:172:21:20:1 qm unlock 156 2021-02-21 17:44:14 ERROR: failed to clear migrate lock: Configuration file 'nodes/pve1/qemu-server/156.conf' does not exist 2021-02-21 17:44:14 ERROR: migration finished with problems (duration 00:00:32) TASK ERROR: migration problems- Do not panic. The server is still online in the sender server.
- This guide is aimed at VMs, not containers. Containers differ from VMs, as such this guide is not fully compatible with containers.
- There's 4-7 components either fighting or out of sync. This is bad, and this is the most reliable way I found to get all working again.
-
Go to the shell of the sender, run:
qm unlock <id of the vm stuck, ex 111>If the command output is
Configuration file 'nodes/pve2/qemu-server/156.conf' does not exist, run it on the recieving end instead. If that succeeds fine, you don't have to hibernate. -
Hibernate the VM in question on the sender node. (broadcasts and sets/overwrites various doings of the VM, hibernate is the least costly)
- There will be around a minute or two downtime fot the VM. Coordinate with the owner if possible (during working hours).
- No data will be lost, including those in memory.
-
Resume the VM.
-
Re-attempt migration. This attempt will fail. (cleanup)
- You should now be where you started. Best of luck.
-
Re-attempt migration.
¶ Creating template VMs
first block by Rasmus, failed attempt in getting 24.04 nicely working
# Ubuntu: pretty bad with systemd-boot, grub2 doesn't like newer xfs
# VM 9028 has already been created
some=noble-server-cloudimg-amd64-root.tar.xz # changes with release
wget https://.../"$some.tar.xz"
modprobe nbd max_part=8
qemu-nbd --connect /dev/nbd0 /mnt/pve/nas/images/9028/vm-9028-disk-0.raw
gdisk /dev/nbd0
# o
# n, end: +512M, type ef00
# n
# c: 1: EFI, 2: cloudimg-rootfs
# w
mkfs.fat /dev/nbd0p1
mkfs.xfs /dev/nbd0p2
mkdir /mnt/cr
mount /dev/nbd0p2 /mnt/cr
tar -xf "$some.tar.xz" -C /mnt/cr
mkdir /mnt/cr/boot/efi
mount /dev/nbd0p1 /mnt/cr/boot/efi
# modify /mnt/cr/etc/fstab to use xfs
mount -o bind /proc /mnt/cr/proc
mount -o bind /sys /mnt/cr/sys
mount -o bind /dev /mnt/cr/dev
mv /mnt/cr/etc/resolv.conf /mnt/cr_resolv.conf
cp /etc/resolv.conf /mnt/cr/etc/resolv.conf
chroot /mnt/cr
apt update && apt -y full-upgrade
apt install -y grub-efi linux-generic
grub-install /dev/nbd0
update-grub
echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | debconf-set-selections
dpkg-reconfigure -f noninteractive unattended-upgrades
pro config set apt_news=false
apt install -y sshguard
apt install -y qemu-guest-agent
exit
umount /mnt/cr/{proc,sys,dev,boot/efi}
rm /mnt/cr/etc/resolv.conf
mv /mnt/cr_resolv.conf /mnt/cr/etc/resolv.conf
umount /mnt/cr
qemu-nbd --disconnect /dev/nbd0
rmmod nbd
- Attach blank disk image
- Attach official cloud rootfs
- Attach and boot LiveCD
- Create partitions on blank disk image
-
mkfs.xfs /dev/sda rsync -av --numeric-ids /official /blank mount --bind /dev/ /blank/dev mount --bind /sys /blank/sys mount --bind /proc /blank/proc chroot /blank grub-install /dev/sda pro config set apt_news=false # disables Ubuntu Pro ads apt-get update apt-get install sshguard unattended-upgrades -y sshguard -w 172.16.0.0/12 # https://sshguard.net/docs.html dpkg-reconfigure -plow unattended-upgrades # https://wiki.debian.org/UnattendedUpgrades apt-get full-upgrade -y exit reboot - Add Cloud-init
- Document what the VM is in Proxmox Notes
Reset machine-id:
rm /etc/machine-id # might be enough
systemd-machine-id-setup # do not run for template, only after cloning
¶ Adding a node
- Install Proxmox with ZFS RAID-1. Set the hostname, timezone is UTC. Other details don't matter.
- Replicate /etc/networking/interfaces from an another node, changing IPs. Apply.
- Join the node to cluster (under Datacenter)
- Add Prometheus node exporeter
- Follow
/mnt/pve/nas/jc/adding_a_node.md