¶ HP Blades
Administrators: Rasmus, ErkiA
For newly commisioned blades configure power profiles for 200-500 W lower idle!
¶ iLO — integrated Lights-Out Management (ft. Remote Console)
Blades are not meant to be accessed directly. All accounts (including blade-local Administrator) have been removed. Access is granted through enclosure login.
- Ask credentials from an administrator. Priviledges are limited to the slot in enclosure — if the blade is moved to an another slot, access is lost.
- Login to the enclosure at
https://172.23.1.101. - Select desired blade from either the list on the left, or the image on the right:
- Open the iLO page:
HTML5 Remote Console(or for blade iLOWeb Administration).- Black screen without buttons might be NoScript at work.
See also: Server Configuration (BIOS, RAID, etc)
¶ Virtual media (ISO)
Before installing an OS make sure you have your desired RAID0/1/JBOD configuration.
Virtual media might be slow! It is random IOPS over VPN over (your home) internet connection, possibly over WiFi, then the browser and local disk. You might want to netboot.
- Open the remote console (detailed a a chapter above).
- Top left →
💿→CD/DVD→Local *.iso file - Servers should default to booting from the mounted image, if not, press F11 on boot splash (Server Configuration)
- The image may be automatically ejected by the operating system (for example after an install), or from the same menu used for mounting it.
¶ g7 console access
g7 blades do not implement HTML5 console access. Therefore you need Internet Explorer (ft. ActiveX and Java). WIth mixed results, icedtea-web might work as well.
Detailed instructions
- From the iLO page,
Integrated Remote Console. - Say yes to a bunch of security warnings and enjoy the magic.
- You should now have a remote console. If not, confirm if you're using IE, and if you indeed have (as on the Windows server) the Java stuff installed.
IE is too new now, use SeaMonkey (or dedicated iLO app) instead. The servers are no longer in use.
¶ Encryption warnings
Check the box and click Ok.
¶ IE Enchanced Security Configuration
In progress to add the local intranetwork to the local intranetwork for IE. For now:
- On the main IE window, top right:
- Security → Local intranet → Sites
- Add
*://172.16-31.*.*to the exceptions.
- Refresh the page in question.
¶ Hardware
¶ Enclosures
Enclosure 'Administrator' is the only user which may have access to iLO via SSH, and as such is used as a prometheus service account.
- (in rack) 2x HP C7000 (P/N 438859-001), each able to house 16 blades.
- Spare HP C7000 (P/N 403318-b21), slightly older.
¶ Switches
K-SPACE bought 2x HP VC FlexFabric-20/40 F8 Module for PVE higher BW migrations and NFS storage.
¶ Blades
- We started with g7 blades (DDR3-age)
- 2023: Most utilized blades are now g9, which seem to use more power (DDR4-age). iLO can be accessed via modern browsers. Some of them have been upgraded to U.2 backplane with NVMe SSDs.
¶ Networking
K-SPACE infra PVE configuration lives in https://git.k-space.ee/k-space/ansible. Here are old configs for an example.
¶ Debian /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno49 inet manual
iface eno50 inet manual
# DHCP Zoo IP from 172.20.8.0/24 subnet + IPv6 address from EENet subnet 2001:bb8:4008:20::/64
auto bond0
iface bond0 inet dhcp
address 0.0.0.0
bond-mode active-backup
bond-primary eno49
bond-slaves eno49 eno50
bond-miimon 100
mtu 9000
post-up tc qdisc add dev bond0 root sfq perturb 10
# To configure static public IPv4 from EENet use IP alias method
# Obtain unused IP and replace X accordingly
auto bond0:1
iface bond0:1 inet static
address 193.40.103.X
netmask 255.255.255.0
iface bond0:1 inet6 static
address 2001:bb8:4008:20:193:40:103:X
gateway 2001:bb8:4008:20::1
netmask 64
¶ systemd-networkd
Configure the bond device:
# /etc/systemd/network/bond0.netdev
[NetDev]
Name=bond0
Description=LAG/Bond to a switch
Kind=bond
[Bond]
Mode=802.3ad
Attach interfaces to bond (other possibilities include using the PCI path: Path=pci-0000:00:xx.y or just using wildcards for interface names Name=enp2*):
# /etc/systemd/network/bond0.network
[Match]
Name=enp2s0f0
Name=enp2s0f1
[Network]
Bond=bond0
And configure the bond network (called wan):
# /etc/systemd/network/wan.network
[Match]
Name=bond0
[Network]
DHCP=yes
# ... and attach static public IPv4 and IPv6 addresses
Address=193.40.103.x/32
Address=2001:bb8:4008:20:193:40:103:x/64
Gateway=2001:bb8:4008:20::1
# Suffix for IPv6 SLAAC addresses
IPv6Token=::dead:beef
# IPv6 Router advertisements
IPv6AcceptRA=yes
# XXX: Unfortunately I haven't yet figured out the gateway stuff for DHCP + Static ipv4,
# XXX: we end up with following error:
# XXX: bond0: Could not set route: Nexthop has invalid gateway. Network is unreachable
### Gateway=193.40.103.1
## TODO: investigate these
# XXX: Microsoft Link-Local DNS name resolution
# XXX: LLMNR=yes
# XXX: Apple Bonjour/Avahi link-local dns resolution
# XXX: MulticastDNS=yes
[DHCP]
UseDomains=yes
Also, disable /etc/network/interfaces by renaming it (for example interfaces.disabled) and don't forget to enable the service: systemctl enable systemd-networkd.
¶ Trunk ports
VLAN trunk port configuration for HP blade servers, this it the Proxmox node network configuration. Before using it understand what it does and substitute IP addresses accordingly
# Slave interfaces
auto enp2s0f0
iface enp2s0f0 inet manual
bond-master bond0
bond-mode 802.3ad
auto enp2s0f1
iface enp2s0f1 inet manual
bond-master bond0
bond-mode 802.3ad
# Master interface
auto bond0
iface bond0 inet static
address 0.0.0.0
bond-slaves none
bond-miimon 100
mtu 8996
# Interface for Zoo has no IP since it's vmbr0 member
auto bond0.20
iface bond0.20 inet static
address 0.0.0.0
mtu 1500
# Zoo bridge for VM-s, has no IP since administration is done via VLAN21
auto vmbr0
iface vmbr0 inet static
address 0.0.0.0
accept_ra 0
bridge_ports bond0.20
# Interface for NFS client network (VLAN 101)
auto bond0.101
iface bond0.101 inet6 static
address 2001:bb8:4008:101:172:20:10:1/64
# pve{1..3}.proxmox.infra.k-space.ee, used as HAproxy backend
auto bond0.21
iface bond0.21 inet6 static
address 2001:bb8:4008:21:172:21:20:1/64
gateway 2001:bb8:4008:21:ffff:ffff:ffff:fffe
mtu 1500
¶ Server configuration
While booting, a lenghty black screen is normal. Pre-splash screen POST codes are usually 3038, 4048 etc.
While F9 and F11 are recognized right away, other F-keys are detected only after the splash, after initializations.
¶ RAID controller configuration
- From splash,
(also exposes some network configuration) - Wait for this section: (image missing)
- Press
F10. Do not press it before, as it takes you to different menus.
¶ Power management
Enc: Enable Dynamic Power, AC redundant
Blades:
Stock configuration draws tons 100% the power, even at idle. Please apply the superior config detailed below.
The profile might shwo up as 'Custom' state, which is expected.
¶ Profile 1
- Go to bios (
F9on boot) Power Management Options→HP Power Profile→Minimum Power usage
Keep in mind the watts are DC of what the blade uses. This does not account for DC conversion, fans, switches, and other nessecities the enclosure provides.
The Blade is currently limited/locked to 1600MHz. Idle is around 80W-85W, all cores 100% is around 160W.
This is the recommended selection as it is very likely your blade will not be anywhere near 100%, often not even 50%.
You may pick the lower performance setting, and change it later as well.
¶ Profile 2
To allow 3200MHz (while still having good power usage while idle):
HP Power Profileshall be set toMinimum Power Usagebefore you do next step.Power Management Options→HP Power Regulator→HP Dynamic Power Savings Mode
Idle wattage is around 110W-115W, all cores 100% is around 260W.
¶ Profile 3
Instead of selecting HP Dynamic Power Savings Mode, when using Linux, OS Control might be an option.
While it manages to rarely idle at 70W-75W, it normally idles at the same 110W-115W.
The difference here is that HP Dynamic Power Savings Mode does that at 1600MHz, OS Control keeps the processor at 3100MHz for a few minutes on idle.
While OS Control might seem better, your results may vary. As the results are quite similar to Profile 2, choosing might be a better idea.