See Also

Proxmox API viewer: https://pve.proxmox.com/pve-docs/api-viewer/index.html

Salt Proxmox docs: https://docs.saltstack.com/en/latest/topics/cloud/proxmox.html

Installing Proxmox

Download ISO from https://www.proxmox.com/en/downloads and either write it to a USB if local, or use the iDRAC to mount it off the NAS if not local.

Boot to the ISO, follow the instructions for setup.  Reboot into the freshly installed Proxmox.

Set up networking

PVE nodes (pve1 and pve2, Dell R720s)

• eth0 (first onboard 10Gb) and enp5s0f0 (first PCI-e 10Gb) are the external connection, bonded to create bond0, and providing VM connections via VLAN trunking
  • VMs and containers access other networks via VLAN tagged interfaces on vmbr999, which is bridged to bond0
  • vmbr999.21 provides a management VLAN interface for the PVE node (10.101.21.0/24)
• eth1 (second onboard 10gb) and enp5s0f1 (second PCI-e 10Gb) are the Ceph network (192.168.254.0/24), bonded to create bond1
  • This network allows Ceph to communicate and synchronize data when necessary

The /etc/network/interfaces for pve1 is here as an example.  The config for pve2 is very similar.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

allow-hotplug eth2

auto eth2
iface eth2 inet manual

iface eth3 inet manual

auto bond0
iface bond0 inet static
        address 192.168.254.1
        netmask 255.255.255.0
        bond-slaves eth1 enp5s0f1
        bond-mode 802.3ad
        bond-miimon 100
        bond-downdelay 200
        bond-updelay 200
        mtu 9000

auto bond1
iface bond1 inet manual
        bond-slaves eth0 enp5s0f0
        bond-mode 802.3ad
        bond-miimon 100
        bond-downdelay 200
        bond-updelay 200
        mtu 9000

auto vmbr999
iface vmbr999 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

#Trunk

auto vmbr999.21
iface vmbr999.21 inet static
        address  10.101.21.41
        netmask  255.255.255.0
        gateway  10.101.21.1
        broadcast  10.101.21.255
        network 10.101.21.0
        dns-nameservers 8.8.8.8 8.8.4.4
        dns-search magevent.net
        vlan-raw-device vmbr999
        mtu 9000

Quorum node (Dell R320)

The quorum node has a similar network config.  Because the hardware is different, however, the physical connections aren't the same.

• eth2 (first port of one of the 10Gb cards) and enp8s0f0 (first port of the other 10Gb card) are the external connection, bonded to create bond0, and providing VM connections via VLAN trunking
  • VMs and containers access other networks via VLAN tagged interfaces on vmbr999, which is bridged to bond0
  • vmbr999.21 provides a management VLAN interface for the PVE node (10.101.21.0/24)
• eth3 (second port of one of the 10gb cards) and enp8s0f1 (second port of the other 10Gb card) are the Ceph network (192.168.254.0/24), bonded to create bond1
  • This network allows Ceph to communicate and synchronize data when necessary

auto lo
iface lo inet loopback

allow-hotplug eth0

auto eth0
iface etho inet manual
# dns-* options are implemented by the resolvconf package, if installed

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto bond0
iface bond0 inet static
        address  192.168.254.3
        netmask  255.255.255.0
        bond-slaves eth3 enp8s0f1
        bond-mode 802.3ad
        bond-miimon 100
        bond-downdelay 200
        bond-updelay 200
        mtu 9000

auto bond1
iface bond1 inet manual
        bond-slaves eth2 enp8s0f0
        bond-mode 802.3ad
        bond-miimon 100
        bond-downdelay 200
        bond-updelay 200
        mtu 9000

auto vmbr999
iface vmbr999 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
#Trunk

auto vmbr999.21
iface vmbr999.21 inet static
        address  10.101.21.40
        netmask  255.255.255.0
        gateway  10.101.21.1
        broadcast  10.101.21.255
        network 10.101.21.0
        dns-nameservers 8.8.8.8 8.8.4.4
        dns-search magevent.net
        vlan-raw-device vmbr999
        mtu 9000

DNS

Add DNS servers to /etc/resolv.conf

search magevent.net
nameserver 10.101.22.110
nameserver 10.101.22.120
nameserver 8.8.8.8