Proper ordering of shutdown and startup is important to prevent data loss and making Yesrod very sad.
Shut down all the trash compactors on the detention level the Dell boxes in the event rack.
The bottom three are most important (labeled pve1, pve2 and quorum). You should be able to just press the power button and wait, and they'll go down gracefully. DO NOT press and hold the power button; this causes an immediate power off and can be destructive to the data on these servers.
Shut down the Synology (black box with 4 hard drive bays located below the core rack switch).
This one is a little different: You DO want to press and hold the power button. The Synology NAS boxes work a bit different than a normal computer; the press and hold does a graceful shutdown instead of an immediate power off. (Yes, that's confusing. Yes, I hate it too.)
Cut power to the rest of the rack via the PDUs at the back.
There are two APC power distribution units (fancy power strips) in the back of the rack. Each has a power switch. Switch both power switches to the off position. Make sure that the Dell boxes and the Synology are off before you do this!
Unplug and pack up the rack.
Unpack and plug in the rack.
Power on the PDUs at the back of the rack.
PDU, power switch, on position, etc. Core net switch should start automatically.
Power on the Synology (black box with 4 hard drive bays located below the core rack switch).
Press the power button on the front of the Synology. Power button should start flashing, then eventually go solid green as the rest of the NAS comes up.
You may also be able to power on the Synology remotely via Wake-on-Lan, try
ether-wake -i vmbr999.21 00:11:32:8c:36:4e
Power on the Dell boxes.
It's a computer; press the power button and it turns on. After a while, the Proxmox cluster will boot to the point where the pfSense VM comes up, and network connectivity should then be restored.
Note that you need physical access to start up the rack; it's not currently possible to completely start the rack remotely. (You may be able to complete steps 3 and 4 of the above startup sequence remotely, but someone still has to be on site to plug in the rack and turn the PDUs/UPS on.)
Shut down the Synology remotely.
Go to synology.magevent.net in a web browser, log in, click the person icon in the upper left, and click Shutdown.
Shut down the quorum server
SSH to quorum.magevent.net and shut down the server.
# ssh [email protected] "shutdown -h now"
Shutdown pve1 and pve2 as simultaneously as possible
As soon as the host running the pfSense VM goes down, you will lose all network access. Therefore, the two servers need to go down at the same time.
# ssh [[email protected]](<mailto:[email protected]>)
# ssh pve2 "shutdown -h now"; shutdown -h now
Have someone on site unplug and pack up the rack.Once all the servers are down, it's safe to unplug and start packing up.