ESXi includes a unique keystroke sequence that allows dynamic serial mode switching.
Dynamic Switching Keystrokes
Once the correct keystroke sequence is entered, the system switches the serial port to the desired mode.
Logging mode: Ctrl+G, Ctrl+B, 1
Shell mode: Ctrl+G, Ctrl+B, 2
DCUI mode: Ctrl+G, Ctrl+B, 3
GDB mode: Ctrl+G, Ctrl+B, ?
Once in GDB mode, you cannot switch modes again using a key sequence. You must use the CLI to switch modes.