You can switch serial modes using the CLI.

Dynamic Switching using the CLI

Use esxcfg-advcfg to set the current mode to none. Then set the new desired mode using the CLI.

Logging mode: esxcfg-advcfg -s com1 /Misc/LogPort

Shell mode: esxcfg-advcfg -s com1 /Misc/ShellPort

DCUI mode: esxcfg-advcfg -s com1 /Misc/ConsolePort

GDB mode: esxcfg-advcfg -s com1 /Misc/GDBPort

Example

If the serial mode is set to logging mode, it would require 2 commands to switch it to DCUI mode.

§. > esxcfg-advcfg –s none /Misc/LogPort

§. > esxcfg-advcfg –s com1 /Misc/ConsolePort