You can switch serial modes using the CLI.
Dynamic Switching Using the CLI
Use esxcfg-advcfg to set the current mode to none. Then set the new desired mode using the CLI.
Logging mode: esxcfg-advcfg -s com1 /Misc/LogPort
Shell mode: esxcfg-advcfg -s com1 /Misc/ShellPort
DCUI mode: esxcfg-advcfg -s com1 /Misc/ConsolePort
GDB mode: esxcfg-advcfg -s com1 /Misc/GDBPort
Example
If the serial mode is set to logging mode, enter these two commands to switch it to DCUI mode.
§. > esxcfg-advcfg –s none /Misc/LogPort
§. > esxcfg-advcfg –s com1 /Misc/ConsolePort