Understanding the PXE boot process can help you during troubleshooting.
Trivial File Transfer Protocol (TFTP) is similar to the FTP service, and is typically used only for network booting systems or loading firmware on network devices such as routers. TFTP is available on Linux and Windows.
Most Linux distributions include a copy of the tftp-hpa server. If you require a supported solution, purchase a supported TFTP server from your vendor of choice. You can also acquire a TFTP server from one of the packaged appliances on the VMware Marketplace.
If your TFTP server will run on a Microsoft Windows host, use tftpd32 version 2.11 or later. See http://tftpd32.jounin.net/.
SYSLINUX, PXELINUX, and gPXELINUX
If you are using PXE in a legacy BIOS environment, you need to understand the different boot environments.
SYSLINUX is an open source boot environment for machines that run legacy BIOS firmware. The ESXi boot loader for BIOS systems, mbootc.32, runs as a SYSLINUX plugin. You can configure SYSLINUX to boot from several types of media, including disk, ISO image, and network. You can find the SYSLINUX package at http://www.kernel.org/pub/linux/utils/boot/syslinux/.
PXELINUX is a SYSXLINUX configuration for booting from a TFTP server according to the PXE standard. If you use PXELINUX to boot the ESXi installer, the pxelinux.0 binary file, mboot.c32, the configuration file, the kernel, and other files are all transferred by TFTP.
gPXELINUX is a hybrid configuration that includes both PXELINUX and gPXE and supports booting from a Web server. gPXELINUX is part of the SYSLINUX package. If you use gPXELINUX to boot the ESXi installer, only the gpxelinux.0 binary file, mboot.c32, and the configuration file are transferred via TFTP. The remaining files are transferred via HTTP. HTTP is typically faster and more reliable than TFTP, especially for transferring large amounts of data on a heavily loaded network.
VMware currently builds the mboot.c32 plugin to work with SYSLINUX version 3.86 and tests PXE booting only with that version. Other versions are likely to be incompatible. This is not a statement of limited support. For support of third-party agents that you use to set up your PXE booting infrastructure, contact the vendor.
UEFI PXE and iPXE
Most UEFI firmware natively includes PXE support that allows booting from a TFTP server. The firmware can directly load the ESXi boot loader for UEFI systems, mboot.efi. Additional software such as PXELINUX is not required.
iPXE can also be useful for UEFI systems that do not include PXE in firmware and for older UEFI systems with bugs in their PXE support. For such cases you can try installing iPXE on a USB flash drive and booting from there.
Apple Macintosh products do not include PXE boot support. They include support for network booting via an Apple-specific protocol instead.
Alternative Approaches to PXE Booting
Alternative approaches to PXE booting different software on different hosts are also possible, for example:
Configuring the DHCP server to provide different initial boot loader filenames to different hosts depending on MAC address or other criteria. See your DCHP server's documentation.
Approaches using iPXE as the initial bootloader with an iPXE configuration file that selects the next bootloader based on the MAC address or other criteria.