vSphere 5.x is a major upgrade from vSphere 4.x.

The following changes from vSphere 4.x affect vSphere installation and setup. For a complete list of new features in vSphere 5.x, see the release notes for version 5.x releases.

Service Console is removed
ESXi does not include a Service Console. You can perform most tasks that you performed in the Service Console by using esxcli commands in the ESXi Shell, by using vCLI commands, and by using VMware PowerCLI commands. See Command-Line Management in vSphere 5.0 for Service Console Users and Getting Started with vSphere Command-Line Interfaces.
ESXi does not have a graphical installer
The graphical installer relied on the Service Console, which is not a part of ESXi. ESXi retains the text-based installer.
vSphere Auto Deploy and vSphere ESXi Image Builder CLI

Before ESXi 5.0, ESXi was installed on the physical disk of each ESXi host. With ESXi 5.x, you can load an ESXi image directly into memory by using vSphere Auto Deploy. You can provision and reprovision large numbers of ESXi hosts efficiently with vCenter Server, and manage ESXi updates and patching by using an image profile. You can save host configuration such as network or storage setup as a host profile and apply it to the host by using Auto Deploy. You can use ESXi Image Builder CLI to create ESXi installation images with a customized set of updates, patches, and drivers.

For complete information on using vSphere Auto Deploy and ESXi Image Builder PowerCLI, see the vSphere Installation and Setup documentation.

Changes in the ESXi installation and upgrade process
ESXi 5.x uses a single installer wizard for fresh installations and upgrades. ESXi 5.x also provides a new option for deploying ESXi directly into the host memory with vSphere Auto Deploy. The vihostupdate and esxupdate utilities are not supported for ESXi 5.x. You cannot upgrade or migrate from earlier ESX or ESXi versions to ESXi 5.x by using any command-line utility. After you have upgraded or migrated to ESXi 5.x, you can upgrade or patch ESXi 5.x hosts using vCLI esxcli commands.
Important: After you upgrade or migrate your host to ESXi 5.x, you cannot roll back to your version 4.x ESX or ESXi software. Back up your host before you perform an upgrade or migration, so that, if the upgrade or migration fails, you can restore your 4.x host.

See ESXi 5.5 Upgrade Options.

Installer caching
Instead of using a binary image to install the system, whatever bits were used at boot time are cached to the system. This caching reduces installation problems caused by accessing installation files across networks that are under load.
Note: Scripted installations cannot PXE boot a server and then obtain the binary image from some other form of media.
Changes to partitioning of host disks

All freshly installed hosts in vSphere 5.x use the GUID Partition Table format instead of the MSDOS-style partition label. This change supports ESXi installation on disks larger than 2TB.

Newly installed vSphere 5.x hosts use VMFS5, an updated version of the VMware File System for vSphere 5.x. Unlike earlier versions, ESXi 5.x does not create VMFS partitions in second and successive disks.

Upgraded systems do not use GUID Partition Tables (GPT), but retain the older MSDOS-based partition label.

VMware vCenter Server Appliance

As an alternative to installing vCenter Server on a Windows machine, vSphere 5.x provides the VMware vCenter Server Appliance. The vCenter Server Appliance is a preconfigured Linux-based virtual machine optimized for running vCenter Server and associated services.

vSphere Web Client
The vSphere Web Client is a server application that provides a browser-based alternative to the deprecated vSphere Client. You can use a Web browser to connect to the vSphere Web Client to manage an ESXi host through a vCenter Server.
vCenter Single Sign-On

vSphere versions 5.1 and later include vCenter Single Sign-On as part of the vCenter Server management infrastructure. This change affects vCenter Server installation, upgrading, and operation. Authentication by vCenter Single Sign-On makes the VMware cloud infrastructure platform more secure by allowing the vSphere software components to communicate with each other through a secure token exchange mechanism, instead of requiring each component to authenticate a user separately with a directory service like Active Directory. See How vCenter Single Sign-On Affects vCenter Server Upgrades