You can upgrade a VMFS3 to VMFS5 datastore.
The upgrade is a one-way process. After you have converted a VMFS3 datastore to VMFS5, you cannot revert it back.
When upgrading the datastore, use the following command: vmkfstools -T|-- upgradevmfs /vmfs/volumes/UUID
All hosts accessing the datastore must support VMFS5 . If any ESX/ESXi host version 4.x or earlier is using the VMFS3 datastore, the upgrade fails and the host's mac address is displayed. with the Mac address details of the Host which is actively using the Datastore