ESXi host fails to enter maintenance mode without any error
It is possible for ESXi host to not enter maintenance mode and keep hanged. There is typically no timeout for enter maintenance mode and even after 1-2 days the process can continue. Thus, we need to understand why the host is not able to maintenance mode and fix the issue. Possible reasons for host unable to enter maintenance mode are:
No DRS configured with Running VMs
If host has running VMs they can migrate to other hosts (assuming enough free resources are available) and then host can enter maintenance mode. For this to happen automatically DRS should be enabled in cluster so that platform can decide where to move the current VMs.
No EVC mode
It is possible that EVC mode is not enabled. Hence a few or all VMs may not be compatible with other hosts for migration purposes. Hence automated migration is not possible leading to host getting stuck while maintenance mode.
RDM configuration or any other VM specific issue
If any VM is configured with RDM (Raw device mapping) then it will fail to migrate and hence host will not enter maintenance mode. For example, if serial port is added to a machine with type "Physial serial port on host" with connected checkbox (enabled / ticked), then the VM serial ports gets connected to host device. In this case VM will not vMoption to other hosts.
Other VM specific issue might be that is you are using standard switch, the network with exact same name may not be available on any other host for automated migration.
In such cases, after you try to enable maintenance mode on the host, select the host in "Host and cluster" view and then click on "VMs" tab to see list of VMs on this host. Ideally one by one these VMs should vMotion to other hosts and host should enter maintenance mode. But if that is not happening look at the final VMs remaining and then cancel "ESXi host enter maintenance mode" task. Then right-click on these VMs and try to migrate them manually. Often the compatibility tests fail or there is warning with reason. The warning could be some dependency on local datastore for VM disk / CD/DVD iso, serial port, CPU type (If EVC mode is not enabled), etc. Based on this you can understand why the migration failed.
Thus, even if at individual VM level vCenter can see that migration wont work, somehow that automation is not used while entering maintenance mode. Thus, we need to manually check the reason for failing vMotion and solve the issue for host to be able to enter maintenance mode.
On serial ports refer: