Portal Home > Knowledgebase > VMware Knowledge Base > Investigating virtual machine file locks on ESX/ESXi
To prevent concurrent changes to critical virtual machine files and file systems, ESX hosts establish locks on these files. In certain circumstances these locks may not be released when the virtual machine is powered off. The files cannot be accessed by the servers while locked, and the virtual machine is unable to power on.
These virtual machine files are commonly locked for runtime:
To identify the locked file, attempt to power on the virtual machine. During the power on process, an error may display or be written to the virtual machine's logs. The error and the log entry identify the virtual machine and files:
vmware.log
file of the virtual machine.# vmware-cmd -l
/vmfs/volumes///.vmx
referenced in the remainder of the article. It is also case-sensitive.# vim-cmd vmsvc/getallvms
[] /.vmx
vmware.log
file. At the end of the file, look for error messages that identify the affected file.The touch
utility is designed to update the access and modification time stamp of the specified file or directory. As such, the command can be used to test the file and directory locking mechanism in the VMFS filesystem, where the procedure is expected to fail on locked files. Using touch is the preferred method because the changes to the resource are minimal.
To test the file or directory locking functionality, run this command:
# touch
Note: Performing a "touch *"
command performs the operation on all files in the current directory.
The touch *
command can result in these outcomes:
touch *
command succeeds, then the command successfully made changes to the date/time stamp and has verified that the file can and has been locked (then unlocked). At this point, retry the virtual machine power-on operation to see if it succeeds.touch *
command fails with a device or resource busy
message, it indicates that a process is maintaining a lock on the file or directory. This may be on any of the ESX hosts which have access to the file. If the message is reported, proceed to the next section.Because a virtual machine can be moved between hosts, the host where the virtual machine is currently registered may not be the host maintaining the file lock. The lock must be released by the ESX host that owns the lock. This host is identified by the MAC address of the primary Service Console interface.
Note: Locked files can also be caused by backup programs keeping a lock on the file while backing up the virtual machine. If there are any issues with the backup it may result in the lock not being removed correctly.
In some cases you may need to disable your backup application or reboot the backup server to clear the hung backup.
This lock can be maintained by either the VMkernel (ESX/ESXi) or the Service Console (ESX) for any hosts connected to the same storage.
Note : VMware ESXi does not utilize a separate Service Console Operating System. This reduces the amount of lock troubleshooting to just the VMkernel. For example, Console OS troubleshooting methods such as using the lsof
utility are not applicable to VMware ESXi hosts.
Start with identifying the server whose VMkernel may be locking the file. To identify the server:
# vmkfstools -D /vmfs/volumes///
less
command in lieu of tail
. You may press G, once open, to immediately scroll to the bottom of the log's output. Use your arrow, page, or scroll keys to locate the relevant output.kill
command. From the above example, the process ID is 3631
:
is the World ID of the virtual machine with the locked file.
# e
sxcli vm process kill --type soft --world-id 1268395
ps | grep
k
command in esxtop to send a signal to, and kill, a running virtual machine process. On the ESXi console, enter Tech Support mode and log in as root. For more information, see Tech Support Mode for Emergency Support (1003677).The files on the virtual machine may be locked via NFS storage. You can identify this by files denoted with .lck.####
(where #### refers to the World ID that has the file lock) at the end of the filename. This is an NFS file lock and is only listed when using the ls -la
command as it is hidden file.
Caution: These can be removed safely only if the virtual machine is not running.
Note: VMFS volumes do not have .lck
files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.
If the file is being accessed by a running virtual machine, the lock cannot be usurped or removed. It is possible that the lockholder host is running the virtual machine and has become unresponsive, or another running virtual machine has the disk incorrectly added to its configuration prior to power-on attempts.
To determine if the virtual machine processes are running:
# vmware-cmd -l
# vim-cmd vmsvc/getallvms
getstate() = on
, the virtual machine has become unresponsive. To address this issue, see Troubleshooting a virtual machine that has stopped responding (1007819).getstate() = off
, the ESX host may be unaware of the file lock.# vim-cmd vmsvc/power.getstate
A lock on the .vmdk
file can prevent a virtual machine from starting. However, since virtual machine disk files can be configured for use with any virtual machine, the file may be locked by another virtual machine that is currently running.
To determine if the virtual machine's disk file is configured for use on more than one virtual machine, run this command:
# egrep -i
Notes:
.vmx
configuration files for the virtual machines that are visible to the ESX host. A Device or resource busy message is printed for each virtual machine that is running but not registered to this ESX host. You must run this command on each ESX host in the infrastructure or specifically on ESX hosts that have access to the storage containing the virtual machine's files.If the .vmdk file is not used by other virtual machines, confirm that there are no VMkernel or Service Console processes locking the file, per the above section, Locating the file lock and removing it . If a host can be determined however the specific offending VMkernel child process ID cannot be identified, the server requires rebooting to clear the lock.
Note: You can also try to migrate the virtual machine to another host and power it on. If that ESX host has the lock for the virtual machine, it should allow you to power it on.
By this stage, you have already investigated for identifiable VMkernel and Service Console processes which have maintained locks upon the required files, however an unidentified child process still maintains the lock. You have identified the server via thevmkfstools -D
command in earlier steps, the lsof
utility yields no offending processes, and no other virtual machines are locking the file.
The server should be restarted to allow the virtual machine to be powered on again.
Note: Collect diagnostic information prior to rebooting if you wish to pursue a root-cause analysis with VMware Technical Support.
Migrate the virtual machines from the server and restart it using these processes:
For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity (1003743).
Note: If VM won't power on it may be pointing to two hard disks in the .vmx. Remove one of the disk from the VM and attempt to power on.
If your problem still exists after attempting the steps in this article, contact VMware Technical Support and file a Support Request: