Can you verify that the VM's disks are not set as independent?
I think I remember a bug in VMware Virtual Disk Development Kit 5.1.4 that would throw this error if the VM's disks were set as independent. Can you try using VDDK 5.5.4 (I have included our best practice for installing VMware VDDK below)?
If you do find that the VM's disks are set as independent you will need to change this setting or exclude the independent disks from the backup job. This is because independent disks are not included in VM snapshots and therefore are not able to backed up using a snapshot-based (agentless) backup solution.
The best process for installing VDDK we have found is as follows:
1) Download and install VDDK 5.1.4 using all of the default settings. VDDK 5.1.4 will install to C:\Program Files (x86)\VMware\Virtual Disk Development Kit.
2) Download and unzip VDDK 5.1.4 to C:\Temp\VMware-vix-disklib-5.5.4-2454786.x86_64, then copy C:\Temp\VMware-vix-disklib-5.5.4-2454786.x86_64 to C:\Program Files.
3) Once you have vBackup installed set the "VMware Virtual Disk Manager Application Directory" setting to: "C:\Program Files\VMware\VMware-vix-disklib-5.5.4-2454786.x86_64\bin", uncheck "Use same application directory for VMware Disk Mount" and set the "VMware Disk Mount Application Directory" setting to: "C:\Program Files (x86)\VMware\VMware Virtual Disk Development Kit\bin".