I'm having the same issue, and the lack of activity from the support team after all this time worries me.
I've searched alot on the issue in the last weeks, but there's not much infos to be found.
There's another post here with the same error message, in an earlier version here.
The solution was to disable SSL verification using a registry key.
VMware's vsphere-vddk-65-programming-guide.pdf says:
SSL Certificates and Security
The VDDK 5.1 release and later were security hardened, with virtual machines set to check SSL certificates.
On Windows VDDK 5.1 and 5.5 required the VerifySSLCertificates and InstallPath registry keys under
HKEY_LOCAL_MACHINE\SOFTWARE to check SSL certificates. On Linux VDDK 5.1 and 5.5 required adding a line
to the VixDiskLib_InitEx configuration file to set linuxSSL.verifyCertificates = 1.
As of VDDK 6.0 both SSL certificate verification and SSL thumbprint checking are mandatory and cannot be
disabled. The Windows registry and Linux SSL setting are no longer checked, so neither has any effect.
My guess is that vBackup just can't work with ESXi 6.5 right now, because SSL verification is probably mandatory now for ESXi 6.5 or requires VDDK 6.x which has it mandatory anyway.
I'm not 100% sure, but I think I've read something about changing the default ESXi SSL keys, but I'm not sure if it's enough to get SSL verification working or if it requires a modification inside vBackup. I'm lacking spare time to investigate further on this, and I've been running without VM backups for over a month now.
I'm debating on waiting some more for a fix (again, the lack of activity from the devs worries me) or spend my time switching to an alternative instead, like XSI Backup which they claim they have version 6.0.11 compatible with ESXi 6.5 since november.
Hopefully the devs will show some good news (at least, please confirm if vBackup is supposed to be working at all -or not- with ESXi 6.5) in the next couple days/weeks. vBackup v5.0.0 (pre-release) has been in private beta for almost a year...