I'm having the same issue, and the lack of activity from the support team after all this time worries me.

I've searched alot on the issue in the last weeks, but there's not much infos to be found.

There's another post here with the same error message, in an earlier version here.
The solution was to disable SSL verification using a registry key.

VMware's vsphere-vddk-65-programming-guide.pdf says:
________________________________________________________________________________
SSL Certificates and Security

The VDDK 5.1 release and later were security hardened, with virtual machines set to check SSL certificates.

On Windows VDDK 5.1 and 5.5 required the VerifySSLCertificates and InstallPath registry keys under
HKEY_LOCAL_MACHINE\SOFTWARE to check SSL certificates. On Linux VDDK 5.1 and 5.5 required adding a line
to the VixDiskLib_InitEx configuration file to set linuxSSL.verifyCertificates = 1.

As of VDDK 6.0 both SSL certificate verification and SSL thumbprint checking are mandatory and cannot be
disabled. The Windows registry and Linux SSL setting are no longer checked, so neither has any effect.
________________________________________________________________________________

My guess is that vBackup just can't work with ESXi 6.5 right now, because SSL verification is probably mandatory now for ESXi 6.5 or requires VDDK 6.x which has it mandatory anyway.

I'm not 100% sure, but I think I've read something about changing the default ESXi SSL keys, but I'm not sure if it's enough to get SSL verification working or if it requires a modification inside vBackup. I'm lacking spare time to investigate further on this, and I've been running without VM backups for over a month now.

I'm debating on waiting some more for a fix (again, the lack of activity from the devs worries me) or spend my time switching to an alternative instead, like XSI Backup which they claim they have version 6.0.11 compatible with ESXi 6.5 since november.

Hopefully the devs will show some good news (at least, please confirm if vBackup is supposed to be working at all -or not- with ESXi 6.5) in the next couple days/weeks. vBackup v5.0.0 (pre-release) has been in private beta for almost a year...

Thanks for the (quick!) reply. This is great news!

Sorry for the "lack of activity" part (I didn't mention "unresponsive support"). Admittedly I didn't open a support case myself (I felt that was a general issue so you were already aware of it), but I was referring to observations made while trying to find a statement on ESXi 6.5 compatibility (no offense):

-latest news in news section on the product page is announcement of v4.0.2 release (about 1 year ago)
-latest tweet on your twitter account is dated august 2015
-screenshots on the product page are dated v0.2.4 (5 years old)
-latest public product upgrade in the download section is almost a year old
-new ESXi 6.5 has been out for about 2 months and there's ("was", until your last post) still no public statement about product compatibility (neither to confirm it's compatible, nor to warn customers to stick with ESXi 6.0 until new compatible version is released).
-AndyP's post here about issue with ESXI 6.5 has been left unanswered for two weeks. I wish you had replied to his post two weeks ago to confirm the problem with ESXi 6.5, that'd have saved me (and most likely other customers) alot of headaches ;-)

I work for a software development company myself so I know sometimes things don't get fixed as fast as we'd like (also sometimes not as "cleanly" as we'd like).

I'm glad to see you are actively working on a solution.