Assuming Centos and enabling whole disk encryption.
If you disable logins and only provide access via ssh with public key on the client side and a pass-phrase....
Is there any obvious point of entry for the vps provider?
well seeing as how they most likely have physical access, that point is moot.
the have access to your "physical disks" so encryption only stops them from taking a blind peek at your data. Nothing is stopping them from breaking the encryption.
They also have console access which means they can login with TTY login
Waaayy too tinfoily, no one is going to break into the datacenter and take the security guards as hostages to hook up a serial console and steal your disk, if you have that kind of issues you have serious problems or pissed some very very very bad guys and should run for your life.
But you will be fine, just lock it down: use ssh keys, disable root logins, install fail2ban, setup your iptables rules to block connections you don't need. You can also install some packages to email you server logs, look into snoopy, logwatch and integrit. Most important tip I can think of: do your security updates on time.
If you really want to lock it down from the internet, you can implement port knocking: https://moxie.org/software/knockknock/