Slow NFS file server when accessing over VPN on public networks

I am currently trying to setup a NFS file server for myself that I can use at home over my local network and at school through my own VPN. While read and write performance on the local network is great I have been having problems getting decent performance with my laptop on public networks using my SoftEther VPN. So far I have determined that the VPN server hardware itself isn’t the bottle neck as connecting to my VPN while on the local network does not reduce performance.

I think the issue is simply my internet speed, as I currently have about 70 Mbps down and 10 Mbps up which is what my SoftEther VPN is using. To test what my the performance would be like on public networks I tether my laptop to my phone and get speeds of about 30 Mbps down and 10 Mbps up when not connected to the VPN. However when tethered and using my VPN these speeds decrease to 5 Mbps down and 5 Mbps up. This means that when my laptop is tethered to my phone and connected to my VPN accessing PDFs on my server with a size of 2 MB take forever to load.

If I want decent performance with my laptop on public networks connected to my VPN will I need a better upload speed than 10 Mbps for my VPN server, or is there something I am missing that could increase performance?

Thanks in advance.