Is it feasible for me to switch?

You don't need 4.9, but having 4.4 or later will make it A LOT easier. Performance benefits are negligible across kernels, the difference is in the ease of setting it up and how reliable it is.

Absolutely, you can without much pain. You shouldn't have too much difficulty, but there are a few tricks that need to be done to make your 1070 drivers work in the VM. Nvidia intentionally disables them if they detect they're running in a VM on a non-quadro card.

Now, if you want to learn more about passthrough, have a look at these two articles I wrote:

That should help you get started!