Vega Pro-Tip: Blacklist the ‘radeon’ kernel module. There’s some gnarly bugs where radeon will attempt to initialize a Vega GPU and then panic. Vega cards are supported by ‘amdgpu’, not ‘radeon’.
There’s a few ways to do that.
You can do it permanently by editing /etc/default/grub file and adding the following at the GRUB_CMDLINE_LINUX_DEFAULT section.
blacklist.radeon=yes
Alternatively, you can do it as a one-time thing by adding that exact same thing to the GRUB boot promp. Reboot your system and hold SHIFT during boot to force the GRUB menu to display. Press ‘e’ to edit, type in the blacklist option, and CTRL+x to boot. There’s on-screen text for help during.
As to getting it working on an older Ubuntu release, I’d suggest upgrading. Updating the kernel, mesa, and xorg to get things running is a huge part of a working graphical system. You’re sticking to an LTS release for stability, and then immediately crushing the foundation of a stable by replacing so much.
In the long run, it’ll probably be less of a pain to fix your Samba issue and upgrade than to shoehorn in bleeding-edge hardware into an intentionally old and stable system. Vega is still very much ‘there be dragons’ territory for Linux, but the progress in recent releases has been amazing.
I’ve had really good experiences with Arch and Gentoo for my 2400G. Fedora kernel panics 100% of the time, even aside from the radeon related issues. Debian has an open bug making all releases a no-go for the Vega chipset (Processor works fine). Ubuntu 16.04 is officially supported by AMD, but Vega isn’t supported in the Pro driver as a week or two ago.
Even worse, a bug was found affecting some Linux 4.17 users using the Vega HDMI port as a primary display. 4.18-rc2 seemed to have it fixed.
I’ve shoved an RX580 in and use that to work around Vega’s issues, but I’d probably be doing that anyway for the DisplayPort.