Here are some pointers from my experience:
There are basically two ways to setup VNC: “headless” and “mirror”.
“Headless” spawns a new X11 server with associated VNC server, not connected to any real output.
This has the advantage that you can resize your VNC server to whatever resolution you want.
“Mirror” mode duplicates a real X11 display as a VNC server, with the advantage that you get OpenGL support “for free”.
Headless is easier to setup, and well-suited for a multi-user experience. Mirror is great for duplicating a screen over LAN(I’ve used it as “network connected extra monitor” in the past).
You should never host a VNC or RDP server publicly. Seriously, I’ve been at hacker conferences where it’s a literal game(with scores and all!) to make creative use of open VNC servers. They are easy to find and easy to exploit. DON’T DO IT. DON’T ASSUME BECAUSE VNC HAS AUTHENTICATION AND ENCRYPTION YOU’RE FINE TO RUN IN ON THE INTERNET. DON’T. (This holds true for almost any externally-reachable service, with maybe the exceptions of SSH and static HTTP servers)
Instead, use SSH tunneling, VPN’s or whatever.
Some alternatives to VNC would be:
RDP
RDP comes from the windows world, but can be used on Linux just fine, as both a client or a server. It has some more modern features compared to VNC, like sharing printers, but usually that’s poorly supported under Linux(and even under Windows, where some features are locked behind enterprise editions and such).
SSH X11 forwarding
Useful to get a single application to run remotely without much configuration.
Performance is not great, especially over slow connections, but you can almost always use this option.
SSH is great at circumventing firewalls and NAT hell due to port forwarding/jump hosts.
Xpra
This is designed to be basically an improved version of X11 forwarding(It works different under the hood, but the user experience is similar, and it can optionally create SSH tunneled sessions as well). You can run a single application or a complete desktop. You can even leave applications running server-side and later re-attach to them.
Xpra brings some important improvements over SSH, such as actual video compression codecs, and automatic selection of codecs(e.g. Xpra will use HQ JPEGs to encode a mostly-still desktop, but will switch to x264 if needed), it detect the available bandwidth and selects appropriate parameters, it optionally shares clipboards, printers, files, sound etc.
Not to mention that it has a HTML5 client that can be used directly in the browser for convenience.
ffmpeg/My devember project
(Don’t actually use this) My devember project has an X11 streamer application that can stream the content of an X11 server to a web browser. It was actually super easy to implement, it’s just a CGI script that responds with some headers and then calls ffmpeg to encode the desktop as MPJPEG(64 lines of server code, could be reduced to ~5 if only ffmpeg output is required).
Don’t actually use it, I just wanted to say that custom solutions and hacks are very much possible, and that ffmpeg can do a lot as well.
Notes on VirtualGL
If you want OpenGL acceleration on something different than a mirror session, you’ll need VirtualGL.
VirtualGL forwards OpenGL requests via a LD_PRELOAD hack to a back-end X11 Server that runs the GL commands, then forwards the generated data back to the application.
This allows you to run most X11 applications with OpenGL support(Some like minecraft don’t work since they load OpenGL libraries in an incompatible way).
Personal recommendation
I’d go with Xpra. Some setup is required, but it’s easy to use once setup, lots of features, very flexible and is easy to secure(use SSH).