Multistream (nginx + RTMP) lags when streaming to Twitch

Hey everyone, I have some sort of a problem and just to want make sure that it's not the servers aka my fault.

I rented a server running nginx + RTMP module and have done this according to the video from @wendell. So far I can't complain, everyone can stream to it without problems. On my streaming PC at home I put all this single streams together like on Team PGP and send it back to the server so it can stream everything to Twitch.

Now my problem starts, sometimes it works without problems but lately the stream stutters a lot in the beginning. I tried to find the problem and I am not sure if I have found it or not but as soon as I turn off the Minecraft server that is running on the same machine the problem seems to be gone a few minutes later.

Can a single Minecraft server cause this stutter problem I have or was it just luck?

The server itself is a VPS running Ubuntu 14.04 on 4 cores with 6-8 GB RAM with 200 GB SSD in a datacenter with 100/100 connection. Since I am still learning all that Linux stuff I am not able to find this problem (if it's caused by the server) on my own and any help is highly appreciated.