Weird modem behavior

So my ADSL modem has been acting up recently. Whenever I have it hooked up to ANY router, the connection drops out every 20-30 seconds. By dropping out, I mean the speed goes down to zero; it doesn't disconnect. Here's the weird thing though: I tried hooking up the modem by itself to my computer, and the connection was pretty stable. It was also stable for at least 3 minutes while I was downloading a game on Steam, but immediately crashed down to zero once I opened a Youtube tab (and this for the whole network, not just my computer). I already tried changing the DNS server but that didn't do anything. Anyone got any advice?

As I understand, this is your modem? How are you handling the connection between the modem and the next device? Do you have it setup as a bridge (aka passing PPP over to the next device to deal with)?

Yeah, it's set in bridge mode and PPoE on the router side. I'll borrow my neighbor's router sometime just to make sure it's a modem issue, and not a router or phone line issue. I already tried swapping routers and that didn't help. I also already tried swapping out all the cables, and that didn't help.