My home is set up with the router (WNDR4500) connected directly to the modem along with a FireTV, XBox, misc entertainment stuff, etc. There is also a packet switch (TL-SG108) upstairs with a line to the router that brings internet to the 3 PCs in the computer room. The internet in the computer room randomly glitches out and slows down ten-fold, but the equipment connected directly to the router can run at full speed, and bypassing the switch upstairs will also mean the applicable PC returns to normal speeds.
When you bypass the switch I assume you are using the line that feeds the switch from main router. If so than yes your switch is probably having issues. What model is it?