GPU Usage fluctuating, FPS Dropping in BF3. Geforce 770

Just upgraded my PC with an Intel 4770k, Gigabyte Geforce GTX 770 and a Cosair HX850 80 Plus Gold. The problem I'm having in BF3 is on Ultra (and even a few times on all settings low) my FPS will suddenly drop to the single-digits and then come right back up, but it happens pretty often in-game. Looking in MSI Afterburner I see when my FPS drops, so heavily too does my GPU Usage.

I've checked and tested my CPU temps, under 100% load they don't go above ~56 C. I think the max GPU Temps I've found are ~69-70 C. I haven't tweaked/overclocked anything on my system either.

I do only have 4gb of DDR3 1333 RAM so I thought maybe that was bottlenecking me, but I don't know.

I use about 5gb of ram while playing bf3 so that could be it.  Thats the only thing I can think of.