You'd have to be insane to overclock with a stock cooler. Furthermore, I would have to go a lot more in depth than I can on a simple forum post to guide you to a nice, stable overclock
Like william said, overclocking with the stock cooler is a bad idea. At the minimum you will need something like a Cooler Master Hyper 212.
Also overclocking will void your warranty because you run the CPU out of spec. I reccomend you get the Intel Performance Tuning Protection Plan, it is a great deal! http://click.intel.com/tuningplan/
Yes, minimum is a Cooler Master Hyper 212 EVO, but it still gets pretty warm even with a modest overclock. I'm at 4.4ghz @ 1.29v and my idle temp is mid to high 30's. Load temps is mid to high 50's.
I'd definitely save up for something better if I were you.
4gHz, maybe, but not much, if any higher. Really, the thermal mass of the stock coolers just can't handle anything higher unless you have an amazing chip.