just want to know whats the best format for quality and small size?
It's more to do with Bitrate, Resolution, Framerate and Codecs rather than the container (container meaning formats like .mp4, .mov, .mkv, .flv etc) though I don't believe all codecs are compatible with all containers.
So, you can use the H.264 (x264) codec for .mp4 and .mkv files for example. The bitrate, resolution, frame-rate, and compression rate are a bigger factor.
There's a lot of info on wiki about this.
http://en.wikipedia.org/wiki/Video_codec
http://en.wikipedia.org/wiki/Comparison_of_video_codecs
http://en.wikipedia.org/wiki/Comparison_of_container_formats
For software, I'd recommend that you use something like FFMPEG where you can manually set everything so you can tweak it all, but I think it's limited in the fanciful effects department, unless you really work hard at it.
The more "basic" things like adding a watermark, adjusting contrast, concatenating video, cutting, fading in and out is fairly easy and mainly straight-forward.
I'd stay away from things like Sony Vegas unless you really need certain special effects which can only be found in whatever program you look at.
Thanks for the reply
I have been using sony vegas pro 10 and the file sizes i get are huge and i have been rendering them in avi. i mainly make gaming videos but coz i live in australia where my uplaod speeds are dismal, i thought itd be better to render to a smaller size