This was ridiculous guys.
I'm not too great with AMD CPU's, but BIOS is the way to go. I could turn on my Phenom PC for the first time in months and play around with RAM frequency. Maybe record it. I'll see if I have time, hahahaha.
Off the top of my head, I know that if you're going to increase frequency, you'll have to increase the voltage, and timings as well. I'm pretty sure there's just a CPU to RAM frequency ratio you can adjust for rough overclocking. Obviously if you increase your Reference Clock on the CPU, the RAM will increase based on the ratio.
Sorry for that part of the community have a stick up its collective ass.
Hey, I'll tell anyone to their face, I'm an asshole.
Although, I see the amon amarth shirt and retract my previous trolling.
Honestly, I dont usually see people trying to OC memory in perticular, dude to being so used to FSB overclocking, but think of it this way. You can adjust the timings/memory clock in the bios. The more you up it, the more voltage is required. The easiest way to do it is one of two ways.
Incrementally increase the memory clock, and up the vcore when it becomes unstable
Set the vcore to the highest comfortable vcore, and and up the clock speed again, until it becomes unstable.
When overclocking memory, the only thing here that is really important to you is the frequency (highlighted in red), and the voltage, the rest of the settings are really just fine tuning, and are not really necessary for a software overclock.
Like most people above, I would generally not overclock in windows, as it is much more unstable.
Simply go into your bios, depending on your board, which you stated was an ASUS, meaning that when you load the bios you have memory settings on the first page of extreme tweaker, just scroll down and set a manual voltage, since you have DDR2 memory, you will need to set a voltage of around 2v to 2.1v
Do not risk going higher than that for the moment until you know you have a stable overclock, with a voltage like that, you should be able to set the frequency to around 1066 with that RAM.
Press f10 to save and exit.....
The settings I mentioned should be relatively stable, but as always just play around on your computer like you normally do, and if you get any hangups you may want to increase the voltage slightly.
Note: If you are still keen on overclocking in windows, use the red highlighted frequency settings, and the memory voltage settings on the main page to overclock.