It's time to explore, but I need a map!

I program for a living and my degree is in computer science but I feel that up to this point I have learned just what is necessary to do what ever job I held at the time. I have started to actually become really  motivated to explore and learn more about my field and not just know how to program in a few languages and have an understanding of algorithmic approaches to problems from a mathematics perspective. So have a quick question for you guys.

I am really interested in hacking. Not from a malignant standpoint where I would want to bring harm I just realize that in order to pull off some of these complicated hacks that people have come up with the creator had to have a really deep knowledge of network architecture, computer architecture, programming (probably on an assembly level) ect...

If you were to recommend three or four books to start with to go down the road of being able to have this kind of understanding and have the ability to find vulnerabilities and understand how things like bot-nets happen and are created, where would I start? Where did you start?