Challenge… Accepted… 
So, you want to be a sysadmin, eh?
You think you have what it takes to ascend past the day to day I.T. gig? Are you tired of building computers, tweaking operating systems, and working within the rules of your parameters? Ready to design, innovate, and build architecture that powers your entire company across 14 countries?
Well then, let’s do this.
Table of Contents
- What is Systems Administration?
-
How can I get into Systems Administration?
I. Technical Writing
II. Google-Fu
III. Enhance Your Calm, John Spartan
IV. Networking
V. Systems
VI. Virtualization
VII. Configuration Management - Education
- Bringing it all together
- Projects
- Getting Your First Gig
- Conclusion
What is Systems Administration?
First and foremost, systems administration is about solving problems. Mathematicians use functions and formula, Data Scientists use prescriptive analytics and reporting services, the sysadmin uses systems, networks, and software stacks to solve problems. If you are looking to build awesome monitoring solutions, stand up servers for cutting edge applications, and experiment with the latest and greatest tech, this gig might be up your alley. But, unless you’re in R&D (lucky duck), those are secondary objectives to keeping the business online and solving problems for development, accounting, marketing, etc.
How can I get into Systems Administration?
A common path for becoming a sysadmin is join a help desk, become desktop support or field operations for a company. This is right and wrong, depending on your experiences and availability of computer hardware.
What are the pros of following a more traditional path?
- You get your foot in the door. It has been said for decades that it’s not what you know, it’s who you know. What easier way to mosey into a sysadmin position if you’re already in the I.T. department doing help desk, application support, or desktop work?
- You will have a unique perspective. Before making that potentially chaotic, unpredictable change on Friday, you remember when you were a young chap on the Help Desk, coming in on Monday morning after NetScaler had been upgraded. Your hands begin to get a little clammy… 14 hours. That was how long the phone rang on your extended shift. Non-stop, as soon as you hung up the ringing started again. 14 hours of repeating yourself, hearing the same frustrated claims of “I have to work!” and “Do you even know what you’re doing?!” – It was a rough day, and even years later it still sticks with you. You turn back to your lab workstation and test a few more profiles, open up a few more applications in the new environment, paying it forward and making sure that what happened to you doesn’t happen to the current team of Help Desk employees…
Not always the case, some changes go by unnoticed. Especially when you have a great change management and ops team. But! Having experienced the effects of a dramatic change on the users and the lower level I.T. folk can have a profound impact on how you work in a more senior role. Something to consider.
What about some of the cons?
- Often, the tasks in Help Desk and Desktop Support are completely unrelated. You learn troubleshooting methodology, and you can build a PC in 3 and a half minutes flat… But what happens when a cluster goes down? Or when users are getting routed to the wrong RDP server? Being able to replace a hard drive and empathize with users is great, do not think that those are not valuable skills to have. However, when you get into the big times, most of the interaction is with upper management, other teams that are building or testing the product you’re supporting, and vendors.
- Delaying your destiny. What if the company doesn’t promote from within? You’ve spent two years absorbing everything at this company, busting your tail and being a top performer… Only to find out they brought in a consultant or outside hire to fill the empty sysadmin spot. It sucks, but it happens. That time on Help Desk could have been spent working with lab equipment or building a backend for your friend’s website.
So, what’s the punch line? How do I become a sysadmin? What are the skills I need?
Technical Writing
Document, document, document, document. Learn to document everything. If you screw up, document it. If you succeed, document it. If you spend 30 minutes Googling something and find an answer, document it.
Don’t just write “I f’ed up”, “I did it!”, or “<insert hyperlink here>”. Document WELL.
10:30 Aug 1, 2015 Company Intranet DNS Change
Changed DNS Server – 10:30
Set DNS for public facing – 10:34
Reboot Server – 10:44
Site Outage – 10:55
Server no longer accessible – 11:00
Opened console access – 11:06
Reverted DNS – 11:09
Testing Site – 11:10
Site Online – 11:12
DNS was written incorrectly – 11:15
Entered correct DNS – 11:18
Rebooting server – 11:20
Site and server online – 11:30
Site outage occurred due to new DNS IP Address being recorded incorrectly. Implemented admin approval so we have a second pair of eyes when a major change to a server is happening. This works similar to having an admin approve Pull Requests on company GitHub. See John for being given admin rights to approve change.
Building lab environment that mimics prod. Will be completed Aug 10, 2015. Using 10.12.1.140. Adding to assets page for future reference.
The above example is wildly exaggerated tale of someone making a change to a DNS server on a company site and the IP Address being copied incorrectly. It’s short, sweet, and to the point. You will not always have such an easy documentation process, and sometimes using short sentences and even paragraphs will be necessary. TAKE SCREENSHOTS. This will save your butt, in more ways than one. In the biz we call it CYA. Technical writing is an asset to any company. But, if you’re a sysadmin or engineer and a great writer, you become that much more valuable. With this comes with research.
Google-Fu
Learn to search and hack a search engine. DuckDuckGo has bang(!) to query a particular site. I think Google still accepts site:website.com “search parameters” to filter. Search Technet and other sources too, learn to chop up and hack your parameters to find the information you need instantly. This becomes insanely valuable. Kind of funny, before I jumped on a Web Dev team (long story), I thought only sysadminds/I.T. techs “googled” their job. Nope, devs do it too! Don’t feel bad if you don’t know something. You’re an engineer, it’s not your job to memorize thousands of things, your job is to find the answer and implement the solution.
Some great sites to assist with this:
Along with writing you are going to learn how to read. Reference documentation. The solution is there. It may be clouded with technical jargon, but don’t rely on videos or handholding to get the job done. Get technical. Dig into the documentation. Don’t be afraid to break something. Which reminds me, you will break something. In production.
Enhance Your Calm, John Spartan
A very important, often forgotten part of the job is the ability to stay calm. It does take practice. As humans, it is our nature to panic, fly off the handle, and press the big red button. Learn to count or sing or hum softly. Practice meditation or visualizing success. You will probably freak out at one point, that is inevitable. But, once you learn to control it, and reign in the sense of dread or panic, you quickly realize that this is your job. “These things happen. That’s why you have a job. Let’s do this.” – That’s what you’ll tell yourself, and after giving yourself 3 to 5 measly seconds of panic and grief, you’ll gather your composure, get to work, and solve the problem. The trick, too, is to not let others control your panic. Rarely will it be you and another team of keen, cool engineers in the room when something goes down. You’ll have executives or managers screaming and flailing. Being calm is infectious. Your cool under pressure demeanor will soon spread to others, and it will gain you a reputation of being a clear headed, logical employee.
Networking, and you can too!
Next, we want to brush off that Network+ book:
Networking. Even if you’re a storage admin, systems admin, infrastructure… Don’t matter. Learn networking. I’m not talking about getting your CCDA or anything, but you should know about subnetting, basic ports and protocols, DNS (DNS is super important, even as a Cloud/Systems admin), routing, VPNs, and VLANs. This will be the roughest road for you, especially if you don’t come from a networking, telecom, or NOC background. But, networking is the core of everything.
Enter Sysadmin
Find a system to administrate:
Learn a system and learn it well. Windows Server, CentOS, Ubuntu, Debian, FreeBSD… Learn a system and learn it well. Do everything with your system. Create a file server, create a database server, create a web server, delete everything and then automate those server installs. You’ll want to bleed into scripting and eventually some development practices. You’re entering the realm of systems administration at an amazing time. The craft is changing, and the thin line between ops and development is becoming even thinner.
I’ve worked with some curmudgeons that refuse to consider anything more than Python or PowerShell one liners, but times are a-changing. On the other end of the spectrum, I’ve worked with admins that are building their environment in Go and spinning up containers to serve company infrastructure as processes rather than hypervisor clusters. It is a very abstract, foreign concept at this point, but with Infrastructure as Code and the Devops movement, I can definitely see this as the new generation of systems administration. So, learn your system and learn a scripting workflow. Go, Python, PowerShell, and Bash are phenomenal tools for the sysadmin. This will take you onto Virtualization and Configuration Management.
Virtual Reality
So, becoming a systems administrator involves buying a server for each task. You want to rent a huge space, and buy a box for each server you need. Two web servers? Buy two servers. Two domain controllers? Buy two servers…
Almost had you, didn’t I? Bah! Curse you and your intelligence. The affordable cost of hardware and power of virtualization has changed the way servers are stood up. With AWS and Azure booming, it’s changing even more. For now let’s assume that you’re going to have to work with some physical gear. Virtualization is a great way to maximize the utilization of those resources. Experiment with any and all that you can get your hands on. Hyper-V, ESXi, KVM, and Proxmox are free! FreeNAS and Synology allow for VMs now, too. Pretty much anything you can get running an operating system can host as a hypervisor.
Once you pick your hypervisor, start learning the strengths and weaknesses. Use PowerShell, PowerCLI, and Bash to automate building VMs. If you have the resources and money, setup clusters. Power down one of your hypervisors and test failover. You can do a lot with an i5 or i7 and 8GB of RAM. If you can afford it, get a rig with a quad+ core processor (or tw) and 16+ GB of RAM. You’ll have a lot more to work with and can emulate enterprise production environments.
Configuration Management Systems
Cookbooks, Playbooks, and Puppet Masters oh my!
Despite industry trends and news, Configuration Management has been around for a long time. CFEngine, an open source configuration management system, has been around since 1993! HOW CRAZY IS THAT? To quote the Cylons: “All of this has happened before and will happen again.” – Heck, even “the cloud” isn’t a new thing. Distributed computing has been around since the 70s! Anyway, as with above, pick a system and learn it. Puppet is the more mature, in my opinion, and it comes with an awesome “Training VM”, with a series of “quests” to get you on your journey.
https://puppet.com/download-learning-vm
Looks like Chef has something similar: https://learn.chef.io/#/
This will come in great use once you start scaling with your successful company. It will also promote you to the “Lazy Admin”: Great sysadmins only do something once. Then, they automate it.
Too Cool for School
Education and Certification
Is it necessary? In my experience, I have seen practical application and enthusiasm accepted over education and certs. It helps to have all of the above, but definitely get your hands on some tech to practice. If you want to specialize and have some credentials, I can’t tell you what to do. I can offer some recommendations:
Microsoft: MCSE for Server 2016
The MCSA is the prerequisite, but if you’re going do it might as well go all the way. These tests are brutal. You can probably wing it just on memorization the first two tests, but the last MCSA and the MCSE tests you’ll probably need some experience troubleshooting and working in a Windows Server or Hyper-V environment.
Linux: RHCE
RedHat Certified Engineer. You’ll need the RHCSA (systems administrator) to get the certificate, but it’s only two tests. The tests are hands-on. You get a terminal shell with X-Window and a few hours. Once you complete the tasks, the server must survive a reboot. After that, you get your score and your cert, ideally.
AWS, Azure, Citrix, VMware, Puppet, and Oracle all have their certifications too. I would think hard before committing to one of those, because you start to isolate yourself down to a path of specialization. Nothing wrong with that, but you have to be happy with the stack you’re working with.
There are also various degree plans out there. Management Information Systems, Computer Science, Information Technology, Computer Information Systems… I would read through the curriculum before committing. In my experience with the University of Texas system:
MIS/CIS – Business degree. Using technology to solve business problems. Exposure to database management systems, computer networks, object-oriented programming, web development, and information security practices. All of the above are wrapped up and mixed with Economics, Accounting, Marketing, and Organizational Behavior.
Computer Science – Math degree, not a programming degree. Programming languages are tools to solve problems, in line with algorithms and data structures. You tend to specialize in the end of your tenure: Artificial Intelligence, Programming Languages, Computer Graphics, Human and Computer Interaction, or Embedded Systems.
Information Technology – Certification paths serve as the curriculum for the degree. Get exposed to a lot of modern technologies. SQL Server, Cisco switches and routers, Microsoft Servers, Linux servers and desktops, Windows desktop troubleshooting. Sometimes the school will pay for your certifications as well.
All of the above can assist in your journey to becoming a systems administrator.
But, you will never stop learning.
Technology is always changing. That’s what the ads on the radio are always telling me. But, it is true. Once you get comfortable with a stack, if you get a promotion or a new Director of IT, you’re probably going to have to learn another stack. If you don’t think you can commit to a lifetime of learning, this might not be the right job for you. There is a lot of opportunity, and if you’re good at learning then the tasks ahead are not going to be as bad compared to someone changing careers and has been out of school for 20 years.
Come Together!
Still reading? You’ve made it this far, and still think you want to get the ball rolling on becoming a sysadmin? Good, good, let the engineering flow through you. Now you’re asking how can you take all of the above and simulate being a sysadmin?
What a great question!
- Setup a hypervisor
- Setup a configuration management (Puppet, Chef, Salt)
- Setup DHCP and DNS servers for your lab (do this with as little interaction as possible, automate!)
- Implement user management and authentication (Active Directory, LDAP + Kerberos, etc)
- Set up replication and failover (LDAP, DHCP, AD/DNS)
- Implement Load balancing on the DHCP or LDAP servers
- Implement monitoring solution (Zabbix, Icinga, Nagios, etc.)
- Configure those monitoring solutions to use SMTP to communicate alerts
- Write scripts to pull system resources and output them to .csv or .html
- Setup file servers using NFS or SMB
- Setup a “Devops Server” (Jira, Jenkins, TravisCI)
- Setup web servers (Nginx and Apache, do both)
- Force web and devops servers to use https
- Auto-renew certs for encrypted servers
- Build templates for workstations for different user groups
- Test that, build a developer workstation and an accounting workstation
- Create roaming profiles, and test that
- Take a project off github that is functional and build an environment for it
- Setup automated backups (Veeam, Bacula, etc)
- Build another environment to test those backups
- Restore a prod server from a backup
- Document your entire process in a blog or wiki
- Keep going!
Fun Projects
Don’t stop there! That list will get you started. Check out OpenStack:
Check out NextCloud:
So, what’s next?
Get a job, ya bum!
Yikes, not sure what came over me. I’m sorry 
But, now that you know what it takes, you’ve built up some of the appropriate skills and experience, it’s time to start looking around your area and see what’s hot.
Those are all great resources, and if you throw your resume on there, you’ll probably get harassed by e-mail and phone. Sometimes they’ll be bleh, but other times you may get hit with a surprise.
I also recommend getting a LinkedIn profile. This will allow to network with people in your company, your field, and in the industries you’re looking to be in. There are tons of groups on LinkedIn to join, malware analysts, systems engineers, cloud architects, it has a lot of potential. You can also search for Jobs on LinkedIn. My last two or three gigs I’ve found on LinkedIn. One job I didn’t even apply for!
Check around for a hacker association or meetup group in the area. I have a few in the area, and when I’ve gone it has been a fantastic experience. The price of admission is often a cup of coffee or a beer, and you get a topic of the week/month, and usually a 15-minute job board, with people willing to take resumes or hear your elevator pitch.
Epilogue… Until we meet again
The road can be long. Don’t give up! Becoming a sysadmin takes dedicate and can be a bit of a process. With hard work, passion, and continual progression you can make it. There are thousands of departments out there with underutilized resources. There are people that are apathetic and couldn’t care less about the job. Stand tall, rise above the rest, and you will succeed. Once you care about the tech, the job is easy.
Revisit this as time goes on to see new and exciting projects to build and experiment with, as well as evolving advice and tools of the trade.