I’m trying to be a bit more active on the forum, so here it goes…
So I love all things Cybersecurity, and anything of that sort. However, when I saw what DevOps meant/is/was (trying to cover my bases), it definitely piqued my interest. I watched a short Youtube video this morning about Ansible and have started becoming more interested in the different aspects of DevOps. What I would like to know is, for those of you what work in DevOps, what are your titles? What sort of day-to-day responsibilities do you perform, and how did you get started in that role?
I’m just looking to get insight into the field without watching videos by complete strangers, or reading articles that are just trying to entice someone else into the same company. So @AnotherDev, and anyone else… let me know and I’m looking forward to seeing the responses!
Wow, thanks man. I hope this helps. I’ll answer your questions and then expound a little bit.
DevOps Engineer 2. Lol. The 2 doesn’t mean much, it’s more to define pay structure and escalation order. At my current job, the DevOps Engineer 1 folks do a bit more operational stuff whereas I do more of the development stuff. At my last job, though, DevOps Engineer 1, 2, and 3 had 0 difference besides how long you’ve been at the company. We all had the same responsibilities and the expectations were the same for everyone. They just didn’t know how to promote people.
Today was pretty standard of a busy/exciting day.
There is a huge money maker that people are wanting dashboards for. So, we are working on getting the logging parsed through our ELK stack. As the devs make changes to the logging, we make changes to the YAML file or the .conf files for Logstash. There were a lot of changes on Friday and over the weekend, apparently, so I went through the YAML config file and cleaned that up, changing out the directory paths and updating or changing the tags. I then verified if our existing grok parse setup was going to catch the new logs. It didn’t catch everything, so I created some new entries.
We also went live with a new application. So I had a list of things to do:
Pointed domain to IP, which meant going into Route 53 and making the change.
Merging the code: This was done by sending an e-mail to the devs and refreshing the page until I saw “Merged” lol. They pushed development branch into master, basically.
Deployed to production: we had already configured the Jenkins job, so I just clicked “build now”. I was aware this was coming, so I built the job preemptively. But sometimes you’ll do this on the fly.
Built database instance: Created a RHEL 7 server, installed MongoDB, configured MongoDB, went through our operational stuff (firewall, naming conventions, associating an elastic IP, things like that), and verified the connections were working (went to the site and accessed something in the DB).
Wrote documentation for database: The server and DB info weren’t documented, so I threw that in with the existing $APP documentation.
Setup Nginx caching for application server.
Wrote cookbooks to be able to easily configure all of the above in the future.
Uploaded cookbooks, tested cookbooks, and pushed code changes to GitHub
As I mentioned, today was an exciting/busy day. Some days I am stuck on something that requires trial and error or a lot of research. One day all of the dashboards for ELK stopped working, so I had to track that down. Another day mongo just stopped starting in dev, so I had to figure that one out. Things like that.
Hm… That’s a hard question to answer. I studied Computer Science in college, but I wasn’t interested in software engineering until my third year. I was still wanting to pursue systems administration. Still do, if that matters.
But, somewhere in my third year something clicked. Then I had heard about DevOps from Dr. Nicole Forsgren. That sounded cool, “DevOps”. Developer Operations, while not as cool, still sounds cool. Anyway, I started reading about DevOps, and realized I had started down that path already. I was a multihat wearing tech at the time, but I had formally studied computer science and knew how software was supposed to work. So, I started learning GitHub and creating more robust scripts and applications. It’s hard to interpret what that means, sometimes. A lot of people think clean code or robust means a file in every folder, or a function in every file. But really, if it’s readable and efficient that’s all that matters.
My first real DevOps-y thing was probably automating KVM. With a script I could install a VM automatically on my host. That was pretty cool. Soon after, or maybe before, I started learning about Chef and Puppet, and later Ansible and Terraform. I didn’t use any of those right away, but I knew what they were and what they did. My advice here is pick one and move on, don’t try and learn the pros and cons of each, because some of that is subjective and at the end of the day, knowing one really well is better than knowing all at the surface level.
I found a DevOps Engineer job that wanted experience with Linux, DNS, scripting, etc. I was pretty good with all of those, for the most part, so I applied. I think I got really lucky, partly because of my resume and cover letter but partly because I was super excited to get a call back. DevOps, above anything, is a matter of culture, passion, and innovation. You gotta be excited about the job and excited about learning to do well.
The job was really awesome. I used AWS, Ruby, Chef, Jenkins, and Amazon Linux. There were other tools involved, but that was my stack. I didn’t know anything about any of those before I started. I had setup Jenkins once or twice, so I knew what it looked like. I had also worked with devs that used it all the time, so I knew what it was for. But other than that, I had experience with virtualization, CentOS/Fedora/Ubuntu/Debian, Python, Bash, C++, Java, and JavaScript. Anyway, needless to say I learned a lot in a very short amount of time. I did a lot of pipeline stuff, what’s commonly referred to as CI/CD, which involved Ruby, Jenkins, Bash, Chef, and the AWS-SDK.
After a year of that I thought it was time to explore what else was out there. A sysadmin job at companyA and a sysadmin job at companyB could be two different jobs. I wondered if it was the same for DevOps.
I didn’t apply for the job I have now, I was messaged on LinkedIn (btw, get a LinkedIn profile). The company seemed cool, the pay seemed awesome, and the gig was basically what I was doing with more technology and a broader scope of work.
Looking back, having a goal was really important for me. When I was a support guy at a financial company, I wrote a five year plan. I wanted to “be an engineer” in five years. I hit that goal in two years. My next goal was salary related, and I hit that in a year. If you have something you can easily reference and is specific or, even better, if you can visualize it, that will take you very far. DevOps isn’t 100% what I would sit and daydream about, but it’s 9,001x better than what I was doing before.
So, to summarize:
“How did you get started in that role?”
I learned about DevOps as a culture. Read The Phoenix Project. Read The DevOps Handbook (don’t let it intimidate you, it’s difficult content. Read it slow, power through, and revisit topics you didn’t understand when you’re done).
I wrote down goals that I wanted career wise, lifestyle wise, and technology wise.
I automated complex tasks, such as provisioning infrastructure or uploading a file to a website automatically.
I changed my resume, big time. I threw a projects section on there and all my web sites, GitHub, GitLab, and podcast sites.
I didn’t have a plan B. Arnold Schwarzenegger said it better, but never did I say “if this doesn’t work out I’ll go be a welder or something”. I said “if I don’t get this role, I’ll ask for feedback and improve, then get back to it”.
Get after it man. Don’t let anything stop you. DevOps mindset and skills will take you into InfoSec, too, if that’s something you want to do. Ignore the haters, the people that tell you it’s not worth it or you’ll not make it are the people that hate their job or hate their lives.
Some skills you’ll want to consider are Linux servers, configuration management (Ansible/Chef), Python or Ruby, Bash, and communication. If you’re not a good communicator or good presenter, practice. Talk to yourself in the mirror, practice eye contact. Practice in front of your spouse or family. @kewldude007 has a contest going on, do a ton of research on DevOps and make a video or a PowerPoint.
That’s my story. Mostly anyway
@SgtAwesomesauce Is in a DevOps role, too. I know he did some consulting stuff early on and moved into systems engineering.
We do a lot of things actually, everything from SysAdmin work to Developer work, you have to be well versed, because you’re going to touch pretty much everything short of the actual hardware, having a Developer background reeealy helps. You don’t specifically train to be DevOps, you move on to it, at least from my observations.
What we do day to day depends on what Developers (and the nature of the product) need the most. Last company I worked for, real-time monitoring was an incredibly critical component of the system. It was an online advertising real-time-bidding system (1.5M requests per second, all sub 80ms). You could lose several house mortgage’s worth of money in minutes if something went really wrong.
Naturally, we paid monitoring much attention, with alerting that would automatically trigger the whole system to stop bidding if there was any significant anomaly in finance metrics. Us DevOps would make sure monitoring and alerting was up to snuff.
Some specific databases were up to us to maintain, Aerospike in particular (damn interesting and performant database, I suggest you look up how it works with SSD’s).
Then there were servers themselves and infrastructure as code(namely Puppet), that we would manage. At that scale servers come and go all the time. We’d manage the hardware (that was co-located in another country(ies)), configure everything (latency sensitive workloads sometimes prefer hyper-threading off, namely Aerospike and Nginx sometimes). That alone was becoming a significant amount of work, so we started designing Consul + Nomad based system that basically enables servers to discover their purpose in life and spring to action just from having it’s IP address and hostname set (+ probably a Puppet run, but you can have that baked into an image). Never seen it in action as I’ve left the company at the time.
Now in the current company emphasis is on development environments + CI/CD. When I joined the was one made with Vagrant, but the damn thing was slow to set up, scaling it to resemble production was not an option. So we’ve started implement it on Docker, containerize all the things. Now the thing springs up in seconds. We’re doing shared environments + a little UI project, where developers can enter a name, press a button, and have a personal environment set up for their amusement/testing/destruction on a server, not their machine. A lot of stuff is on PHP, really annoying.
Monitoring and infrastructure as code take the back seat, not nearly as refined as in the last company. The infrastructure is by at least an order of magnitude smaller, and not physical even, there are only a few actual servers. Before DevOps was established, SysAdmins would do stuff by hand, and they still do, the way things are it’s borderline practical to even implement IaC, so we focus on environments. Eventually those containers will go in production. Still a loong way away, this company is not very flexible, there are some certification issues to consider even, due to business nature (we need to checksum our code in some countries, and to notify a certain government whenever it changes), those things will impact our implementations.
A DevOps day is more dynamic than a Developer’s day, with a lot more communication going on. Our stakeholders are Developers, so we work closely with them, solve their problems, get their tickets. Implement systems that they use and they deploy code to.
Pretty random. I was a C# developer (still junior level), on that high-performance system, but I had an affinity to manage the Linux (happened to host the monitoring) stuff of our team, the rest were Windows guys. So one day I just got a suggestion from my manager “let’s make a DevOps team in our unit” (there was already a company Devops team, and we needed a lot of attention). I just said yes. And the rest is history
Naw, thank you man, you’ve opened my eyes to amazing things with this reply!!
That’s how my current job is, they don’t really “promote” people as they just place them into a position that’s different.
Those sound like AMAZING responsibilities that you had for today. That’s what I was looking at getting into, so hearing you talk about DevOps like this, is amazing. I currently work as a Tier I help desk technician, but I’m looking for more of a development-type role or operations like that. I love security, but the policies are a bigger thing that I’d like to avoid in that field, which is what my current job is wanting me to specialize in.
I don’t even know where to start with what you said here, but it’s a huge inspiration for me. I’ll make sure to check out the books you’ve recommended, and I’ll go through some projects to see if I can find some inspiration in those.
Do you have any recommendations for finding things to automate, or scripts to write? I’ve dabbled in Python and Bash, but nothing super advanced (in Python, Bash I love).
My title is Devops Engineer, but I’m more of a deployment architect.
I do a lot of work with Terraform and Ansible. Well, 99% of my work is in either of those tools.
I was hired on as a “Linux expert” for the team, to round us out with someone who knew more about the underlying OS than the rest of the team.
I wound up doing mostly terraform stuff. (meh, I’d rather be working on containers and stuff, but w/e)
Oh, this is a funny one.
I started doing computer support for family and friends when I was 12. Resetting routers, installing printers, configuring wifi, etc… Ya know, geek squad stuff. This was the windows 98/2000 days. Back then, nobody knew what Linux meant.
By high school, I was doing support for local small businesses. Pretty much decided that I didn’t need college by the time I was 16 and I started working towards starting a business. The problem is that it was feast or famine. I’d be flush for cash one month and a couple months later I was down to $50.
I decided to look for more consistent work and i threw my resume up and looked around at linkedin and monster and whatnot. I applied for pretty much everything with “linux” in the title and go a couple bites. Then the interview came.
The job title was Linux DevOps Engineer. I knew nothing about devops, but they wanted an interview anyways. You know an interview went well when you spend 20 minutes talking about cars rather than work related stuff.
Worked there for two years and wound up moving after multiple failed promises from my boss. He was the nicest guy (we keep in touch), but he kept promising me a team and never came through on it. (dammit, now you’ve got me sentimental for the old days)
That leads me to where I am today. I’m working at a company that’s doing big things. I’m not going to say what company, or what “big things” are, because I’m really not sure how much I can say without violating NDA or whatever and it all kinda blends together.
Suffice to say I’m working at a company where we have multi-region and multi-cloud fault tolerance. If AWS’ us-west-2 goes down, we have eu-west-1. If both of those go down, we have GCP in Dallas, I think? It’s some really crazy infrastructure and I’d love to talk about how we did it. I’ll do a talk at one of the conventions once we go “live” or something.
How did I get started?
I’ve always learned by doing. I don’t underestimate the value of reading, but frankly, I don’t have the attention span to read books. I pull out the important snippits from sparknotes and continue past go, collect $200 and proceed to be awesome.
Literally think of a thing you do manually and automate it.
Do you do backups? If not, start doing so. Then automate them.
Automate the testing of your backups. A backup is no good if you can’t restore from it. Have a cronjob or something that kicks off once a week and restores a section of your documents to /tmp from the latest backup or something. Log the restore job to an ELK stack, import statuses into Prometheus, from there, send email alerts if something doesn’t go well. (hell, send email alerts if something does go well)
Honestly, don’t learn Jenkins unless work wants you to. Gitlab-CI is the new beez kneez.
Regarding Ansible, just find a task you want to do and make it work. I recently started writing scripts for deploying Kubernetes nodes on Rock64’s running ARM. (more about that in a few weeks, there’s your sneak peek folks)
I know this is a bit rambley and I’m sorry for that, I think I hurt my back in the gym today. If you have any questions, I’ll be around.
Oh boy, I can second that. You could start a Development company with just GitLab, and never need any other service else until centralised logs and monitoring come into play
I somewhat “finished” high-school equivalent at 21 and began a 3-year training as an event technician. And that’s how to get into DevOps! Okay, srsly, I hated that job but still got self-employed after those 3 years (for some reason) and hated that even more, 6-7 work days a week, 10-12 hours a day. A nightmare and you don’t even make that much money. So 2016 was when I was pretty much done, got a lot of debt, major depression, suicidal, no job, toxic relationship with a bad woman, no future or prospect into what to do.
Had an old 2013 MacBook Air, installed Ubuntu on it since I remembered that I liked to fiddle with GNU/Linux back in school days. So much fun and so much to learn!
So I thought “Maybe I can score a job in IT, any stupid first level tech-support would be fine and sufficient, I only need to make enough money to live and pay my debt, that’s all.”
After I did LPIC1&2 Certs I got hired right away and worked in 1&2 level support for a hosting company, made more money than I’ver ever had.
Now I make almost the double of what I got when I started in 2017, hopefully actually doubling my salary in 2020 so I can hit the milestone of doubling income in 3 years.
Anyway: I have no “real” education in IT, I haven’t studied Computer Science or anything. It’s all me, some machines and keyboard I/O which sometimes is very useful because I’m not stuck on how things should be but it is sometimes very frustrating since I don’t know what I don’t know and don’t even know what I’m searching for or trying to find.
So that’s “my story”.
In the last job and the current job, I work somewhat close with Devs on either components, micro-services or a software product. I’m therefore a little bit in the loop on CI/CD and the tools to make DevOps happening and see this as my next logical career step. Currently, I’m on Level 3 Support & Consulting, Team Lead and internal Processes & Projects, so I’m busy developing some soft-skills outside of pure technical topics and try to get into the DevOps Skills on my own time.
The developers I’m working with have all Developers as job-titles but are wildly organized in different teams with different roles and tasks and I’m afraid I can’t talk much about it, mostly, because I have no idea since I’m not part of Software Engineering which is the overall department that has the underlying teams of developers.
Hopefully, I could help you a little bit with the provided info although I think @AnotherDev and @SgtAwesomesauce have much deeper insight in DevOps since they work in those fields while I’m the wildcard runner up to it, I guess:
Awesome, man! Looks like there are other great ideas in this thread too that should help you out.
Lol yeah, it’s kind of annoying, especially when I ask in every interview is there opportunity for growth. I need to get more specific with my question.
Hey man, that’s how I started, and that’s how a lot of people I worked with started too.
Nice. The best thing I can recommend is, regardless of your path, definitely T-shape or become familiar with multiple technology. By that I mean, if you go into Web development and then DevOps, don’t forget the networking or server component. On the other side, if you get into Sysadmin and infrastructure, take a look at what the devs are building and learn how it works, learn some JavaScript or C# or Java so you can read the language your counterparts are writing in.
Keep doing what you’re doing. Spend at least five hours a week learning something new. You don’t have to ace or master something in that time, but if next week you’re better at understanding something than you were last week then you’re ahead of the game. A lot of people in the industry go home at five and shutdown. Nothing wrong with that, but continuous education is vital for continuing in your career.
I don’t even know where to start, but here it goes…
It’s nice to hear about this, as I’ve been working in IT (help desk, jr. sysadmin-type role, etc.) for about 11 years now or so. The developer portion is the big part it’s sounding like I need to work on.
I’ve seen this a few times, and definitely intrigues me more, I like the theme!
This is a big reason why DevOps interests me so much, is that it’s looking at things from an automation-type perspective, and I cannot WAIT to get into that more and more. I never knew that’s what interested me, until I started working my current MSP and we have 0 automation whatsoever.
I believe that’s actually what I’m looking for haha, but the fact that it’s possible to get to that point is super encouraging, thanks @DevBlox!
Haha well I’m glad you finally had an opportunity to respond, I was certainly waiting for it!
Is that a specific title I should keep an eye out for, or is more something that your company just came up with to fill a specific niche?
Just to make sure I’m on the same page, this is Terraform right?
I love Linux stuff myself, out of the team that I work with, I’m supposedly one of the only ones with applicable Linux knowledge, so I’m hoping I can continue to grow on that in one way or another.
The two jobs I had before this current one for my MSP had interviews like that. I should’ve known I was just being crazy when I decided to switch from there to here, but I guess it looks better on a resume’ y’know?
Haha my bad… though seriously I get it, super hard to not miss them when they were so great!
Unfortunately that’s where I’m at too, my wife is a HUGE reader, and I enjoy some entertainment reading, but the study guides (especially for Security+ right now) are just so hard to get through. I’m a practical, hands-on learner, don’t get me wrong, I always learn well in a classroom, but just reading hardly gets me anywhere. Projects are what help me learn the most every time.
I do perform backups with my wife, but it’s more like cold storage external HDDs, and they’re performed monthly. I’ll have to see if there’s an automated part I could add to that.
This is something I’ve honestly been pushing for my current company to start doing. We have a “proactive backup manager” that performs this manually everyday (I think) and it’s just honestly so ridiculous. You could have everything automated, but they’re “afraid we won’t be notified if something goes wrong”.
No worries in the slightest, I didn’t find it ramble-y at all, in fact very informative! I hope your back feels better, I know the struggle, far too well…
Is GitLab also the one where you can create a website? I think I started one of those myself… I’ll have to look into when I have some time.
Y’know, I had actually never heard of those before, so I’m adding them to my list of things to look into for sure!
That’s the same boat I’m in, which is actually why I’m looking at DevOps, because as I stated a bit earlier, it’s seeming like my work is wanting me to be certified in Security+. but more-so for the policy-type stuff, not the actual pen-testing, architecture, etc.
Your “story” is awesome @BookrV, thanks so much for sharing it!
You’ll have to make a post about that some time so we can all keep up on with what you’re learning, it’s always fun to read things like that!
Hands down you definitely did, truly thank you for your reply!
Haha yep, and apparently since I’ve moved across the country, they now use my location as an excuse to keep me at my T1 position instead of promoting…
That’s my plan, the hardest part is finding where to start and keeping it going. My biggest problem is, I have so much that I get intrigued by, but then all of the “requirements” and whatnot start me at a different point and then I get lost and/or overwhelmed with everything.
I’m gonna start adding this to my weekly routine, making sure I get items added in as I can.
Again all, thank you so much for your responses! I am just so overwhelmed with the support and replies from you guys, it’s amazing. Also, to have a topic that has some sparked some activity makes me even more excited haha.
GitLab used to be just an alternative to GitHub, solely for version control via git. There is a feature of making static pages, I think that’s what you’re referring to, unless you’re referring to repositories as pages (those are not just pages, that’s not what they’re for). Now GitLab has a feature, where you can create automation scripts in a special file in your repository, and does it very well at that. This basically means that instead of having two tools (eg, GitHub and Jenkins), you can just have one, that does both of their jobs. And it’s directly integrated in the same system/UI, meaning not only you have less stuff to look after, you also get a killer workflow. Important to note that you can host it on-premises (inside company network) if you need to.
From a job perspective, ignore that for the most part. I’ve never met 50% to 70% of a job requirements.
If you mean from a learning perspective, you might be surprised how it all comes together. Learning programming made me better at Algebra and Calculus. Learning FreeBSD made me better at RHEL and Debian.
Look at the common things in a job or role you want and make a list. Then, pick one and start on that one. If you choose Python, for example, but you’re also interested in networking, then once you get the basics of Python use it to create networking tools or web servers. Use Linux as your system when developing or deploying these tools and you’re covering three good bases so far. Some things are hard to retain because you only encounter them once or twice every year or two. You’ll get really good at certain things fast but some things just take experience, like with everything.
Definitely man, I used to do way more than that, and some weeks I still do, but for the most part an hour a day is a great pace to keep learning and moving forward. If you get in the zone then yeah, go on that 3 hour sprint.
Grow a beard, seriously, everyone knows that whiskers = knowledge.
Get a Macbook or Surface Book. Either is fine, you will only end up running GNU/Linux via convoluted means.
Learn phrases like “Awesome epic shit” perhaps you just deployed something amazing, or more likely, had an awesome epic dump after the 15 lattes you drank after staying up all night to help the moron app developers.
Stickers, it’s all about the stickers on your laptop… correct orientation is completely optional.
Thanks for explaining that! I’m not sure what I was thinking of then, but maybe something GitHub introduced at some point, I dunno, neither here nor their anymore.
Thank you so much for the encouragement! That’s what I’m trying to do, just pursuing and trying to figure out if this is something I’m even capable of.
I was actually meaning learning mainly, the job perspective I just push by most of the time. I totally get how everything starts working together, but I keep being told to bounce around from subject to subject…
I actually only ever used to use Linux. Had a complete Linux setup, then got broke, got a free Mac, and just haven’t really used anything further from that point. I’m interested in re-purposing my gaming system into a Linux server for homelabbing, but not sure if I will.
Yeah currently I’m doing a free udemy course on AD just to sharpen my skills with it and all, trying to make sure that I learn at least something this week, like we talked about haha.
@Dynamic_Gravity I actually had a surprising amount of hits when I searched for the term “DevOps”, but I didn’t fully look at each position to see if they were truly related.
If you have any questions, I’ll be around.
