Linux for Software Engineering (Not a Distro): CI/CD Edition

[AnotherDev]

@Adubs It’s happening.

A long, long time ago in a forum not so far away, there was a discussion taking place about how to improve certain aspects of the forum. From that, an idea thread and a series of *BSD blogs spawned. After that, the small ember of inspiration seems to have faded. Oh, but fret not, for we have a new challenger! @Adubs has expressed interest in learning some new skills that can increase his paycheck significantly bring him inner fulfillment and satisfaction.

So, I figured I’d give my hand at it. This thread is going to be a work in progress and I welcome others to contribute. I didn’t want to do a blog or wiki because that seems like a self-interest endeavor whereas with this I am trying to create a shared, unified resource from experienced, professional Linux users that have used the operating system in a software engineering capacity. For this particular thread, using Linux in the process of Continuous Integration and Continuous Delivery.

Some of the topics that will be written about include:

Automated building and deployment with Jenkins

This will consist of standing up an environment with worker nodes to fetch, compile, build, and release the software. Probably going to go with the Docker image but, there is a nice repo in CentOS and FreeBSD so I might go the “bare metal” route as well.

Version Control with Git

Might go GitHub and GitLab just to over differences. GitLab is a lot easier to self host but I think GitHub has a lot more accessibility.

Infrastructure as Code

I am choosing Ansible, but if anyone is more comfortable with Puppet or Saltstack, go for it! Professionally, I have just worked with Chef. But I want to learn something new as well as help others see how all this stuff works. If anyone is interested, I have a Chef server that I host that I can go over rebuilding and configuring and showing how to write cookbooks.

Containerization

Build images and deploying images with Docker and Kubernetes. This is something I’m still gaining a deeper understanding on, but I think a L1T Kubernetes cluster would be pretty badass lol.

Monitoring

We’re going to spin up an ELK stack with three Elasticsearch nodes, one Logstash node, and one Kibana node. From here we will grab logs from our dummy applications and parse through them and make dashboards (BUZZWORD OF 2019!). I welcome anyone experienced with Prometheus and Grafana to join in and participate.

There is no rhyme or reason for this, but I think it will be a fun endeavor for anyone to participate in. No need to wait for Devember, if you want to containerize an existing application you have or if you’ve been wanting to learn logging then feel free to use this thread to express your R&D efforts or demonstrate your results or blockers.

---

First things first, we need to get ourselves a Jenkins server.

NOTE ABOUT MY CENTOS SYSTEMS

I upgrade the kernel to at least the latest LTS kernel but, generally, I run the latest mainline kernel. The default CentOS 7 experience is kernel 3.10. If you wish to upgrade you can use this script below (change kernel-ml to kernel-lt for LTS kernel):

For MBR:

#!/bin/bash
yum update -y
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org 
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml -y
sed -i 's/saved/0/g' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

For UEFI

#!/bin/bash
yum update -y
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org 
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml -y
sed -i 's/saved/0/g' /etc/default/grub
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
reboot

Feel free to run each line individually if you don’t want to create a file and chmod u+x it. My Jenkins server is on kernel 5.3.2-1

All of the instructions should work on any CentOS, RHEL, and (probably) Fedora server.

I am using KVM as my hypervisor but you can use whatever you want. VirtualBox, VMware, Hyper-V – Just make sure you set the networking to Bridged to avoid any confusion when connecting to the server or, eventually, the site.

Once you complete the installation, I recommend installing vim and git on the machine.

sudo yum install vim git -y

I am installing the weekly release of Jenkins, but they offer an LTS version as well (the steps are near identical, I’ll include under this).

sudo curl -o /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
sudo yum install java

Java is required for Jenkins to run. Sorry.

For LTS version

sudo curl -o /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
sudo yum install java

Jenkins by default runs on port 8080 (unless you’re on FreeBSD, then it runs on 8081 or something like that, can’t remember but oh how I raged). We’ll need to open the firewall.

sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

If you like IPTables or another firewall then God help you you’ll have to find those instructions on your own.

I set Jenkins to run on startup and then start the service. I’m using systemd because I believe in a pure, unadulterated system, the init.d commands do work, though.

sudo systemctl enable jenkins && sudo systemctl start jenkins

Depending on your system, it could take several seconds for the service to come online COUGH JAVA COUGH.

Hit up your browser and visit the serverIP:8080

For example:

172.16.0.119:8080

You’ll be presented with a initial authentication screen:

Follow the instructions on your server and copy that password:

After that you’ll be prompted to install suggested plugins. I highly, strongly, very much recommend doing this to save yourself some frustration and time.

Important Note

Sometimes Jenkins will load to a blank, white page. Stop the server on your server and start it up. Sometimes these services will fail to install, giving a red box instead of a green box. Don’t panic! (Did you bring your towel?). Restart the Jenkins service and try again. If the installation moves forward and you’re not sure what plugins were installed, check this list. If you’re lost, confused, or unsure of what to do, just ask for help.

Next, you will be asked to create an Administrator. I strongly recommend doing this:

I typically name the user sysadmin or systemsadministrator (cause that’s what you are, baby! No turning back, saddle up and git gud).

Next, confirm the instance URL. In a production, enterprise environment you’ll want this to be the domain name, usually starting with the word “jenkins”, but here the IP is fine. Well, since you brought it up, in a production environment you’d have port 8080 actually be port 8443 and have that forwarded to 443 with an SSL cert.

After that, you should be set. Try closing the browser and logging in with the user you created:

TODO

// write application to have Jenkins build
// setup worker node for Jenkins
// create Jenkins project
// aesthetic things (change the blue button to green, etc)

[Adubs]

Liked, commented, subscribed, and notifications enabled.

[AnotherDev]

// setup worker node for Jenkins

Alright, here is where we’re going to get into some gray area. Some people recommend a service account, some people recommend using LDAP/Active Directory – I’m gonna use the good ol’ Jenkins account that gets created when you install the service

Go ahead and create your worker node. I’m using CentOS – Make sure it has a static IP.

On the Jenkins Master (where Jenkins is installed)

Edit /etc/passwd and give Jenkins a shell (I use /bin/bash, use whatever you want as long as it’s installed. Should be /bin/false by default)

sudo vim /etc/passwd
---
jenkins:x:997:995:Jenkins Automation Server:/var/lib/jenkins:/bin/bash

Save the file and sudo su jenkins into the account.

From here, we’re going to generate an ssh key for jumping into the master.

ssh-keygen -t rsa -b 4096 -C "jenkins-master"

/var/lib/jenkins/.ssh/id_rsa.pub should be fine. I did not give it a password.

On Jenkins worker node

Now head over to your new worker node and create a user called jenkins.

sudo useradd -d /var/lib/jenkins jenkins
sudo passwd jenkins

I use a random password for Jenkins (that you’ll need real quick).

Created an authorized_keys file

mkdir /var/lib/jenkins/.ssh
touch /var/lib/jenkins/authorized_keys

Copy the Master’s /var/lib/jenkins/.ssh/id_rsa.pub file into the worker’s /var/lib/jenkins/.ssh/authorized keys file.

Try and ssh into the worker server from the master as the jenkins user.

sudo su jenkins
ssh workerIP

If you get permission denied or asked for a password, authenticate with the password one time, then try going in with the public key. You might have the dreaded Unix File Permission Issue™.

On the worker node /var/lib/jenkins/.ssh needs to be chmod 700 and /var/lib/jenkins.ssh/authorized_keys needs to be chmod 600. This can be a huge pain if you don’t catch this early. Let me know if you have questions and I can try and duplicate your issue.

Head over to the Jenkins web console and hit up Credentials > Global > Add Credentials (it’s on the left, see second pic).

From there you’ll want to give a description (you can leave ID blank) and enter the jenkins username and then select Private Key > Enter Directly

Next, copy the /var/lib/jenkins/.ssh/id_rsa from the master and paste that into the web console.

Save the file – Leave passphrase blank unless you created one when using the ssh key.

Hit up the home page by clicking Jenkins at the top and then go to Manage Jenkins > Manage Nodes (scroll down) > New Node

Give it a name (I creatively named it Worker01) and click Permanent Agent. Click OK

Description is optional. Executors I always pick 2 to 4:

The maximum number of concurrent builds that Jenkins may perform on this node.

A good value to start with would be the number of CPU cores on the machine. Setting a higher value would cause each build to take longer, but could increase the overall throughput. For example, one build might be CPU-bound, while a second build running at the same time might be I/O-bound — so the second build could take advantage of the spare I/O capacity at that moment.

Agents (nodes that are not the master) must have at least one executor. To temporarily prevent any builds from being executed on an agent, use the Mark this node temporarily offline button on the agent’s page.

For the master, set the number of executors to zero to prevent it from executing builds locally. Note: master will always be able to run flyweight tasks including Pipeline’s top-level task.

Remote Root Directory: /var/lib/jenkins

Label is optional

I select “Use this agent as much as possible” and “Launch Agents via ssh”

Host: workerIP
Credentials: Jenkins User you created earlier

Keep this agent online as much as possible

Select Save and go back to the Nodes page. You’ll notice worker has a big fat red X over it. Click Worker01 and select “Install Agent”

Hopefully you get a long messy STDOUT that ends with

<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.33
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Go back to the Nodes page and see the Red X has gone away.

Congrats! You have setup a worker node for Jenkins! Now we can move onto building a Project and getting some software on there.

[AnotherDev]

Also, I recommend temporarily adding auto refresh when installing plugins:

[candybar]

Do most people run worker nodes by default? For small organizations is a single server usually sufficient?

Thanks.

[AnotherDev]

Do most people run worker nodes by default?

For personal projects I’ve done a lot of stuff on the master node (including writing bash jobs/scripts to just remote in and run a lot of commands). At the enterprise/professional level I’ve always used worker nodes. My last job we had 2 for PHP work and 2 for Java work and at this job we have 2 for Windows and 2 for Linux.

It’s not at all required, though. I went through the motions because knowing how to do it is useful, especially when you start ramping up to bigger and better things, my friend!