the company I work for just ordered a blade server HW with an AMD Epyc CPU (24 Cores). Now we are not sure what Linux System would make sense to use.
The server is mostly for building C and C++ software. The biggest project to build is Yocto. The build itself will be performed inside docker container.
Because of the fact that this server has ECC RAM (64GB), ZFS could be an option. There are 2 PCIe4 SSD’s, one with 256GB for the OS and one with 3.4TB for the work.
In the end I have to administrate this system and I have never used ZFS. (I’m a SW developer, not an admin.)
If ZFS is the way to go, that does not leave many distributions to choose from. I don’t want to add ZFS my self.
If ZFS does not make sense, then debian might be a option, or maybe not ?
As my desktop OS I use Manjaro, not a system I would use for a server though.
Any thoughts on that?
Thanks
PS:
I searched the forum but could not found a question like this.
I took a brief look into Proxmox but I could not found an out-of-the-box docker support. Somewhere I found a documentation how to install Docker in Porxmox, somehow running it in a VM. But maybe that was an old document. That is why Proxmox was not on my list.
But you say it supports Docker, then I will try it for sure.
The HW should be arrive during the next week. I will report how it works.
natively Proxmox does containers. there is a little fiddling to do to get prebuilt docker images onto Proxmox. it is not a difficult process though. The VM method has a guide on the Proxmox support wiki.
Thanks for the video link. I watch it and some others too to make decision about the OS and FS.
Today the new Server HW arrived, pretty loud this rack server
I installed Proxmox and because there is only one NVMe for the OS and one for data, I decided to go with the
good old ext4 for both. After all what I heard and read about ZFS, that is not the right FS for a single drive. There are 7 empty drive slots, maybe in the future it can be useful. And BTRFS is still not stable enough, at least that is my conclusion about the information I have so far.
I installed docker on the debian level, not in a LX container. That way it is simpler to move all the stuff from our current build server (1 core VM) to the new one.
Now I have to wait that the company calls me back who managed our IT to bring the server in the network (is it currently on a DHCP address).
From my point of view, using Proxmox was the right suggestion from you. That way the debian can be used as it is it, we have a simple way to install VM’s an LXC’s, and using ZFS in the future is also very simple.
The only thing that makes ECC a requirement is if you want to be able to guarantee no memory errors occur. So it’s still a great idea and honestly it really should be the default for every system, desktop or server, but I digress…
I also run proxmox on an epyc machine. Proxmox is an ideal distro for me because it’s basically debian and thus stable enough, but they make selected packages relating to virtualization are more up to date, and make sure the kernel version (even the testing ones) don’t skip ahead of what ZFS explicitly supports (which causes it to stop working).
Basically I have my data archive, various personal services and my “desktop” running on top of it. You can even make the desktop VM reasonably low in latency by keeping it (and the memory hugepages it uses) confined to a particular NUMA node. Different PCIe slots will be connected to different NUMA nodes, so your GPU and VM should be on the same one. I use a displayport fiber cable and an extron network compatible USB extender to connect to my machine in the garage.
I’m still reading, but so far, the discussion at the end shows me, that there is still some things not clear. The article is from 2015, maybe there are more information out yet. But there is some thing that was mentioned often, that is, if X happened, ZFS can read good data from the pool because there assumed some kind of redundancy.
The system I’m talking about has one SSD for data. Of cause ZFS was not designed for one drive systems, therefor it would have been the wrong choice in the first place, ECC RAM or not.
The idea with ECC RAM and ZFS comes from, I think, it was highly recommended for TrueNAS.
That is interesting to have desktops running on a server with access to the GPU. Display port over fiber cable? I have to look into it, could be interesting for some customers.
But this sever is only used for building C++ application. The only ‘GPU’ in there is one with an VGA output and I’m not sure there are free PCIe slots in there. This machine is about 4cm thick
Today we had our first test run with triggering a github action to build on the new server by using a build environment in a docker container. The complete build for that application (Qt5 with OpenGL for ARM 64) took about 11 sec. The old build server took about 3 minutes. Now I have to move my Yocto build system over, hope this will work.
All this server stuff is very new to me. I’m a C++ SW developer for over 25 years but had only desktop computers so far. I see, there is a lot to learn.
ZFS is possible but I wouldn’t bother - I’d go with lvm + either btrfs or XFS , whichever gets you more performance.
Docker let’s you choose storage drivers, if you choose btrfs or zfs don’t bother using anything filesystem specific for image layers, plain old overlay is the way to go.
Reason for Debian testing is because it’s minimalist and that makes it easy to reinstall/resetup and “testing” because everything is basically running the upstream latest versions so you’re not missing out on anything and you actually have a chance of getting support from developers.
Debian has entire repos that are timestamped, and plenty of tools to help you build a repeatable install.
Other one, I keep hearing good things about but haven’t tried is opensuse MicroOS. The stories I keep hearing is that it’s built to be used the way I’d like to use Debian testing - it’s on my “to try” list.
years ago I used debian (unstable) on a my desktop development PC. Worked good until debian got into the feature freeze. Then there was no updates for a couple of month, time to switch. Linux Mint was the one for many years then. Now I’m using Manjaro, on my office PC and private PC.
Using debian testing is, I think, not the right way for a server. Maybe for a server at home, but not for a server the whole SW development relies on. Of cause, i can be wrong with that. I’m not an admin, just a C++ developer who was just not fast enough to step back with the others
Currently Proxmox it still running and there is a nice web UI, not for docker. But for Docker there are other options. The server is mainly running jobs from github. But with proxmox it is easy to install VM’s and LXC’s. I already have a request for a database for testing.
Proxmox is based on debian stable but I think it’s using it’s own kernel which is based on an ubuntu kernel, 5.13 I think.
As I mention before, ext4 is the FS now. With BTFS, there are still to many bad things to read about. Looking into XFS, I think that was not as good as ext4 with many small files. And ext4 is properly one of the most stable and tested FS out there.
The last time I used SuSE was with version 7, bought it in a shop on CD-Roms
The thing with SuSE is, after I switched to debian based system I never looked back. No idea why.
Interesting, I wonder how this works (e.g. release engineering wise, I wouldn’t be expect a feature freeze in sid/testing as much as just maintainers may be having lack of brain cycles/attention).
ZFS doesn’t care if you run on a single drive/ssd or pairs, or various combinations of RAID, it assumes you know what you are doing and plan for redundancy/backups accordingly.
So, assuming you know how much redundance you need, if you don’t care about optimizing performance and/or plan for expandability, you can go with single drives, mirrored drives, RAIDZ1, RAIDZ2 in whatever VDEV (The zfs naming for a disk pool) configuration
What ZFS gives you over xfs/ext4/lvm/md is capabilities in terms of filesystem management, you can break down your drive into multiple datasets and assign different settings/quotas to it, you can take snapshots and revert to them instantly, and you can have incremental backups sent to a second server in a matter of minutes
If you need any of that with the other filesysems, you will need to add software, combine different tools, and then some …
If you don’t need redundancy, snapshots, backups, compression, flexibility in managing disk quotas then yes, ext4 or xfs will just get you going …
As for my vote on which distribution to go with for a Server … now that Centos is gone, I’d say Rocky Linux or Oracle Linux … they track closely red hat releases, are more conservative than Debian/Ubuntu in applying breaking changes and Oracle Linux has the option of buying support, if you feel like it. I know the Oracle name stinks a bit but … the product is solid, if slow in getting cutting edge updates …
@MadMatt
After all, from what I’ve heard about ZFS it is an amazing file System. The only other FS that is going in that direction would be BTRFS but I guess that will still take a while.
That is my use case. There are no data stored on this server that is not on some other system, mainly github. Even the Dockerfiles are on github. Might take a while to rebuild all from scratch though. And I’ve been explicit told, that this server will not be in the in house backup system.
There are some videos on youtube there explain howto setup ZFS on a single drive. But after I watched Wendel’s Video on ZFS I came to the conclusion that the original designers never thought of single drive usage. I did not meant it wouldn’t work.
That is at least 15 years ago. I don’t know if Debian has changed something in there process since then. And there is always the possibility I did something wrong. As far as I remember, the freezing was about package versions, not bug fixes.
The feature freeze was the way to get the next release version-stable.
I’m developing SW for a kind of embedded systems (that is a i86_64 platform). Back then debian stable was the chosen way. In the last years that turned out was a kind of mistake. We would like to update some lib, mesa for example, but for that we first would have to update libc. The risk is to high, customers around the world are using our devices.
Now we are on the way to switch to yocto, that way we have more control about our systems and because we are starting to use ARM CPUs now (i.MX8). Yocto is one of the main reasons for this build server.
