Proxmox - How to do it right?

Software & Operating Systems Linux
1

Background:
So I wanted a new GPU but couldn’t get hold of one… But I could get a whole computer for a similar price to what the GPU alone was going for and I caved.

I am now the proud owner of a 3080Ti and have moved it from my new computer to my actual gaming computer. The rest of the new computer will be used to run Proxmox and my VMs.

Topic:
I have a 10850K machine to replace all my old servers. I will run Proxmox on it with pfSense, PiHole, a webserver, a Windows and/or machine for hosting some game and media servers, a couple of sandbox VMs, and hopefully one with a passthrough of a 980Ti for some secondary computer gaming if it’s not too awkward.

I currently have one bare metal Windows machine for hosting all my games and media servers. And another running Proxmox to do all the other stuff. Everything is set up in a very basic way and not done “properly”. I love tinkering with things and making computers do stuff, but I’ve never really done things the right way and want to look into it.

By that, I mean:

  • I need to look at how to run my RAID array - should I switch to ZFS or something rather than RAID 5 on a RAID card? Performance is not fantastic on said RAID card.
  • Backups… Yeah, I’m one of those idiots. I know the rules and blatantly ignore them.
  • VLANs and proper firewall routing, etc?
  • PCIe Passthrough - might screw around with splitting the GPU between VMs (as CraftComputing recently posted a video on this)
  • Anything else I’m not thinking of because I don’t know about?

I’m doing some testing and comparisons between the AMD 5950X in my main rig and the 10850K in the new machine this weekend and then I need to work on replacing the servers. I look forward to everyone’s ideas!

4 Likes
2

Hey, I’m not sorry you got a whole system for the price of a card, as you have a new toy!

I am not sure what questions you have for people though, or what feedback you sought?

You mention your raid card is a bit slow. ZFS can’t fix that, but striped mirrors (raid10) might be the best speed, as long as Raid card is set up in IT mode?

Backups- even a pair of external USB drives, switched Sunday before lunch to backup would be better (okay arbitrary time, but you make it a routine, and that’s the thing you will appreciate)

1 Like
3

If it’s going to be your main router, then don’t. Unless you like getting beaten having your whole internet access go down when something goes wrong with your host or you have to reboot it. A separate, low-power consumption physical box for the router is just better, unless you have some kind of redundancy in-place (and you mentioned replacing everything with 1 system). Even an Intel NUC with an el-cheapo USB NIC (for WAN) would suffice if you don’t have Gigabit (or more) to your house (and I’ve ran such a setup for a small office with 20 people with an 100 mbps WAN connection).

Highly biased, so my answer will always be yes. I love software RAID because you can run it anywhere with no headaches like compatible RAID cards in case something goes awfully wrong (I hear the situation is not as bad as it used to be, but still, better make the switch).

Just do it!

VLANs are optional if your switch supports it (or if you have direct connection between your router and servers / *nix boxes). Visit my recent wiki (still in beta, need to implement IPv6 rules) to make a secure network for home. You don’t need lots of rules, just the default-deny on WAN from pfSense / OPNsense and blocking some additional stuff from going out of your network and just having separate networks for untrusted devices.

Add / block whatever else you feel you need based on your threat model (e.g. make a proxy and block outbound 80 and 443 connections, maybe also block port 21 and maybe email ports if you aren’t using email clients and only use web mail). I’ll probably improve on this tutorial with an optional more restrictive policy, like blocking everything and only allowing http and https outbound through the proxy. But for now, home users don’t need lots of block rules, which is why I named this “easy to follow.”

Do you really need that? I’d personally just do 1 VM for Windows and passthrough the whole card. Unless, of course, you got a higher-risk threat model and want a separate gaming VM and a video-editing VM that requires some GPU acceleration. Or, as mentioned in CraftComputing’s video, have more people play on 1 card.

3 Likes
4

wow, I guess there Was a lot more stuff to comment on!

Well Done!

1 Like
5

Blimey, that’s a lot of information to take in. Haha! Thank you!

It is my main router and I’ve been running it that way for a few years - I have no issues with it and can easily chuck the ISP provided router on the network to run things for a bit while I get stuff sorted out. So this isn’t an issue. I run nothing “critical”, I just toy with things and have a few useful services running to make life easier.

My opinion on the RAID cards comes from the fact that the drives are almost always pinned at 100% and the response time in Windows has a habit of rising to the thousands of milliseconds regularly. I know that I can get 400-500MB/s sequential reads out of three drives (4-drive RAID 5) yet I never get that, even with ideal workloads. I’ve been looking at alternatives for ages and nothing quite does what I want within the Windows ecosystem, but if I host that server on a Linux-based machine, then the floor is open. ZFS (software RAID) does have extra benefits, as you pointed out.

That’s easier said than done. I have loads of data that I should backup but most of my drives are over 50% full. I can’t justify spending thousands on hard drives to allow me to back things up so I never have. What I was thinking was to do incremental backups of my main PC’s OS drive (and personal data) and for each of my servers - but otherwise to ignore media, etc.

VLANs are supported on my switch and I will have 4 or 5 network ports on the VM Host - plus most of the stuff will be virtual anyway. When I get around to building everything, I shall have a look at that tutorial. Thanks!

No, I don’t really need split GPU stuff but it’s something I thought should be a thing for years and now it is, I want to try it. Haha. Just passing through the GPU to one VM was the intended setup.

1 Like
6

Interesting. Have you checked if there are issues with the RAID 5 array, like a failed disk and constantly having to run parity calculations constantly? Also, I encountered 100% disk usage on Windows, on both HDDs and SSDs, so that may also be an issue (flipping Window, man…)? Is this behavior consistent when the Windows VM(s) are turned off?

If you have a decent internet connection and no data cap or a big data cap, try finding a temporary solution, like using BackBlaze ($6 / month unlimited data) or abuse the $100 coupon codes buy a Linode server with few CPU cores and RAM, but big storage, to make a backup server for a day or 2 while you format the drives to ZFS, then download everything back.

1 Like
7

If only I had a decent internet connection… It would take me weeks to upload my data. Haha! Slightly fewer weeks to download it again, but still weeks. In all seriousness, I get okay internet, but 67 down and 19 up is not great in this day and age.

As for the RAID array: it’s not “almost always” as I earlier stated - that’s a slight exaggeration. But if I ask the drives to do anything, they seem to get hammered far more than what I’ve asked of them should. There are no errors on the drives and health checks have said they’re okay. They’ve also been doing this since first set up and my previous array with different drives and a different RAID card did it too. I think it’s how Windows handles storage and how RAID arrays work - just doesn’t seem right to me though.

8

They could be SMR drives.

9

The old drives were 4TB Seagate NAS drives and the newer ones are 8TB Seagate Ironwolf drives. I’m pretty sure I checked when the SMR stuff was in the news and both were not?

10

Okay, so looking at actually setting things up - how do I go about ZFSing? And can I set up users and permissions within the GUI on Proxmox for a RAID Z1? And is that the equivalent of a RAID 5? What should I know?

EDIT: And what about caching? I currently use Primocache on Windows - can I just set an SSD as a cache within ZFS and then not need any other software?

11

3, 5 or 7 hosts in a cluster with shared storage.

12

Okay, I’m lost… I’ve been following this thread:

And have complete step 7 fine. Step 8 seems irrelevant to me as I’m not following on past step 9. I just want a share I can access from Windows.

I’m stuck at the samba config bit…

server role = standalone server
create mask = 0777
directory mask= 0777
[share]
comment = root share
browseable = yes
path = /storage/share
guest ok = no
read only = no

My file doesn’t contain stuff that looks quite like that. I have a few of those lines - am I supposed to add the [share] section myself or should that already exist?

13

So I’ve done some more research and think I’ve figured out the syntax for the smb.conf file - and I think I’ve got it set up right. I have restarted the smbd service and tried to connect to the share with Windows.

I don’t seem to be able to connect either by searching for it (browseable is enabled) or by typing the direct //IP/share location. Has Proxmox got some firewall rules blocking it or something?

EDIT: Woohoo! I had some assistance from someone in another Discord - but I have got it working. I’m also amazed at the speeds I’m getting according to Bonnie++ with compression enabled. Even without compression, the numbers are on the high end of what I’d expect.
Now to set things up properly and reconfigure it again… Wish me luck!

14

I thoroughly, thoroughly, thoroughly recommend you install Proxmox Backup Server to a remote storage VPS. These don’t have to be expensive, mines £8/month for 2 vCpu and I’m barely using it at half capacity.

It’s just fantastic once you’ve got it setup. It’s rescued my setup multiple times as I’ve explored hardware and configuration. What’s more, is it’s allowed me confidence to try things out.

I’ve hosed my setup again? Ah well, scrub the VM and pull a backup. I’ve hosed the whole server? Ah well, get the prox bootable usb out, and pull all the VM backups. It’s pretty much that simple!

Whilst this doesn’t solve the data on your NAS being backed up, it will make your self hosting life way less painful when it inevitably goes wrong IME.

Zfs - yes!

1 Like
15

While this does sound great, I have a thing against spending money… Haha! However little it is.

I do have old hardware that I can chuck it on, though. Probably not a bad idea, definitely.

I’ve actually got a fantastic system setup now and I’m really pleased with it. So a backup is great idea before it blows up! Well… I blow it up. Haha

16

hi, care to share the fix for posterity?

I thought the share was a whole unit:

[Steam Folder]
comment = Steam Folder
browseable = yes
path = /FlashArray/SteamStorage
guest ok = no
read only = no
create mask = 0777
directory mask= 0777

with the mask stuff being a definition of the share.

then differing shares can have different perms…

17

Definitely! It’s a totally viable option. I choose to host offsite so if the house burns down I don’t lose it all, kinda thing. Or if all my computers are stolen, I still have my ‘setup’, but I do run a local PBS instance as well actually for VMs I’m not that fussed about, or have a large amount of data I’m not willing to pay someone else for the space to store. I literally use my old PC for that.

I guess my main point of that post wasn’t actually the remote VPS part, just about how awesome/fantastic PBS is as a backup solution. It’s pretty much set and forget, and now with PVE 7 + PBS 2.0 you can even do file level restore! So if you just made one bad edit, you can restore the file to the VM and the whole thing is rescued.

18

I’m not sure exactly what you’re asking. I think what I had done is I had the global settings set to 0777 which is a no from a security standpoint.

Within the share, I have reduced it to 0750 and have two users, one that owns the share, one that does not, but still has read and execute access.

The last issue I was having was trying to connect to the share which was because I was using the directory, rather than the share name. The share name in your case would be “Steam Folder” as that’s what’s in the [ ].

Hope this helps - let me know if not.

1 Like
19

Welcome to the forum!

I would suggest people first try to host backup servers at their parents house, especially if they live more than a few towns away. That way, if they are on a pension, you can help them pay the electricity and internet bills (which is what I do). You get a remote backup location and you have “tech support” that can help you switch HDDs in case something happens (make sure you label them correctly on the outside). Sure, you have to deal with your own infrastructure, but I believe it to be worth it in the long run.

Otherwise, if their internet sucks and have no other option, or they have massive power outages often, a VPS might be better. And obviously, if you don’t have parents or other family members, a VPS may be the only option.

I see you use Good Boy Points (GBP), so I guess you shouldn’t have the Internet issues we have in the US.

Also, if you use a VPS or any untrusted 3rd party for backups, you need encryption. Check out restic:
https://restic.net/

There’s this option too, but you can’t blame anyone but yourself if you lose important data forever. At least encrypt your most important stuff and put them on a free cloud.

Here’s my samba template:

[folder-name]
        comment = "description"
        path = /path/to/folder
        public = no
        writable = yes
        browsable = yes
        guest ok = no
        readonly = no
        hide dot files = yes
        force create mode = 0775
        create mask = 0775
        directory mode = 0775
        directory mask = 0775
        force directory mode = 0775
        force group = "group-name-without-quotes"
        valid users = @"group-name-without-quote"

I create Samba users and local users. The local ones, I add inside certain groups (like, “family group” or “admin” or “servers” etc.). Each group has its own share. You can put individual usernames under “valid users,” but when you administer a few dozen users and shares, things start to get messy very fast. So all you have to do is add them to groups and they will automatically have access to all the shares available to the group (after a smbd restart).

Remember when you add new users, to use the shell “/bin/false” in order to prevent them from logging into the server via SSH (and don’t set them a local user password, just a samba password). I can answer any additional Samba questions, to the best of my knowledge.

1 Like
20

I don’t disagree at all, it’s a really solid choice that I do forget mention as it’s unfortunately not viable for me, as my sibling doesn’t even own a TV, and my mother has no need for a higher speed line. So for me (sadly) VPS is the most cost effective solution for now.

When my friend gets his own place, we’re going do a drive swap/ship a drive.

My line is pretty rock solid, just wish I could get more upload. Capped at 900/100Mbit D/u for now :frowning:

1 Like