I don’t know what I was doing wrong, but seems like the forwarding works in bind9. I tried the same settings multiple times, but it only worked after setting dnssec-validation to no, then back to auto. I was getting broken chain of trust messages.
Kinda crazy how long a DNS query can take sometimes. Typically it’s just 200ms, but sometimes I get this:
time ping level1techs.com -c 1
PING level1techs.com (172.67.73.46) 56(84) bytes of data.
64 bytes from 172.67.73.46 (172.67.73.46): icmp_seq=1 ttl=56 time=201 ms
--- level1techs.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 201.485/201.485/201.485/0.000 ms
real 0m2.238s
user 0m0.002s
sys 0m0.004s
ping took 200ms, meaning 2 seconds were allocated to DNS query and response back. Repeating the test yields only 0.007ms, which is what you should be expecting.
time ping level1techs.com -c 1
PING level1techs.com (172.67.73.46) 56(84) bytes of data.
64 bytes from 172.67.73.46 (172.67.73.46): icmp_seq=1 ttl=56 time=255 ms
--- level1techs.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 254.551/254.551/254.551/0.000 ms
real 0m0.262s
user 0m0.002s
sys 0m0.004s
Just look at this! Random all over the place!
time getent ahostsv4 level1techs.com
real 0m0.011s
user 0m0.011s
sys 0m0.000s
time getent ahostsv4 craigslist.org
real 0m0.946s
user 0m0.007s
sys 0m0.005s
time getent ahostsv4 youtube.com
real 0m0.554s
user 0m0.005s
sys 0m0.006s
I don’t blame bind, just my connection, but this can contribute to a slow internet experience. I remember when our main DNS failed and everything felt slow and we couldn’t figure out why. All hosts were querying ns1, timing out because of no response, then querying ns2. Apparently one of the scripts we were using to reload the zones broke and left the config unworkable (I don’t remember what line we had to remove, but it was something minor - if only we would have used named-checkconf before applying the changes…).
Anyway, I got a working config inside a DNS container, now I need to apply it on my router. I’ll probably end up making one or two DNS containers anyway, so I can have some redundancy (and using keepalived to prevent the slowness I mentioned earlier in case the main dns fails automatically).
I was trying to learn more about iSCSI and how I could implement it in my infrastructure. My general idea was that I could use iSCSI as just a block device and pass it to a VM, but seems like Proxmox is incapable of that. Or maybe I’m just a brainlet (nothing new, lol) and I cannot figure out how.
I was hoping maybe there was a way to tell a VM “use this device for your OS and you manage it” and whenever I wanted to live migrate a VM, tell the hypervisor “pause the VM, transfer data from RAM to another host, detach iSCSI” and on the other host “attach iSCSI, resume VM activity with the RAM contents you received.”
This is in fact how I was doing it, but with NFS. I had a NFS share which I configured in the datacenter level in Proxmox, all VMs had qcow2 disks and when I live migrated, I didn’t have to copy the disk contents to another location (like you’d have if you ran with LVM/-Thin or local ZFS), just copied the RAM contents and the disk was already there, mounted in the same directory waiting for the VM to be resumed.
I was really hoping I could have something similar with iSCSI, because it would make management a bit easier. With iSCSI targets on a zvol, I can just snapshot the ZFS volume for each VM and send the data over to another pool or another NAS. But with NFS, I would have to either snapshot the entire share containing all the qcow2 (or raw disks maybe, since I wouldn’t be using qcow2 for anything because ZFS takes care of snapping), or create a NFS share for each VM which would be nuts.
I wouldn’t really blame proxmox for this, in all fairness, I’m struggling doing this with LXD, which is the reason I tried it on Proxmox. The same idea was to have an iSCSI target mounted on the host and have each container use their own targets. It is technically doable, but in LXD, the concept of pools is what boggles my mind. I have to create a pool for each container, which is dumb. And then, for each pool, I have to do a dir, which makes things overly complicated.
NFS, while having its downsides, is still the easiest and most straight-forward method of sharing the VM / container with other hosts and easily live migrate them. From the perspective of storage resources, it really makes way more sense, as opposed to having 2 different VMs / containers and use things like keepalived or corosync + pacemaker. The only thing coming close to it is diskless booting VMs with readonly access to the rootfs and their own /var and /etc locations basically. You still waste some space though, unlike just moving a VM over, but it depends on the scenario and requirements, sometimes having another VM always up makes more sense than doing HA (especially if you already have another way to load-balance them and not just have an inactive standby VM).
Back to iSCSI, I found this thread on proxmox forums.
Multiple ways work:
- adding a disk in PVE (host), create LVM on top and use for multiple VMs
- using iSCSI to have the backing device for on VM
- adding iSCSI inside of the VM, completely bypassing PVE
The broken English is a bit hard to grasp, I can only guess that the second option was meant to be exactly what I am trying to do. But I found no way of doing that. When I add iSCSI in datacenter storage, if I check “use LUN directly,” then Proxmox will be treating the target as its own storage, basically be able to format it with LVM and use it for VMs.
If I don’t check that, I’m still unable to add it to VMs, neither when creating the VMs, nor when they are already created.
I wonder if Ceph has a better handler and if it can do what I’m thinking of. Well, if I were to run ceph anyway, I’d probably be setting up the hosts with local storage on them and try a shot at a hyperconverged infrastructure. I think Ceph RDB might be that?
For now, it seems like if I want to be able to take snapshots for each VM or container separately, I need to use NFS. For LXD, I need to have a pool for each container, which makes things a bit annoying. Well, now that I think about it, technically if I used iSCSI, I would’ve had to create the target, then add the initiatornames in the host conf. Not that different than having to create a new zfs mount-point, set nfs share value on it, then add the entry on the lxd host in fstab or something, then create a new pool using the dir driver.
For iSCSI, I’d also have to format the disk and mount it - unless… I think I got an idea… kinda hacky, but doable. In either case, if I create an iscsi target and format the disk on the host and mount it, or if I just mount a nfs share, I can do it without having to create multiple pools in lxd. I just need to create the container, stop it, move stuff to another directory, mount either the iscsi or nfs into the target, then move the data from the container to the freshly mounted point. Ugh, I don’t like this, but it’s what I might end up doing anyway.
I probably want to experiment this in VMs before I go live with it. Thankfully, making a VM template and cloning via zfs should be easy. Now’s one time where I wish I had a low-power hypervisor, like a NUC, another odroid h3/+ or a zimaboard. I wonder how qemu/kvm works on aarch64 nowadays, since lxd --vm option still doesn’t properly work on aarch64. Or maybe I can get away with lxd vms on my odroid h3+ itself, I have the RAM and probably the CPU for it anyway, just need to install qemu… I was always curious how lxd vms compare to firecracker in terms of resource consumption. Maybe I’ll take some time to install opennebula in a VM and run firecracker on it, just to see how it goes.
According to the debian wiki, VMs in LXD require the package qemu-system-${ARCH} (x86_64 / aarch64). https://wiki.debian.org/LXD
I do have it installed, because I have the package “qemu,” but it still doesn’t work on my system. Not to worry though, because I’m soon going to test firecracker.
Yep, basically zfs send / receive a snapshot of a zvol to another zvol and make the VM template in virt-manager. I’d say it’s simpler than even proxmox (because you don’t have to go through as many steps, it’s mostly pre-selected well in virt-man). But still need a quick one-liner for whatever you’re cloning in any hypervisor.
I have 3 alma vms, planning to make a cluster of some sorts and test firecracker (probably in opennebula), then wipe it out and test lxd vms. If neither are better than qemu, or if firecracker requires too much work to get working (I think it does need a kernel module or something), then I’ll just drop them (well, the ideal is to only use containers, because of hardware limitations, especially RAM). I still think the resources should be lower for both lxd and firecracker compared to qemu (at least the later), but not all the container OS are available also as vm options in lxd.
Not that it really mattered, since they’re containers (the memory is system shared, even if you put a limit on it, it doesn’t allocate all of it to the container), but I decreased the maximum allocated ram on squid to 512mb and to yt-dlp container to 256mb. They barely use 14 and 31MB respectively after they’ve been rebooted to apply the changes (although the changes might be live, not quite sure, but I had to reboot squid anyway, because it had nesting enabled by default, which it didn’t need).
The OS is basically negligible, but I might still have a hard time running all the programs I’m planning to due to RAM limitation. Proxmox says it’s using 5GB of RAM (out of 8), despite only having 2 containers running and nothing else (free tells that 977MB are cached, meaning proxmox is actually using about 3.5GB, which is nuts - uptime is 8 days).
Thankfully I won’t be using proxmox (not even sure why I’m getting such high consumption, it shouldn’t be using more than 2GB of RAM, really), but I’ll be limited to 4GB of RAM on my n2+. But then again, probably most containers should be fine with anywhere between 256 to 512MB, with maybe a DB container requiring 1GB. I should still be able to run anywhere between 5 to 8 containers on the n2+. I didn’t think I may need a homelab upgrade so soon, but then again, I was planning to buy more sbcs to play with after the initial lab was set up (which I obviously never got around to).
Well, the initial plan was to have some kind of load balance and HA (when half the cluster goes down, to have the other side take care of the load and max out the system until the other side comes back up and containers or services can be moved back there).
I might need to look into newer SBCs, but I really don’t look forward to more SBC adventures. I’d love to get a quartzpro64, but that thing won’t be available anytime soon (not to mention lack of support from any distro at this point - I’m struggling enough with the hc4 as it is). I don’t even want to touch stuff like Rock Pi 5 or Orange Pi 5 without first knowing what they use to boot (I think the rocks have a weird bios-like thing anyway). But Rock 5 and Orange Pi 5 seem to be the only one on the market with 16GB of RAM. But quartz64 with 8GB of RAM is pretty cheap… I don’t want to hear about any other bootloaders than u-boot and petitboot. I found a single reference online that the orange pi 5 might be using u-boot (which would be nice).
Did you give this a try? IIUC there isn’t really snapshot-ing, or disk data transferring (only ram is sent over the network), qemu has an rbd backend that Proxmox just happens to configure.
My homelab consists of 1 proxmox host, 1 libvirt host (with a proxmox VM if I ever want to test stuff locally), 1 LXD host, 1 NAS (with iSCSI and NFS) with 2 zfs pools and since yesterday, my own PC also on libvirt.
I don’t have the resources to deploy a ceph cluster, I might try it under virtualization, just to see a theoretical setup and how it would work.
In proxmox, with NFS backend, using qcow2 vdisks, there is no snapshotting happening either. Just the RAM contents get moved over to another host when live migrating and the disks stay in place, as all the hosts can access the NFS backend concomitantly, but they won’t try to touch the vdisk unless they are the ones running the VM.
I believe with a ceph block-device backend, the same would apply here. You’d export the vdisk in ceph and when a live migration happens, you don’t have to snapshot, send the contents over, then change the disk location on the hypervisor, it will all just be seamlessly pointing to the same location.
The problem I’m dealing with, is that it seems iSCSI doesn’t seem to have the same feature? Which really makes me wonder why I can’t figure it out. The procedure should be just as straight forward. Unlike NFS, iSCSI can only be mounted on one host at a time, so it’s understandable you’d first need to unmount the target, then on the other host connect to the target, then resume the VM operations.