My pearls!
I got frustrated with it. Expect to run updates way more often than pfsense.
What are you using it for specifically?
I recently decided to go headless OpenBSD (not that there’s another choice) for my gateways. Still working out the config, but I’ll be glad to be done with Ubiquiti and pf/opnsense once I’ve got it working.
Sidenote: OpenBSD has uncontroversially supported wireguard for a while now
Adblocking, including killing stuff that attempts to “phone home”. Could use a Pi-hole VM, but having everything integrated was kind of nice, especially since the hardware doesn’t even break a sweat given its workload so having it on there is kind of efficient.
I used to do my firewalling through Shorewall on Linux, switched to pfSense since I wanted to have it on its own machine. I kind of like having the UI, which I guess is sort ofironic since I do most other things through the CLI (eg. my answer to what file manger I use would be “zfs”).
Maybe I should give OpenBSD another look, last time didn’t go so well, but that was a very long time ago. As in, back in ye olde, dark, 32bit days. 
I read through both Absolute OpenBSD and The Book of PF and would recommend doing so to anyone wanting to use it as a firewall.
For the ad blocking, I’m sure you could grab whatever lists pi-hole uses and use them in vanilla pf… if they’re URL’s, you might need to resolve them to IP addresses though.
For my new VM server I acquired a couple of 100GB retired EMC SSDs. These are formatted with 520byte sectors and are not usable in most normal hardware without reformatting them.
Whether these will actually show up depends on the SAS controller, in my Dell/Lenovo x3650 M4 they were found by the onboard RAID controller, but did not show up on the OS (Proxmox) in any way.
On another machine, also running Proxmox, with an LSI2008 based HBA this showed up in the logs:
[919353.253564] scsi 7:0:4:0: Direct-Access SAMSUNG SS162511 CLAR100 DC0D PQ: 0 ANSI: 6
[919353.253573] scsi 7:0:4:0: SSP: handle(0x000d), sas_addr(0x5002538453b04ee2), phy(16), device_name(0x5002538453b04ee1)
[919353.253575] scsi 7:0:4:0: enclosure logical id (0x500304800024fdbf), slot(4)
[919353.253577] scsi 7:0:4:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
[919353.253896] scsi 7:0:4:0: Power-on or device reset occurred
[919353.261531] sd 7:0:4:0: Attached scsi generic sg22 type 0
[919353.262333] sd 7:0:4:0: [sdu] Spinning up disk...
[919353.262957] end_device-7:0:4: add: handle(0x000d), sas_addr(0x5002538453b04ee2)
[919354.285403] .ready
[919354.285755] sd 7:0:4:0: [sdu] Unsupported sector size 520.
[919354.289912] sd 7:0:4:0: [sdu] 0 512-byte logical blocks: (0 B/0 B)
[919354.289913] sd 7:0:4:0: [sdu] 8320-byte physical blocks
[919354.295266] sd 7:0:4:0: [sdu] Write Protect is off
[919354.295268] sd 7:0:4:0: [sdu] Mode Sense: b7 00 10 08
[919354.296247] sd 7:0:4:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA
[919354.322140] sd 7:0:4:0: [sdu] Unsupported sector size 520.
[919354.331985] sd 7:0:4:0: [sdu] Attached SCSI disk
To be able to use these drives we need to reformat them to a usable block size (aka, a multiple of 512 bytes).
Note that this resets some of the SMART information.
apt-install sg3-utils
# sg_scan -i
...
/dev/sg22: scsi7 channel=0 id=5 lun=0
SAMSUNG SS162511 CLAR100 DC0D [rmb=0 cmdq=1 pqual=0 pdev=0x0]
# sg_readcap /dev/sg22
Read Capacity results:
Last LBA=192365851 (0xb77451b), Number of logical blocks=192365852
Logical block length=520 bytes
Hence:
Device size: 100030243040 bytes, 95396.3 MiB, 100.03 GB
sg_format --format --size=512 /dev/sg22
SAMSUNG SS162511 CLAR100 DC0D peripheral_type: disk [0x0]
<< supports protection information>>
Unit serial number: 10NEADB00535
LU name: 5002538453b03a10
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=192365852 [0xb77451c]
Block size=520 [0x208]
A FORMAT UNIT will commence in 15 seconds
ALL data on /dev/sg22 will be DESTROYED
Press control-C to abort
A FORMAT UNIT will commence in 10 seconds
ALL data on /dev/sg22 will be DESTROYED
Press control-C to abort
A FORMAT UNIT will commence in 5 seconds
ALL data on /dev/sg22 will be DESTROYED
Press control-C to abort
Format unit has started
Format in progress, 58.43% done
FORMAT UNIT Complete
# sg_readcap /dev/sg22
Read Capacity results:
Last LBA=195371567 (0xba5222f), Number of logical blocks=195371568
Logical block length=512 bytes
Hence:
Device size: 100030242816 bytes, 95396.3 MiB, 100.03 GB
I then installed Proxmox on these, just clicking through the installer, which resulted in this:
root@vanagloria:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices are configured to use a non-native block size.
Expect reduced performance.
action: Replace affected devices with devices that support the
configured block size, or migrate data to a properly configured
pool.
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35002538453b04ee0-part3 ONLINE 0 0 0 block size: 4096B configured, 8192B native
scsi-35002538453b03a10-part3 ONLINE 0 0 0 block size: 4096B configured, 8192B native
errors: No known data errors
smartctl confirms this SSD uses 8k sectors:
root@vanagloria:~# smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.4.103-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SAMSUNG
Product: SS162511 CLAR100
Revision: DC0D
Compliance: SPC-4
User Capacity: 100,030,242,816 bytes [100 GB]
Logical block size: 512 bytes
Physical block size: 8192 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Logical Unit id: 0x5002538453b04ee0
Serial number: 10NEADB00868
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Thu Mar 18 16:56:23 2021 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
ZFS is supposed to automatically detect the correct ashift to use, so either that mechanism failed in this instance, or Proxmox blindly sets asfhif=12, whereas in this case it should have been ashift=13. In either case, reinstalling and manually setting ashift to 13 fixes this issue.
Found this for replicating at least part (not familiar with pi-hole, so dunno what else it might do) of the Pi-hole functionality in OpenBSD:
I really am going to go down this rabbit hole, aren’t it?
Oh awesome! Now I don’t have to figure that out.
PSA: Proxmox switched from Grub 2 to systemd-boot for systems with root-on-zfs.
Took me a while before I figured out why I couldn’t get IOMMU groups to show even though the system was configured the same as my other Proxmox machine…
I’ve been using DNSBL in pfBlockerNG but I have to admit I’ve been horrible at testing or measure any metrics- it was kind of a ‘fire and forget’ setup that I did.
The logs coming out of pfSense syslog have no discerning field if its from unbound or from some other firewall rule so I can’t think of any easy way to make a dashboard tracking just my unbound activity.
That said, the native dashboard pfSense has is pretty decent- I’d love to do something in Splunk though.
I had hoped access to node_exporter in 2.5.0 would allow me to get some useful data out of pfSense and into Prometheus (and into my Grafana dashboard from there).
Unfortunately the node_exporter plugin has been broken. Supposedly it’s fixed now, but the fix doesn’t appear to be retroactive as removing and reinstalling the plugin hasn’t really helped. Then the Wireguard drama I happened and I kinda stopped poking at it as I’m considering moving off of pfSense.
I have been playing with the idea of using Loki to parse logs as another input for Grafana, but, well, it’s kinda not moved past the “playing with the idea” stage so far
In general I rather like Grafana’s monitoring stack, it’s pretty darn lightweight for the functionality it packs.
Currently I do the more general network monitoring through ElastiFlow. It’s not quite ntopng, but it’s good enough. It does use ElasticSearch though, so it likes to omnomnom memory…
If I could move that over to Prometheus/Loki that would be great, of course. Maybe one day…
Splunk forwarder for freeBSD worked on pfsense last I tried. I need to get into other log collectors and parsers but just haven’t had the steam.
ARs’ journalism knocked it out of the park-
20:00.0 Infiniband controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
20:00.1 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.2 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.3 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.4 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.5 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.6 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:00.7 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
20:01.0 Infiniband controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
Finally got SR-IOV to work. The way there…was not pretty…
Had to install the Mellanox OFED drivers, unfortunately they kinda don’t support Proxmox, but hey, Debian is close enough, right, right? Well…sort of…
Since the kernel is different I built new modules:
./mlnxofedinstall --without-dkms --add-kernel-support --kernel 4.4.106-1-pve -s /usr/src/linux-headers-5.4.106-1-pve --without-fw-update
This will put a custom installer in tmp, so grab that, untar it and run ./mlnxofedinstall --without-fw-update. It will complain it can’t remove a bunch of stuff, among which…most of Proxmox. Yaaaaaay…
So go right ahead, force the uninstall through (what could possibly go wrong? Right?), by touching the file apt tells you (not) to. Now try the above command again. It will likely fail because apt tries to run some Proxmox specific script.
The solution to that can be found here, but in case that link goes dead:
# mkdir /usr/share/proxmox-ve
# touch /usr/share/proxmox-ve/pve-apt-hook
# chmod u+x /usr/share/proxmox-ve/pve-apt-hook
Now remove the rest of the stuff mlnxofedinstall wants to be rid off and then let it do its thing.
After that, reinstall Proxmox with apt install proxmox-ve, and reboot.
Voila! Easy! And totally risk-free!
Having a bit of a time to get NFS over RDMA working with those virtual functions.
Same problem as this (unsolved) Stackoverflow question. Looks like some sort of version mismatch. RDMA does work fine on the host, though I’m not confident it’s actually using the Mellanox OFED module as it doesn’t appear to be built and installed by default.
Just so many different permutations of possible configurations to try…
Well, went through some more iterations, ended up installing Ubuntu Server in the VM, and trying with that without any more luck. So I put things aside for a week or so.
So when I get back to it I reinstall Debian, set up all the basic stuff (rdma packages, load infiniband modules on boot, set up networking, and configure fstab). I kinda expect it to just not connect with RDMA but give it a go anyway before I give up on RDMA. But…it just works…?
Not that I mind, didn’t have to mess with building/installing Mellanox’ OFED in the VM, which is a win. Just hope it’s not a fluke and will break with a gust of wind from the wrong direction.
Fingers crossed…
Bit the bullet, after years of procrastination, and migrated my email setup.
Pulled my old fetchmail → procmail → courier-imap setup from the physical host into a VM.
While doing that I moved the actual mail directory to an NFS share on my NAS, both for automated backups (VMs aren’t backed up as often), as well to avoid having to worry about VM size. I also migrated from Courier IMAP to Dovecot (which was shockingly painless with the provided Perl script)
For now I’ve just dropped spam filtering as the Spamassassin setup I had was pretty crufty and slow. Eventually it will get replaced with rspamd. Provider still filters out most of the cruft so this mostly picks out the stragglers anyway.
Thunderbird seems to really not like the change to Dovecot as it refuses to connect without even an error (but there’s certificate errors in Dovecot’s logs). SeaMonkey, and Claws, work fine with the self-signed certificate though, and since none of this is exposed outside the intranet a self-signed cert really is good enough. Just gotta poke Thunderbird until it works (probably nuking the profile will fix it, just really don’t wanna 
)
While looking for another client to verify whether Thunderbird was the problem I couldn’t help but notice the sad state of the email client landscape, after going through the Wikipedia list it appears there’s really like 3 actively developed proper GUI FLOSS mail clients for Windows (Thunderbird, Claws, and SeaMonkey), and the situation for *nix isn’t much better. Ouch.
As for outgoing mail, hosting provider deals with that. It’s apparently such a big hassle dealing with all the blacklist extortionist practices that it feels like it’s really not even worth trying to selfhost outgoing mail.
Both my OS SSDs bombed out over the weekend, one on Friday, the other Sunday night with basically the same SMART error:
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x73 [asc=5d, ascq=73]
Which, according to documentation I found means: MEDIA IMPENDING FAILURE ENDURANCE LIMIT MET
However, one drive has 8% used indicator, the other 0% (iow, the “worst” of the two supposedly has 92% write endurance left).
Strongly suspecting these are some of those famous time bombed SSDs, especially given the usage stats on the 0% drive, but couldn’t find any information confirming or denying that.
Full SMART data for one of them, in case someone can glean something from it:
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.4.106-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SAMSUNG
Product: SS162511 CLAR100
Revision: DC0D
Compliance: SPC-4
User Capacity: 100,030,242,816 bytes [100 GB]
Logical block size: 512 bytes
Physical block size: 8192 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Logical Unit id: 0x5002538453b04ee0
Serial number: 10NEADB00868
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Mon Apr 19 11:53:01 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x73 [asc=5d, ascq=73]
Percentage used endurance indicator: 0%
Current Drive Temperature: 28 C
Drive Trip Temperature: 58 C
Accumulated power on time, hours:minutes 36736:41
Manufactured in week 43 of year 2013
Accumulated start-stop cycles: 322
Specified load-unload count over device lifetime: 0
Accumulated load-unload cycles: 0
Elements in grown defect list: 0
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 373.721 0
write: 0 0 0 0 0 724.017 0
verify: 0 0 0 0 0 26705.219 0
Non-medium error count: 6
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Completed - 36656 - [- - -]
# 2 Background long Completed - 36513 - [- - -]
# 3 Background long Completed - 36507 - [- - -]
# 4 Background long Completed - 36471 - [- - -]
# 5 Background long Completed - 36442 - [- - -]
# 6 Background long Completed - 36431 - [- - -]
# 7 Background short Completed - 36431 - [- - -]
Long (extended) Self-test duration: 90 seconds [1.5 minutes]
Contacted the seller, so let’s see how that goes.
Created a post on the STH forums about the SSD issue as the seller offered to send new drives. However if it indeed is a firmware issue then they’ll just die again once they reach whatever threshold is the problem.
The good news is that they just go in read-only mode. Since after first one died I just ran the pool in degraded state while waiting for a replacement.
Of course, once the second drive bombed out I kinda didn’t have much choice anymore. Since all there is on these is the OS (reason I ran it degraded at all) I just dd’d the partition of the newest failure onto a “new” SSD (one of the HGST 800GB SSDs that I have as spares for the VM array). Put it in the server, rebooted and I was back in business.
I did find out that not having _netdev as a mount option for iSCSI mounts results in systemd bombing out on boot (I mean, does it really have to try to drop into a rescue shell when it can’t mount a random disk?). Ironically if I just Ctrl-d’d to continue booting the iSCSI drive was mounted without issue. So yeah #justsystemdthings I guess…
i remember hearing about this… on hpe or intel drives also
Yes, looked up some of those articles, but didn’t find this specific model mentioned.
At least the HGST SSDs I have are past the highest “sudden-death” threshold, that I managed to find anyway… (40k hours).
I’ve always found ‘rolling my own’ email overwhelming. I need to bit the bullet and set something up to learn.
Ever think of downloading OBS and making videos of this stuff and posting on YouTube?