I wonder if rook.io performs well on a pi. Say you get 6 Pi4s with fast USB storage attached how much this will work towards distribution IO and network requests. I can imagine a tidy nice setup using PoE.
I wish RPis had encryption extensions.
Beyond the usual issues of lack of ARM software to replace x86/x64 software, the performance gap is still a factor as you won’t have the the I/O speed benefits–using two USB 3.0 SSDs will require a powered hub or a hat with M.2/SATA that has external power so rack space wise it won’t be easy from a cable management angle. There are silent 1U ARM servers with 8-32 cores which support RAID and make better sense than a Pi.
From helping others run ARM clusters(mostly Blender), your mileage is going to vary as power delivery will be a mess if/when expanding the storage options. If I recall there were a few Pi specific accessory/case makers which have pre-fab cluster solutions for 4+ Pi4 setups with SSD in mind, however space for a PoE hat isn’t there.
At the moment I’m working on a custom ARM cluster solution but its more about being airflow optimized with lots of noise dampening.
RPi most likely avoided going in that direction for cost reasons just like why they’ve never offered an eMMC storarge module connector… at least on the Pi 4 there is native boot from USB 3.0.
Rockchip based boards like the Pine64 have encryption, only downside I’ve found is being stuck using Armbian as some stuff is either broken on Ubuntu or certain Hats aren’t supported on it.
One 2"5 USB SSD for each system. I acknowledge the I/O issue but my original question was if Rook.io (which provides Ceph) can scale horizontally. Having reads and writes distributed between 3 or more RPI4s each with a Single USB3 SSD and a Gigabit connection might yield faster results and better redundancy the more nodes you have.
You need a single ethernet cable for each system connected through a PoE hat. Each system with a short USB cable connected to a 2"5 drive.
Do you really need RAID with Ceph?
Big server might work for OP considering he is already on the homelab boat. Or not if he is trying to get away from it. I personally don’t like having power hungry rack mounted noisy stuff at home. I like experimenting with stuff but I’m not really into this homelab scene.
It still depends upon the SSD and/or enclosure’s chipset, I’ve found a low priced enclosure I had bought(ASmedia) ran worse on Linux(ARM and x86/x64) so your mileage is going to vary.
No not enclosures. Single disks. An army of Pi4s with a PoE hat plus something like a Samsung T5 distributing reads and writes across the cluster.
When it comes to home labs in a hobby side, I’m not sure many would be splurging on a Samsung T5. For cluster usage I’ve found cheap SSDs work reasonably well for AI image/video processing with a Jetson Nano-Pi combo as long as the enclosure doesn’t become a bottleneck–PNY’s CS900 are quite snappy even if they lack cache.
This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.