Chef Nick: Modern Problems require Modern Mega Upgrades

wendell · May 6, 2026, 3:40pm

Who is Chef Nick?

I didn’t know either! But they have a large operation and as such they should probably not be operating from a filing cabinet full of USB SSDs…

What We’re Starting From

Look! Look at this pile-o-wires! Up in the upper right is the “ISP Wireless” Chef Nick mentioned. Madness!!! I’m really channeling Ozymandias here…
Something about weeping, despair and looking upon this..

So this turned out to be a project – much larger in scope than I expected when I started. When you look at a project like this it’s not just dropping off equipment.. it’s helping change mindset and helping build durable processes.

They knew they needed “a server” – but what… beyond that?

Enter 45Drives.

The cornerstone – the key piece of technology here that saves the day – is 45Drives. Simultaneously unglamorous and the single most important piece of equipment to Chef Nick’s entire operation.

We talked workflow. Can we offload footage from the cameras in real time? Yes.. their studio cameras can do that. But they aren’t yet ready to adopt those processes. For now, the Storinator will be a smart file server. Mac ACLs , synchronization and other similar tasks.

Soon, maybe, streaming footage directly from Camera to Storage. It’s a possibility.

Server Part Deals

Ah, but a server without drives is not a server at all. Enter server part deals:

18 * 27tb (They’ve already upgraded to another 36 27tb!)

We also added a bunch of SSDs so they could have a fast flash pool as well as mechanical storage pool.

But then with the storinator in place… we had a NEW problem… They had ONLY the server. You saw the mess!! There was nothing to plug it into.

UniFi to the rescue!

Fortunately, Ubiquti answered my calls and sent over the motherlode

UniFi Enterprise Fortress Gateway

UniFi Pro Max 24 PoE 10gb SFP backhaul to the Top of Rack Switch
UniFi Network Video Recorder Pro

Here’s our Top-of-Rack 10/25gb switch. There are almost 20 10 gigabit connections here in this rack. Editing stations and high speed devices connect here. The Storinator has a 2x25gb nic and uses the uplink ports on this switch.

unifi network cameras page

Finish Server Closet

The cheese was thoroughly not radioactive. I know because of my Radiacode 103. It was delicious, though. That’s like a $5k block of cheese!

Misc Rack Bits :

In addition to our Gently Used APC Netshelter and the Unifi Network switch We also used some metal mounting rack studs I need to link. And a rackmount harness for all their mac studios that worked well!

I saw macs! In a rack! And I’m not Dr. Seuss

RackMac Studio - Sonnet That’s what they used for moutning their Mac Studios.

Patchbox also supplied Chef Nick with misc mounting hardware for all the rack gear.

Noah and all the Chef Nick staff were great hosts! They’ve got a really serious YouTube operation going and, I think, I might have surprised them with how unserious I am about our YouTube operation. I’m in it for the technology, solving problems and the adventure – the lulz if you will – and, occasionally, the unsolicited hug.

Thanks Noah and the gang at Chef Nick’s studios for hosting me, and taking up the mantle of being the responsible party through what is I’m sure your own life adventure of supporting a ~ 40 million sub youtube channel!

Dutch_Master · May 6, 2026, 8:04pm

Would have been great if you’d documented the process of building/installing/migrating their stuff. I recall a YT channel FibreNinja (who has been offline for years at this point, the original channel has disappeared now too) who showed his job as in progress, really informative video’s to learn about networking etc.

wendell · May 6, 2026, 9:17pm

We can do that in a generic way, but many people are sensitive when it comes to the fine-grain details. It’s not a lot different from a basic setup with a couple security groups, some different shares for different people and then special-to-them software configuration.

What do you imagine is magic you didn’t see? I could probably cover it in a separate “this is how I’d do it generically” type video.

Dutch_Master · May 6, 2026, 10:00pm

I understand the sensitivity, especially when security is involved. It’s not so much how their operation is set up and divided on the S/W side, more like how to route cable (best practices), tools used, possible upgrade paths for future expansion, etc. Jeff Geerling recently put up a video routing a network cable inside his studio, which IMO was quite entertaining as Jeff is not exactly a network engineer

PS: earlier this year I suggested a video idea to Amber about yourself and LMG sys-admins Shawn and Dan to talk all things admin. I also suggested same to LTT. Amber liked the idea and I hope it’ll be out soon ™? I noticed since Linus has “proper“ admin staff on his payroll your appearances on the LMG channels are sadly pretty much absent (I know, you’re busy and stuff, but still a sad loss)

jebb · May 7, 2026, 8:20pm

I would have loved to get more of the gory details on the storinator setup, in particular as far as the choices and trade-offs you made on the zfs topology. I remember now I had the exact same thought when you helped upgrade Gamers’ Nexus’ storage server a few months ago.

Noah (is it Noah?) in the bit at the end of the video says “we’ve ordered 18 more hard drives that are all running on a separate VDEV from the original ones so now we have 4 discs of failure prevention…”, but that really raised more questions than it answered for me (pretty sure you wouldn’t have set up all 18 HDD in a single RAIDZ2 VDEV).

EniGmA1987 · May 7, 2026, 10:24pm

Not to derail this too far, but I saw you use the EFG and I immediately thought of this @wendell

gist.github.com

https://gist.github.com/galvesribeiro/89ce0232c8bc1971af84aee84746dc66

EFG-Broken.md

# Why Your Ubiquiti EFG Can't Push 25 Gbps Inter-VLAN — and What's Actually Going On

## Or: How I Reproduced the Problem on x86, Tried to Load the Missing Modules on the Real Device, and What That Tells Us About Ubiquiti's Kernel

---

## TL;DR

Ubiquiti markets the Enterprise Fortress Gateway (EFG) as a 25-gigabit-class router. The product page lists two 25 GbE SFP28 ports for WAN/LAN, and Ubiquiti positions the device as a flagship for medium and large enterprise deployments. Its silicon — a Marvell Octeon CN9670 — supports hardware-accelerated forwarding through purpose-built network engines (NIX) that should sustain tens of millions of packets per second. The UDM Beast, Ubiquiti's next-generation gateway, pairs a Marvell Octeon CN10K SoC (with ARM Neoverse N2 cores) with a dedicated Marvell Prestera-class switching ASIC accessed via PCIe — capabilities that, properly used, would offload most of the per-packet forwarding work into hardware.

This file has been truncated. show original

What do you think of the test results? and have you seen similar? Seems so unfortunate to have 25g ports and not even be able to push 10gb inter-VLAN traffic.

wendell · May 7, 2026, 10:43pm

Yup, and I talked about this in a review…somewhere… it CAN work when the switching chipset handles the traffic but NOT when the cpu handles it. I have a little expose coming on the Poe extender switches – they’re limited from 2.5 gigabit switch chipset switching capacity to like 500 megabit when it has to be done by CPU.

I have an nvidia bluefield 2 I have used in a similar way to program and hammer and I’d be curious about more specifics from your testing because I focused on DPI sides of it i.e. intrusion detection.

also theres one thing this analysis misses and that is the switch fabric of an upstream switch can manage the traffic from port A to port B. I.e. you could do 25g cut through on a 25/100g aggregation switch if the control plane on the EFG lets the asic know in hte top of rack switch the traffic is permissible between subnets and the rest of the configuration is valid (remember, sonic is capable of this but… unifi’s plumbing for that is indeed a bit sus). In this kind of scenario the inter-vlan traffic never hits the efg at all. But that might be an a-typical config for what you’re imagining.

Stiehl · May 9, 2026, 6:46pm

What kind of vdev layout did you go with? I assume they mostly edit off the flash pool, then move it to the slower rust pool when content ages out.

I’d probably have done 6 wide Z2s here for the rust, unless they were planning to edit directly off it.

I hope they have some sort of actual backup solution, both on and offsite… all well and good to set up a giant storage array, but it might as well not exist if they don’t have backups!

Also, that is a LOT of hardware that got delivered really quickly considering you didn’t know you were going to need all of it when you went onsite.

I wish my delivery services were that reliable!

gee_one · May 10, 2026, 1:45am

What were the red siren type lights for? There was one at around 5:50.

Was it just a prop or was it used to signal something, like don’t unplug drives because files are being ingested.

ozlay · May 10, 2026, 9:28pm

No UPS in the 12u rack?

wendell · May 10, 2026, 10:33pm

two cyberpower 1500s, in the bottom, but the usb interface was frustrating to use and they are a little undersized. I’d probably instead recommend eaton

pazusiti · July 21, 2026, 11:54pm

Feels like this upgrade is going to save you a ton of headaches long term. I’ve been down the USB-SSD chaos road myself, and centralizing everything made a bigger difference than I expected. One thing I found handy was keeping a small scratch SSD pool just for active projects so the big array never gets hammered by constant reads and writes. Keeps things snappy without messing with the main storage.