Theory crafting a system for maximum simultaneous Micro SD card ingest

Hello! I am looking for some advice for reworking a system to be able to ingest data from as many micro sd cards as possible at one time.

At work, we take a lot of footage and we need to get that footage from the various sd cards and ingest it and copy it to a local server. Currently we are running a Threadripper Pro 7965X system which holds a bunch of USB addon cards. these breakout into USB hubs that hold SD card readers. I think right now we can do roughly 200 at once with our current set up. Getting this up to as high of a number as possible would be ideal for us. Pie in the sky goal is in the neighborhood of 1000 cards at once.

I was just curious if there some other solution working off of our current Threadripper pro system that someone out there knew about that we don’t. Some franken combination of other PCIE cards that could potentially unlock more sd card slots at once.

i can provide more details if needed but some things i am under an NDA so i may not be able to answer specifics. I am not sure if this is even possible, but I told my boss i would ask around for ideas. thanks!

Not going to happen on 1 system.

1 is usb total connected devices is 250 to 270. I know LTT did a video on that a while back.

2 at that many devices trying to write to a ssd / hdd / nvme drive is going to be saturated to the point of it taking days to write out all the data. Not sure if a low end nas would even do it. The higher end nas with tons of drive thru put won’t be an issue.

The fundamental problem is that USB is congestion. USB works like old school ethernet hubs. Every packet in is rebroadcast to every other device on the same bus, and every USB port/daisy chain from a given controller just makes the bus bigger and bigger, like adding hubs on hubs on hubs. Latency gets worse and worse because each device has to wait for its turn to talk on the bus, and only one device can talk at any given time.

So, in reality, you’d need to create a custom PCB with somewhere between 25 and 100 USB controllers for the 100 USB devices you want. See where the performance degredation really hits you. 4 devices per controller? 1? 7? Once you know how many devices, in your use case, a single controller can handle, you can go about designing (or having designed) a custom PCB that will have enough hardware for what you’re trying to do.

Keep in mind, USB controllers are not cheap. Somewhere between $1 and $3 each. For every controller, you’ll also need a physical USB port (they can’t be shared because, as mentioned, bus architecture). Those are also around $1 each.

The next challenge is how to get all that data off the board. 60MB/s * 1000 = 60,000 MBs. That’s 60 GB/s. So 100GB fiber optics could do it. However, the physical space needed for 500 USB ports is so massive that you’d probably end up making many smaller devices and networking them together rather than having a single monolithic board with 500 ports on it (I’m assuming you can at least share 2 cards per controller without congestion). So you’ll probably want to design some sort of fiber channel/fiber ethernet solution into the board as well, so that you can plug all these devices into a switch. Ethernet only really makes sense if you are building individual boards that only soak up data from 20 cards each or less. (60 * 20 = 1200 MB/s, which is right around the limit for 10Gb ethernet).

Either way, you’re going to need some sort of infrastructure level switch to handle all that data, with a backplane capable of handling the 100GB networking load.

The final piece is the custom firmware to drive all those controllers and keep them working, aggregate the data together, and ship it off to the network interface for handling. Plus whatever software you use on the server side to ingest 60GB of data and do something with it (this is no trivial task).

This is…a monsterouly complex project to do as requested, as you can see. The other option, buy a LOT of cheap mini-PCs, or raspberri pis, and treat them as $5 microsd card readers. You’ll end up spending a lot getting power supplies and networking gear for the 100s of devices, but that would work too.

If you do go the 100 raspberri pi route, get benchtop power supplies and use that to create a common bus that drives maybe 20-30 pis each, rather than getting a wall-wart for each one. It will be MASSIVELY more efficient.

Wifi with so many devices so close together won’t really work, so expect to have to use physical networking of some kind, so an ethernet hat for each pi.

1 Like

thank you for your extensive reply! this is more or less what i expected to hear but i wanted to do my due diligence. it’s certainly above my skill level to implement (i only build computers really) but i will bring up the mini pc idea. I think the realistic option is they just need to have more set ups like the first one if they want more cards being read at once.

thank you!

I’d say it’s theoretically doable over PCIe in some fashion or another.

One would need a sh**load of PCIe-SD controllers and a tree of PCIe switches.

SD duplicators are a thing. Which you might benefit from if you want to erase 200 SD cards at one time.

Why? Seriously why?
Is this about some fun fantasyland build? Or is this a serious question?

If the latter, what are you currently using? What are the limitations?

right now we have a threadripper pro 7965x that runs pcie USB C cards that connect to hubs that run SD card readers

we record hundreds of hours of footage a week onto hundreds of cameras a that all record onto micro sd card.

so right now, we can upload around 200 cards at once on one system. the limitation in this system right now is that the motherboard we have only holds 6 PCIe cards. and also based on one of the other replies, we will probably hit some kind of ceiling because all of this runs over USB, but that has not happened yet.

1 Like

MEthinks you could use PCIe switches.
Each PCIe slot would connect to PCIe switch at the trunk, with N branshec downstream.

Something like this, only on a card:
Microchip Switchtec™ PFX/PSX Gen 5 Fanout PCIe® Switch Family

Biggest model has 100 PCIe5 lanes that can be split into 52 ports.

So, with 16 PCIe5 lanes looking at your Threadripper per port, you could have 50 downstream ports PCie5x1

Or you could use EPyc board with bazzilion MCIO-FF-TA1016 connectors ( each carrying PCIe5x8 for a cable):

ASRock Rack GENOA2D24G-2L+

Using such a switch on each such PCIe5x8 cable could get you
much wider tree…

And dual EPYC system board would get you much more lanes to begin with… :roll_eyes:

Do you need that much CPU power for this task? You could get 2 or 3 Naples/Rome EPYC machines with the same number of PCIe lanes (but 3.0/4.0, not 5.0 - but I assume your USB cards are not PCIe 5.0) for the same cost, more if you go second-hand.

Oh, one more thing - there is a new kid on the block WRT PCIe5 switches - Kandou.

Their ultrasmall PCIe “Zetti” switches have just 6 lanes total.

It’s a weird small (and probably ultracheap) switch that might be just what you need.

You could Bifurcate those PCIe5 lanes into PCIx5 ports and use one Zetti switch to convert each one of those to 4 or 5x PCIe3x1 ports, which you could use for your PCie->SD bridges…

If that’s not enough, you could split those host PCI5 ports before that through a couple of big PCIe5 100+lane switches from MIcrochip or Broadcom…

Another thought is can any of the cameras be connected to your network and then either save the files to /or upload a network share?

So you are currently inserting 200 SD cards at the same time and write that to where? The local machine or some network share?

What speeds are you getting?
Is speed a concern?
Or is it just important that you can insert 1000 SD cards?

You need GPU mining boards.

18x x1 slots, so you can host 18 USB cards per board. One x16 slots for a fast NIC.
That gets you the ports you need as USB has inherent limitations (as stated above) so the only way around it is more USB controllers (also, as stated above).

This reasonably can give you 400 MB/s ingestion per card, exactly what you want to offload 2 high performance SD cards simultaneously.

So you’ve got 36 SD cards ingesting at 7200 MB/s without chaining per unit.

Getting it off the rig will require 100 Gb ethernet and a wicked NAS.

This will let you dump a 4GB video file in 20 seconds. Hardest part would be keeping it fed as you’ll be swapping cards non stop.

Add in some chaining to increase cycle times and lower bandwidth, and you can easily beat your current setup. Reasonably you’d be sub $1000 for this setup.

4 Likes

Now, the best way you can alleviate this is to do like we do and run it all back from the cams to central servers.

You can use fiber or plain old ip over ethernet. SDI is dead outside of racks. Obviously, you’ll still want to record locally in the event of frame drop. but this scales infinitely.

Unless you’re shooting outside documentary/reality style workloads, this is the superior method as a single snake can keep everything fed and your time sync is SOOOO much easier to maintain since it’s captured at the cam op.

How often are you doing these ingests? TBH, it sounds like your corporation got itself into an insane unnecessary problem by not networking its cameras and 1) requiring physical access and intervention to save the data and 2) allowing physical access to compromise the data entirely.

Could you stagger the ingests? Meaning, if you do this weekly now, instead of 1000 simultaneously every week, do 200 daily?

Other thought, have you run benchmarks to see if you’re actually performing the task more quickly trying to do 200 at once rather than 50 or even 10? Or this just “I can pop it in and walk away or let it run overnight.” for convenience? Can the storage system even handle 1000 cards at once if it did prove to be faster?

What it sounds like you really need is a robot like old tape systems used, but with more modern automation. Ideally, you’d dump your cards into a box or hopper, the machine would pull 1 out, place it in one of its multiple readers, remove it when told that card is done, and place it on a stack to go back into the field.

2 Likes

It’s obviously could be for covert mass surveillance as part of globalist silent/covert armies.

They use civillian swarms ( that’s why they REALLY need immigrants) to plant the stuff in apartments and offices and cleaners etc daily rotete the SD cards etc.

Then someone reads them en mass and they go to AI scan and archive.
They keep it otuside network to keep them low profile and minimize the leaks.
This is just an evolutive step of their crap that they have been doing, for example in EU for decades. Check the while thread:
New bits about “Project Gladio” ar coming out. It was based on old Nazi “Werewolf Project”, but with new twists.

But as they are evolving, so are their tech-toys. In EU vassal states they use RF BrainScan as their leading edge - essentially irradiating human bran with a couple of RF beams and using SDR to scan the feeble reflections for the fingerprint of brain activity.
Check the whole thread:
Remember the RF BrainScan that was R&D-ed by usual “humanitarians” on a MASSIVE scale across EU vassal states during the last decade ?

I really don’t know if OPs company is really stupid or if OP is just making up a scenario in his head.

Having to read form 200 SD cards simultaneously, really sounds like a fantasy from someone who never worked a minute in video production :grin:

What is the average card read speed? MicroSD has a range of possible speeds (from a few megabits/sec to around a gigabit). Several people have talked about total bandwidth, and that could be a limitation if the cards (and readers) are really fast.

USB is a tiered-star setup - you start at the root, and that can branch out (with hubs) to multiple ports/functions. Some of those functions can be additional hubs. There’s a good description here: USB Component: USB Network

There is a limit of 5 hubs between the host and any device. There’s also a limit of 127 devices per host (not per port). Hubs are also devices, so the number of card readers per host controller will be less than 127.

You can get a 4-host/8-port controller card here: https://www.bhphotovideo.com/c/product/1701838-REG/sonnet_usb3c_8pm_e_allegro_pro_usb_c_8_port.html
I’ve never used that one, but it’s a good brand, and it has 4 hubs and a PCIe3x16 interface, so there should be a lot of bandwidth.

You could have close to 1000 readers from two of those cards if you want to go crazy, but I’d suggest the following:

4x Allegro host cards, providing 16 hosts and 32 total ports.

Each port then gets a 4-port USB3 hub, with each of those ports connected to a powered 8-port USB3 hub. Plus a reader into each port, and voila - ungodly numbers of MicroSD slots. This setup provides 32 ports using one upstream interface. There’s a lot of flexibility with the arrangement of hubs, just bear in mind that a lot of many-port hubs probably have multiple hub chips internally, since most chips top out at 7 ports.

You can make up enclosures (32 plus some spares) with the 5 hubs and 32 card readers, power strip, etc. If you have card readers with multiple slots, you can reduce the number of USB ports. 32 ports times 32 card readers = 1024 total.

You’ll likely run into physical limitations though - long USB cables can be problematic for higher speeds, and you’re gonna have one heck of a medusa setup :wink:

The big trick here is the multi-host adapter cards. That’s what makes it possible to have a ridiculous number of USB devices attached.

Note that there may also be software / driver issues with that many devices - you never know if there are hard limits in the code until you do something this crazy.

Good luck :slight_smile:

thanks for all of the replies! because our video files are decentralized and not at all local to one place, offloading via network is not an option for us.

from talking with them, the way we run it now they are already having some issues at the current scale, so i don’t think adding more USB readers to one system is going to be the path forward.

also yes, the threadripper and the motherboard are completely overkill for this task in my opinion but i did not pick these parts. i just put the machine together for them.