I am looking for a dedicated fast networking setup for shared storage and the Proxmox backplane. However, I see a bunch of cheap Intel network cards but I don’t know what they are compatible with. Can I use any old SFP+ connector? If any connector is fine do I need to be worried about stability under Linux?
I am pretty much looking for a network card guide for Proxmox.
Remember a network connection is only as fast as the slowest component. So are wanting to know about NICs or networking?
Most™ SFP NICs should work under Proxmox.
NICs as they are going to be directly connected. (No switch)
Then you really don’t have anything to worry about other than the specs of your pcie interface. A single port 10G NIC should be a home in anything modern. And my recommendation is to get a NIC with an RJ-45 port rather than SFP+ port for the sake of simplicity.
The 10GTek Transceivers are really good as far as comparability goes. They’re cheap on NewEgg and I’ve found they work with most of my NICs including standard Intel gear.
You said you’re not using a switch, but if you decide to add one, I recommend MikroTik as they have good options that are affordable.
10GTek also makes directly attached copper cables (with the transceivers permanently attached). These are really great for short runs while still getting the speed of Fiber, and might be what you’re looking for.
(photo from NewEgg, but you can find them lots of places)
I am looking for a card that can fit in a small form factor system. I need the card to have two ports but the two port devices all seem to be to big. I need a half sized bracket with a card that isn’t to big. I don’t necessarily need SPF+ but from my research there isn’t much of a cost difference.
Any ideas for a card that can fit in a small system?
I run this on my TrueNAS server. It comes with the 1/2 hight plate.
In my experience, Intel X710 and E810 just take any Transceiver or DAC.
Switches on the other hand, some are very smug about both.
Ok I am looking to check out and I need a sanity check as I feel like I am screwing up.
I am getting a Intel X520-DA2 Dual Port 10GB Network Adapter E10G42BTDAG1P5 HH with SFPs 901227
For the cables I am going with a 10G OS2 LC to LC Fiber Optic Cable Singlemode LC Duplex UPC OS2 Fiber Patch. It looks like it should fit the SPF on the card but I am scared I am going to burn it. Where do I find information about length?
X520-DA2 supports both, SFP+ and RJ45 (says SFP right under your circle 
), just depends on who designed the board and connectors around the chip.
Usually in the datasheet for the transceiver. I am running these at home.
For lengths of 5 meters (25 ft) or less, DAC’s may be preferable for cost reasons.
I get that but I don’t understand wanting SFP+ if copper is your transmission medium. It adds expense without a benefit. Now if one wanted a fiber connection then it would make sense.
Without a sketch of what needs to connect to what, I am guessing as blind as you are.
You probably can find used intel x540 10Gb cards below $50 on ebay. It is picky about the switch you use. It may work only at 1Gb if you pick a wrong switch. For point to point network, I don’t see any issue.
If you have more budget, intel x550 is highly recommended. I don’t find any issue with those.
The cheapest option might be something like intel x520-da2, with direct attach copper interfaces.
Thanks for the catch
I forgot about the Intel product pages and was just trying to go by connector. I think I will hold off and do more research
How many PCI lanes have you available on the storage and the proxmox nodes?
Is power consumption a factor?
Do you mind buying used?
In any case, the cheapest option you have for direct-connect networking in proxmox at 10Gbit or above is old infiniband cards:
These run 20USD apiece
cables should be 10-20USD depending on length, you do want to keep servers at 3meters or less from each other, 5 meters max …
Copper uses way more energy and also is a lot more expensive.
I also run some old Chelsio 520. The thing about these old cards is that they are EoL and sometimes miss some power saving features or hinder the system to save power because the don’t support ASPM. Also mine needed an extra fan to stay under 75°
I understand the benefits of fiber, that doesn’t change my statement. At the end of the day the question is what transmission medium is going to be used.
