My experience is rather limited to what I have done so far, and all of that is after Ltts video on infiniband with CX2 qsfp QDR cards, or now, OEM HP mCX345 fcbt (I think) FDR / 40gbe VPI cards.
I first of all would advise against infiniband since software support is very limited.
To have common software use the link, you need “IP” packet simulation and that t is done via “ipoib” which is CPU bound.
You can get up to about 27Gbit on ipoib as far as I have seen poster on some forums with heavy tweeking, which is better then the 10GbE VPI mode that comes with the Cx2 VPI cards, but Cx3 40GbE is just easier and less CPU bound.
And since its Ethernet, all software will use it.
And also, smb direct only on cx3 and above.
In case you want that.
Let me now try to answer your precise questions.
“Does it have to be one single run of cable”:
No, but Direct attach copper cables have transcievers on both ends and you shouldn’t even think about cutting and splicing that!
I think fiber is OK to be spliced or trunked, though I have never done it.
Logic said its fine, but there should be some “gotchas” with the fiber polarity, and the loss over the run.
Better wait till someone else green lights this.
“Does a QDR switch support 10gbe”:
Well, it obviously depends on the hardware,
but if the switch is advertised as QDR and or 10GbE it should be fine. Mellanox calls this “VPI” for Virtual Protocol Interconnect if I’m not mistaking.
There are EMC branded mellanox 6XXX fdr switches for example, that come with OEM software that only supports infiniband.
Sth has a long thread about those, and how to convert them if possible.
So without software trickery, no 10Gbe or 40GbE on those.
In terms of transcievers, those seem to be very accepting, I had cx2 for point to point 10GbE on the cheapest QDR DAC cables possible without problems, and the same cables are now doing 35Gbit point to point on my cx3s in 40gbe mode.
But transcievers come with a bit of software on board, so it probably depends again.
If advertised as QDR, they should be OK though, as seen in my 40gbe example.