Atermiter X79 motherboard - PCIe lane issue? Chipset issue?

[chaslinux]

I’m running into an issue getting a proper POST from my X79 motherboard when I introduce a new PCIe (Fibre channel) card to it. With the added Fibre Channel card the motherboard hangs on the American Megatrends BIOS screen with an A0 code. When I pull the card the system posts fine.

I’ve franken-monstered my system a bit. Here’s a bit of context:

I’m running a XEON E5-2690v2 CPU, which appears to have 40 lanes according to Intel’s documentation on the chip, so I think it’s not the issue.

I have a Samsung 980 1TB NVMe drive in the NVMe slot.

I have a Radeon R9 380 card in the 2nd PCIe slot (since the NVMe has a large heat sink that prevents putting the video card in the first PCIe slot).

I have 2 x 3TB drives connected to the SATA ports.

Here’s the kicker: I also have 3 x optical drives connected to the SATA ports (it helps a bunch when backing up physical media).

As I mentioned earlier, when I put the Fibre Channel card (which I think is PCIe 4x) in either the top or bottom PCIe slot the system starts to post but hangs with an A0 post error.

Here’s my plan of attack, thinking to try the following solutions:

• First remove the 2 x 3TB drives… I have storage elsewhere on the network, and I can use these in a caddy.
• Replace the NVMe drive with a SATA SSD. I’m not that hung up on the NVMe, and if it lets me run the fibre card I’d buy a 1TB SATA SSD.
• Remove 2 of the 3 optical drives. It sounds silly, but I’d like to do this last since I’m still in the middle of backing up a lot of my physical media collection (sent over the network to our media centre).

Q: Is this a PCIe lanes issue? CPU seems okay with up to 40 lanes?

The motherboard has 3 slots that appear to be 16x length (the bottom one might be 8x, but pretty sure I saw the top two are PCIe gen 3.0 16x). It also has a couple of 1x slots (not in use), and of course the NVMe slot - in use.

Lastly should mention PSU is a Corsair 650W I picked up a few months ago. It’s been running fine, so I don’t think it’s power-related.

These motherboards are a bit bizarre in that when you buy the motherboard you’re not guaranteed a particular chipset, so it’s possible the chipset is saturated. But I don’t think this is the case since AFAIK those 16x slots should be controlled by the CPU, not the chipset.

[Sean_Dudley]

Hi there!

(Disclaimer: not a pro sysadmin or fibre channel expert!)

1.Is top or bottom an absolute requirment: have you tried the card in any other slots & are there any pcie devices you can remove/replace just to narrow down the potential obstructions? Also, does the offending card work in the media center pc/server?

2.This answer [Diagnostic code "A0"? | Tom's Hardware Forum] doesn’t clarify anything to me but is included for the sake of completeness. Is it possible that your bootloader isn’t configured properly for this and can’t take control (or a driver issue maybe)? You may be right that removing drives will solve this.

3.I’ve had similar problems of hanging on POST by having the wrong memory type installed (e.g. ECC RDIMM instead of the required ECC UDIMM). Not sure how realistic this one is in an otherwise working system, but I’m given to understand these HBAs work by reading from system memory by default, so it might be worth investigating(?).

[chaslinux]

Hi Sean, thanks for your answer. Here’s what I’ve tried since:

I pulled all the SATA cables and the NVMe drive. This left the R9 380 and the fibre card in. At this point I got a different BIOS error code (92), which seems to indicate an issue with an add-on card.

I happen to have a different model fibre channel card, so I tried it, same 92 result.

Thanks for the note about the RAM. I’ll check this next. My system does post without the fibre channel card (with the nvme and r9 380). It’s possible I have ECC RDIMM as I didn’t order the RAM with the motherboard. I’ll pull a stick to check.

I also noticed the BIOS on these motherboards have a zillion PCIe settings. I didn’t do anything unusual with the BIOS so I suspect most things are auto. Will do some research into that next. I do have an identical card in the media PC (which has a Sandy Bridge Gigabyte motherboard, NVidia GTX 980, but, obviously no nvme drive).

Thanks again,

Chas.

[chaslinux]

Another update. I do in fact have ECC RDIMMs, but swapping them for non-ECC unbuffered RAM didn’t seem to make a difference.

To be clear: both types of RAM post fine and my system loads with the NVMe drive and the R9 380 video card. I’ve been using the ECC RDIMMs now for several months. It’s only when I add the PCIe fibre channel card that the hanging issues appear.

I removed all ECC RDIMMs and put the unbuffered non-ECC RAM in and the system posted fine without the fibre channel card. When I added it I got A0 again (with the NVMe in, 92 with it pulled). So power on the PCIe bus seems to be likely the issue.

I think I mentioned in my reply to Sean that there are a lot of PCIe-related BIOS settings, but I haven’t swapped anything in this area from factory settings.

One thing I noticed is the motherboard also has a PCIe power connector which I don’t have power plugged into. I do have PCIe power cables going into the R9 380 card, so I figured this wasn’t needed. But now I’m thinking it could help as issue 92 seems to be related to PCIe power. If I leave the fibre channel card out, everything is peachy without the additional power.

[Sean_Dudley]

Sorry to take so long, getting back!
On a recent stream, @tekwendell suggested, increasing the pci bus speed, to fix an issue someone was having.
This definately has worked for me (I think), in dealing with SAS-HBA boot issues, but in my case this involved; increasing a ‘PCI latency’ setting (from 32 to 128). Maybe looking for something like that might help?
Sorry again, for the delay!