V100s 32GB PCIe ECC Errors

High-Performance Computing Machine Learning, LLMs, & AI
1

I recently purchased an ebay Tesla V100s 32GB for LLM usage in llama.cpp

I power limited it to 200W, and have never exceeded 78C die temp.

Within a day of use, I got a hard crash and uncorrectable ECC errors. This event resulted in 32 retired pages. When I received the card, there were zero retired pages.

After this event, I ran the card through 2hrs of gpu_stress and got zero errors. I have been using it for a few more days with LLM workloads and have seen no further ECC errors in my kernel logs or nvidia_smi.

Should I return this card?

I don’t have enough experience with datacenter GPUs to know if I’m getting ripped off, and the product will degrade further.

I appreciate any insights.

image
image1417Ă—601 148 KB

2

Pull the cooler, clean and repaste the GPU, and replace any torn or misaligned thermal pads.

3

Yeah I’m kicking myself for not doing that before installing it. I have assorted thermal pads and paste on hand :man_facepalming:
So you would not return it? In theory I have ~30 days to return it if anything fails.

4

Can always reach out to the vendor and see what they suggest.

5

If the OEM paste was never replaced it’s very likely dried out and cracked… worth a shot before exchanging for a similar vintage card with old paste.

1 Like
6

I did some more research and discovered that the hard limit on retired pages is 63. After that the card will just encounter memory corruption if more cells go bad. The expected retired pages at this vintage seems to be “a few” pages, not half the table…
A burst of errors could indicate the card was previously worn out or damaged in non-ECC mode, or maybe the seller reflashed it to clear the tables. Either way, I will return this as it’s too much financial risk when I can just re-purchase another one.

Thanks for the helpful comments!

1 Like