Server Crashing: Epyc 7402P with Nvidia Tesla A100s Ubuntu 20.04

Swamp7 · September 13, 2023, 3:35pm

Seeing if someone could give me some guidance on troubleshooting a new AL/DL Epyc 7402P server with Dual Nvidia Tesla A100 40GB GPUs. This is my first Epyc server (moving from Xeon E5-2680 v3s…I know very old but they work for GPU servers very well, all of the load is on the GPUs) and I had a ton of problems with the original Threadripper 2920X back on Ubuntu 16.04 when it launched…

Motherboard: Supermicro H12SSL
RAM: 256GB ECC 2133 Samsung
CPU: Epyc 7402P
GPU: Dual A100 40 GB
Storage: 1TB 3.0 NVME OS, 4TB 4.0 NVME for datasets

It has been running for a few weeks now with no issues but this morning while ssh in, I have this error. Searched on for solutions online but nothing to pinpoint the issue. The BMC shows no Health Event Logs or component problems. From what I can see in the logs, the problem randomly happens only while starting a new python job via docker container…maybe a problem with RAM and swap space filling up?

This is a research server for a university and I need it to stay stable, it is already book for the next few months, and I need to finish my own research work.

Any help would be Greatly appreciated,

Thanks,

Jake

MadMatt · September 13, 2023, 3:44pm

is the python docker container accessing the gpu via vfio?

Swamp7 · September 13, 2023, 4:10pm

No passthru or VM everything is running on bare metal…create a new user with elevated privileges and assign a pquota usage limit for the users docker container data with XFS.

MadMatt · September 13, 2023, 4:39pm

Look at this:

github.com/pytorch/pytorch

DDP gets stuck on A40 GPUs

opened 12:37PM - 22 Feb 22 UTC

closed 08:18AM - 24 Feb 22 UTC

hhb072

### 🐛 Describe the bug I tried to run a DDP code on A40 GPUS and get stuck at… the first iteration of the training model. The code can run normally on a single GPU. Below is a simple test code borrowed from [[email protected]](https://discuss.pytorch.org/t/nn-dataparallel-gets-stuck/125427) . It can be excuted by `python -m torch.distributed.launch --nproc_per_node=2 script.py ` [DDP test code](https://gist.github.com/hhb072/02373276b3e78eaa854b035e5ee8de02) ### Expected behaviors Train without errors. ### Versions Collecting environment information... PyTorch version: 1.10.2 Is debug build: False CUDA used to build PyTorch: 11.3 ROCM used to build PyTorch: N/A OS: Ubuntu 18.04.4 LTS (x86_64) GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.27 Python version: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-4.15.0-167-generic-x86_64-with-glibc2.27 Is CUDA available: True CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA A40 GPU 1: NVIDIA A40 GPU 2: NVIDIA A40 GPU 3: NVIDIA A40 GPU 4: NVIDIA A40 GPU 5: NVIDIA A40 GPU 6: NVIDIA A40 GPU 7: NVIDIA A40 Nvidia driver version: 470.82.01 cuDNN version: Probably one of the following: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.4 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.1 /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.1 HIP runtime version: N/A MIOpen runtime version: N/A Versions of relevant libraries: [pip3] numpy==1.21.2 [pip3] torch==1.10.2 [pip3] torchaudio==0.10.2 [pip3] torchvision==0.11.3 [conda] blas 1.0 mkl [conda] cudatoolkit 11.3.1 h2bc3f7f_2 [conda] ffmpeg 4.3 hf484d3e_0 pytorch [conda] mkl 2021.4.0 h06a4308_640 [conda] mkl-service 2.4.0 py39h7f8727e_0 [conda] mkl_fft 1.3.1 py39hd3c417c_0 [conda] mkl_random 1.2.2 py39h51133e4_0 [conda] numpy 1.21.2 py39h20f2e39_0 [conda] numpy-base 1.21.2 py39h79a1101_0 [conda] pytorch 1.10.2 py3.9_cuda11.3_cudnn8.2.0_0 pytorch [conda] pytorch-mutex 1.0 cuda pytorch [conda] torchaudio 0.10.2 py39_cu113 pytorch [conda] torchvision 0.11.3 py39_cu113 pytorch

and this:

github.com/pytorch/pytorch

using multi thread lead to gpu stuck with GPU-util 100%

opened 11:18AM - 26 Jun 19 UTC

junedgar

high priority needs reproduction module: cuda triaged quansight-nack

I tried to inference using multi thread, but gpu stuck with GPU-Util 100%. The… pytorch was builded from source in branch v1.1.0. But if i install the torch from pip or conda, the some code can work smoothly. Is there ignore some import process in building? I followed the guidance： ``` conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing conda install -c pytorch magma-cuda100 git clone --recursive --single-branch --branch v1.1.0 https://github.com/pytorch/pytorch.git cd pytorch git submodule sync git submodule update --init --recursive export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} python setup.py install ``` cc @ezyang @gchanan @zou3519 @ngimel

They sound similar and point to a mismatch between python/cuda/pytorch and or a breaking change in socket libraries for inter-GPU communication that may be fixed by setting an ENV variable ?

Swamp7 · September 15, 2023, 11:37am

Setting NCCL_P2P_LEVEL to NVL made no change, still get watchdog: BUG: soft lockup CPU stuck error.

However, after replicating the issue several times uninstalling Nvidia Drivers and Cuda toolbox, then reinstalling the latest Full Cuda Toolbox and after a reboot I get a memory error. Replaced the DIMM with a new one and it’s been running smooth for the last day with out issues.

Strange because all the DIMMS passed a 24 hour memtest with zero errors just a week before…

Swamp7 · October 10, 2023, 1:14pm

Posting the solution for future users with similar problems,

After changing RAM still experienced problems, swapped CPU and Motherboard no change. A different server with Epyc 7742 and 6 x RTX4090s started crashing with the same problem, watchdog: BUG: soft lockup

AMD EPYC systems have a known serious compatibility issues/problems with the core nvidia NCCL library that distributed pytorch (and other frameworks)

Pythorch Thread:

Nvidia Info:
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs

Solved issue by disabling IOMMU and SVM Mode in the motherboard BIOS.

system · July 10, 2024, 7:14am

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.