Training on 7900 xtx

[Iron_Bound]

I’ll try to record my notes here for future people, as I just purchased a 7900 for gaming and ML experiments.

After try the Shakespeare example, I Wanted to train a qLoRA, this is a super efficient 4bit adapter that goes with an existing model.

The process only trains 1/10000 of the model weights so you actually do some training at home… without a computer designed to fight god!

To run the training I had to fix an issue with the bitsandbytes python library, thankfully there was a fork with a solution for Rocm, and with that solved managed to start the train with llama7b, let see how it goes in 8hours!

[Iron_Bound]

Paused the training and didn’t restart from the checkpoint so starting over today.

I’ll look into batch size and config, before I try this multi decoder project GitHub - FasterDecoding/Medusa: Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

[LA-MJ]

What is your experience so far?

[Iron_Bound]

It’s been solid using the rocm-pytorch container.
And apart from an issue the bitsAndBytes library it’s been going well.

Currently training a model to play Pokemon Red!

[Brokefoot]

I am looking at 7900 xt for nearly the same efforts. Would you say the price is worth it. I’m in Canada and it hard to find a 7900 xt for under $1000.
Amd.ca has the 7900 xt for $1172.

[Iron_Bound]

My recommendation would be to buy the card for gaming first and secondly use it for AI.

I didn’t have the option of second hand 3090’s so take that into account before dropping 1k on a card.

[Iron_Bound]

Looking at my wandb review and I’ve trained 219 hours now.
Since it’s been hard to find details around 7900 training, I’m sharing the training metric’s below.
I hope its helpful in future!

iron-bound