Is Deepseek a meme/hype, revolutionary, or just another AI

[greatnull]

If anyone needs a sensible chuckle ,here is something for you

OpenAI Furious DeepSeek Might Have Stolen All the Data OpenAI Stole From Us

[mutation666]

I almost want to bet they are running custom firmware to fix the nerf

[TryTwiceMedia]

you can’t do that…

[Calvin]

I’m curious about the training requirements and running requirements we are finding with these latest AI models.

Testing the 14b parameter distillation on my 7800XT since it fits entirely within VRAM and it’s very fast. Even with it not being the highest end or, now, latest card the main limiting factor feels like memory access. The speed plummets alongside actual GPU usage when a model falls out of VRAM and has to use RAM.

It makes me wonder if the main limit on AI performance on GPUs is entirely just VRAM and some of the cost for improving their chips performance overall is being paid for by inflating the chips they use for their high VRAM cards or by limiting VRAM on non specialized cards.

As in, could we be getting away with far far lower spec cards if they were making ways available for adding tons of VRAM. Instead only the very highest end chips seem to me (not someone super informed) to be getting the high VRAM capacities and incredible prices to match.

[lemma]

@Calvin

Instead only the very highest end chips seem to me (not someone super informed) to be getting the high VRAM capacities

Yup. Next VRAM step up from 4060 Ti 16 GB, 7600 XT, and A770 (the latter two are also 16 GB) in the currently available product lines is 7900 XT, 5090, or workstation cards. If used is an option then also 3090 and 4090 (or older workstation cards).

[Kougar]

@Calvin

It makes me wonder if the main limit on AI performance on GPUs is entirely just VRAM and some of the cost for improving their chips performance overall is being paid for by inflating the chips they use for their high VRAM cards or by limiting VRAM on non specialized cards.

As in, could we be getting away with far far lower spec cards if they were making ways available for adding tons of VRAM. Instead only the very highest end chips seem to me (not someone super informed) to be getting the high VRAM capacities and incredible prices to match.

Pretty much yes, memory capacity itself is the limiting factor for running models locally. It’s the entire reason why these models have to be “dumbed down” with quantization and smaller, lower parameter models like 14B instead of 32B or 70B.

If one really wants to run a large LLM they can do a hybrid approach using VRAM+RAM but it does slow down the processing speed. I’ve been playing around with the 32B Q8 build of DeepSeek R1 via LM Studio and it utilizes 23GB+20GB. I don’t dare try the 70B model though, need I think you need either a 5090+64GB or a minimum 128GB of system memory to run a useful 70B quantization level.

[igormp]

You can do 70B at Q4 with two 24GB GPUs, like the 3090 as an example (what I have).
Performance is acceptable.

[Iron_Bound]

Nope they did the hard work of optimizations, implementing compression on inter GPU communication and fp8 training.

Notes:

DeepSeek: Bridging Performance and Efficiency in Modern AI

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement...

deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf

[ubergarm]

Most of the “R1” models people are running at home are not R1 at all, but “Distill” versions of various other models like Qwen-2.5 which have been supervise fine tuned on some CoT output from the real R1.

I’m looking for any threads here on L1T of folks who have successfully run the actual DeepSeek R1 671B model (likely quantized to ~200GiB).

I’ve managed to get it going locally at just over 2 tok/sec with 8k context (but only 4 active experts instead of default of 8).

There is some chance that a super fast read IOPS block storage device (either network or NVMe array etc) plus a bunch of RAM for disk cache might be able to serve it up at a better price/performance ratio than expensive unobtainable VRAM.

Exciting times!

[quilt]

@ubergarm

There is some chance that a super fast read IOPS block storage device (either network or NVMe array etc) plus a bunch of RAM for disk cache might be able to serve it up at a better price/performance ratio than expensive unobtainable VRAM.

Doubtful at least at acceptable speed… During inference all the weights in the model need to be accessed once per token generated so there is a hard theoretical limit to performance based on bandwidth:

tok/s <= bandwidth/model_size

If you have a 100GB model you need at least 100GB/s to get 1 token/s.

[ubergarm]

So I’m serving 212GiB of weights off of 96GB RAM at just over 2 tok/sec.

I agree that tok/s is directly related to memory bandwidth in most cases.

My hunch is that MoEs don’t have all of the experts weights active at the same time. Assuming the generation isn’t switching between all of the experts for each token but votes to use the same experts for a series of tokens that a disk cache can keep those weights in the warm path while the unactive weights chill mmap()'d on a fast storage device.

It is a bit of speculation, as I don’t fully understand MoE and specifically the full sized R1 model architecture.

It is cool to watch btop while doing this because when the model begins to count words or speak in a different language, I can see it burp/slow and the disk cache dumps and refills with those expert’s weights.

I don’t see a thread about folks running the full R1 671B model on here yet (people with 256GB RAM can keep the whole quantized thing loaded). I’ll double check and maybe start one.

I have a post on r/LocalLLaMA that blew up yesterday about it. The folks on this forum are a bit more tech savvy and have interesting big RAM and fast IOPS hardware too so might be interested in benchmarking.

But yeah, “acceptable speed” for what kinds of applications or hobby uses you are totally right.

[OldSchool157]

This was a big week for companies to have their earnings calls, so I would say that this was moreso business as usual, but the market news folks always feel the need to have a headline/story to go along with the market movement.

In reality the answer is typically that the large investment firms make big moves, and they won’t ever tell you all of the strategies at play. Also, options/derivatives markets are MUCH bigger than the actual equity investments, so they can make large buy/sell movements on the actual stock in order to influence the underlying value of all of the options that they bought/sold.

[quilt]

@ubergarm

So I’m serving 212GiB of weights off of 96GB RAM at just over 2 tok/sec.

I agree that tok/s is directly related to memory bandwidth in most cases.

My hunch is that MoEs don’t have all of the experts weights active at the same time. Assuming the generation isn’t switching between all of the experts for each token but votes to use the same experts for a series of tokens that a disk cache can keep those weights in the warm path while the unactive weights chill mmap()'d on a fast storage device.

Hmm, right, that seems to help a lot. Apperently R1 activates 37B parameters at a time, but I have now idea how often it switches ?

For interactive use or coding assistance I’d probably want >100 tok/s, since the “thinking” can take quite a lot of tokens. I only experimented with the distilled ones locally but it uses easily 1000 tokens for relatively simple prompts.

[Darin755]

It is very fast from my testing. However, it also is pretty heavily censored.

[Calvin]

@quilt

For interactive use or coding assistance I’d probably want >100 tok/s, since the “thinking” can take quite a lot of tokens. I only experimented with the distilled ones locally but it uses easily 1000 tokens for relatively simple prompts.

Yeah, that’s my experience too. I’ve had some prompts where the thinking section is far far larger than the rest. I had one recently that was super interesting where the model entered a sort of thinking loop and kept repeating a task, noticing it’s errors, then trying again only to fail again. It was surreal, like seeing someone with some sort of mental condition where it just couldn’t get it right.

@Darin755

It is very fast from my testing. However, it also is pretty heavily censored.

I haven’t noticed censorship causing any major issues. I got it to talk a bunch about Tienanmen Square, Winnie the Pooh and the like. At least with the distilled models the censorship isn’t that strong, I don’t know if the purer/larger models are more censored.

[progressEdd]

I wanted to wait until I finished reading the technical report. Wrote my takeaways on my blog.

@progressEdd

My takeaway is that the team worked smarter than harder for example they did the following:

• developed a data pipeline that produced higher quality data in a higher volume
• applied the Mixture of Experts architecture with their own spin
• optimized the training process with their model architecture to fully utilize their available hardware and optimize cost, as well as, suggesting hardware level optimizations

Some additional commentary, I think OpenAI, Facebook, and Anthropic should really be afraid because the team was able to squeeze out the performance on lower grade hardware and even suggested hardware level optimizations.

Microsoft and other cloud providers will be laughing their asses off because they have another model that will be cheaper to run on their servers.

Nvidia, if they listen which they normally do, will adopt some hardware suggestions. See the following section:

3.5.2 Compute Hardware

3.5.2 Compute Hardware

Higher FP8 GEMM Accumulation Precision in Tensor Cores.

In the current Tensor Core implementation of the NVIDIA Hopper architecture, FP8 GEMM (General Matrix Multiply) employs fixed-point accumulation, aligning the mantissa products by right-shifting based on the maximum exponent before addition. Our experiments reveal that it only uses the highest 14 bits of each mantissa product after sign-fill right shifting, and truncates bits exceeding this range. However, for example, to achieve precise FP32 results from the accumulation of 32 FP8×FP8 multiplications, at least 34-bit precision is required. Thus, we recommend that future chip designs increase accumulation precision in Tensor Cores to support full-precision accumulation, or select an appropriate accumulation bit-width according to the accuracy requirements of training and inference algorithms. This approach ensures that errors remain within acceptable bounds while maintaining computational efficiency.

Support for Tile- and Block-Wise Quantization.

Current GPUs only support per-tensor quantization, lacking the native support for fine-grained quantization like our tile- and block-wise quantization. In the current implementation, when the NC interval is reached, the partial results will be copied from Tensor Cores to CUDA cores, multiplied by the scaling factors, and added to FP32 registers on CUDA cores. Although the dequantization overhead is significantly mitigated combined with our precise FP32 accumulation strategy, the frequent data movements between Tensor Cores and CUDA cores still limit the computational efficiency. Therefore, we recommend future chips to support fine-grained quantization by enabling Tensor Cores to receive scaling factors and implement MMA with group scaling. In this way, the whole partial sum accumulation and dequantization can be completed directly inside Tensor Cores until the final result is produced, avoiding frequent data movements.

Support for Online Quantization.

The current implementations struggle to effectively support online quantization, despite its effectiveness demonstrated in our research. In the existing process, we need to read 128 BF16 activation values (the output of the previous computation) from HBM (High Bandwidth Memory) for quantization, and the quantized FP8 values are then written back to HBM, only to be read again for MMA. To address this inefficiency, we recommend that future chips integrate FP8 cast and TMA (Tensor Memory Accelerator) access into a single fused operation, so quantization can be completed during the transfer of activations from global memory to shared memory, avoiding frequent memory reads and writes. We also recommend supporting a warp-level cast instruction for speedup, which further facilitates the better fusion of layer normalization and FP8 cast. Alternatively, a near-memory computing approach can be adopted, where compute logic is placed near the HBM. In this case, BF16 elements can be cast to FP8 directly as they are read from HBM into the GPU, reducing off-chip memory access by roughly 50%.

Support for Transposed GEMM Operations.

The current architecture makes it cumbersome to fuse matrix transposition with GEMM operations. In our workflow, activations during the forward pass are quantized into 1x128 FP8 tiles and stored. During the backward pass, the matrix needs to be read out, dequantized, transposed, re-quantized into 128x1 tiles, and stored in HBM. To reduce memory operations, we recommend future chips to enable direct transposed reads of matrices from shared memory before MMA operation, for those precisions required in both training and inference. Combined with the fusion of FP8 format conversion and TMA access, this enhancement will significantly streamline the quantization workflow.

progressEdd's llm and ai corner

I just finished reading the deepseek v3 paper. If the abstract isn’t detailed enough the conclusion is a much better summary without getting into the weeds of the tech. My takeaway is that the team worked smarter than harder for example they did the following: developed a data pipeline that produced higher quality data in a higher volume appl…

[vciso-tech]

ignoring the censorship issues & it’s origins, it’s very fast. I’d just be careful what data you’re sharing, but if you’re doing generic prompting these issues shouldn’t be a problem.

[Iron_Bound]

Following the reproduction efforts has been fun

Open-R1: Update #1

[Kougar]

At risk of going off topic the censorship criticism is a bit ironic given the way things in the US are heading. climate change is once again a verboten phrase being erased from all Federal websites, and datasets regarding it are currently being purged. But DeepSeek R1 actually acknowledges it is real and discusses it in fine detail with a 20 point list of examples. If “stargate” ever came around one would have to wonder how censored it would be itself given the current administration’s ongoing data purge on a dozen different topics.

[Darin755]

Climate change is a primarily political thing. Democrats don’t actually do much they just use it to lock in voters. It is sad since it is something that needs to be addressed at some point. I have a hard time believing that the oceans are going to boil or whatever extreme other things.