Hello again folks, I am trying to figure out what laptop to buy for my university deep learning needs. I don’t plan on doing training on this laptop only inference. Training on laptops is dumb and modern AI models need 20gb+ vram i.e llms and stable-diffusion when paired with reasonable amount of data, so I will ask the uni for some kind of cloud account or some other resource for training if it comes down to it.
On the other hand inference is cheap and both Intel and AMD have shown that you can run llms and stable-diffusion on their “NPUs” slapped with system RAM as their main source of memory for pulling data from. So I would like to know what are my options here?
Ideally I want something under 1.5~2kg thin and light type of laptop but with Ryzen9 8945HS or Intel Ultra 7 or 9 since these CPUs meet the 40TOPs requirement that seems to be the next minimum amount that PCs need to have going forward. Also the higher the ram the better off I would be since this Intel demo you can see the task manager ram increases as he loads the model on to NPU which clearly has some way to talk to RAM directly. Which is why I am thinking of getting highest possible ram config of the laptop(32gb+). I wasn’t able to find any video of Ryzen/ Rocm doing the same with their NPUs which is comparable to the intel one I mentioned before.
What are you running? Helps to know.
Lots of hardware requires software that specifically supports the hardware down to the actual application like CUDA.
My advice dont use a laptop directly for ML use:
- https://colab.research.google.com/
- Rent a cloud GPU
- Connect to a machine with a powerful gpu i.e. 4090
I be running fedora on it, if something goes wrong drastically I might switch to Ubuntu based distro.
I did consider that but I want to keep my tech stuff as minimal as possible since I expect to move a lot not only daily but also in terms of housing so I won’t be looking into egpu 4090. As for the colab stuff I expect my uni to give training resources i.e whatever education cloud perks are available form gcp, azure or aws.
What I am more interested in is being able to do inference locally for my projects, assignments and stuff. That way I don’t burn through my cloud credits that I get from the education account nor do I have to get a thick laptop which needs huge amount of cooling. Which is why I am focusing on these NPU based laptops, they are available in lightweight form factor and don’t require that high amount of cooling.
I am currently getting my masters in data science and I am not sure this idea is the best. Most of your time is going to be spent building, training, and iterating on models. Once the model is trained you will be making some inferences but unless you are deploying it to a production server you would leave it on the hardware/system you trained it on because you built it around that software stack, ie CUDA/Rocm. Almost guaranteed you would need to modify the code/dependencies to deploy it to the NPU and I don’t see the benefit if you already have it running on a GPU or TPU. I could be wrong here, but so far in my studies and experiences in DL this is what I have observed.
I would expect the NPU to lack software support and be underpowred.
That said the only viable option I can point too is an apple m2/3 series laptop since they have fast memory and lot of it i.e. 128GB.
You could look at this review or maybe the localllama subreddit for inspiration:
@Iron_Bound I agree. And that unified memory and the wide bus really help too and they have been doing the NPU thing longer than most so their software has matured a little. However, it still takes work to deploy.
The 40 TOPS thing is a Microsoft requirements for some kind of copilot key or AI pc certifications. Even a rtx 3050 has a couple hundred tops for reference. These npus are there to enable power efficient basic „ai“ tasks like background blur, speech to text and marketing. Not for serious work.
Not to mention probably lacking Linux support. On intel drivers might be better, but the software using npus likely won’t be Linux focused.
Get a nice laptop for your coursework without looking at the ai capabilities. If it turns out you need more a better laptop is likely not the answer. Rather a budget workstation or rather using university resources.
@thecoderx @Iron_Bound I’ll look into your consideration of using apple’s stuff but they lack maturity in terms of linux support imo especially on arm based hardware even if they have much more experience in making NPUs.
Another thing @thecoderx mentioned I need to retrain the model again and again after updating the code and given the apple’s distain for sticking close to what the industry is doing could prove challenging, during training and inference where instead of working on the code I am fighting apple’s restrictions.
That is what I had in mind, use uni resource for training which seems likely since dedicated CUDA labs might be a thing for computer graphics and simulation courses. That way I can keep my setup as light as possible, no need to carry a bulky gpu based laptop whose fans will be as loud as an airplane even before the training begins.
I would suggest a laptop with solid thunderbolt/usb4 like Lenovo
Laptop cooling pad
Egpu dock
GPU to put in dock
Gaming laptops and workstation laptops are expensive and aren’t particularly fast
Better to use that budget on an egpu with less power and thermal constraints
and you can UPGRADE the GPU later
Found details on intels npu thing
meteor-lakes-npu/
I apologize if my response about the Apple NPU was misleading, I think the point is that you could run and even train the models on the iGPUs which have access to potentially 128GB of memory. It may train slower than a discrete GPU but the amount of fast ram available to the iGPU is potentially much higher than you could get on a discrete GPU.
You typically don’t have to retrain the model once it is trained to the accuracy that you want. What I am saying is that changing hardware will change the libraries that the model depends on and therefore so will some of the code and the environment you have it running in.
@Iron_Bound great article! I was surprised to see that they are optimized for int8.
Meteor Lake’s NPU is a fascinating accelerator. But its has narrow use cases and benefits. If I used AI day to day, I would run off-the-shelf models on the iGPU and enjoy better performance while spending less time getting the damn thing running.
