What is your setup for coding with/alongside AI?

Dev Zone Thoughts on AI
1

I’ve been wanting to do more coding lately and I’ve heard some good things about how useful AI can be for projects. So I’m curious how you guys are setting things up and integrating them.

I want to run everything locally (7800XT 16GB). I’m interested in working on existing Github projects so I’d like the AI to be able to read through the code and give me information on how it’s structured (it can be hard for me to get into someones already existing codebase). I’d be interested in using it for things like code completion or providing a starting point for code but I’m not planning on trying to have it do the entire thing for me.

I’ve mainly used VS Code (though I’m thinking of moving to VSCodium).

I’m running Bazzite but I might move to Bazzite-DX soon.

Also if you guys have any suggestions for good baseline habits and structures that would be useful. I’ve been thinking something like setting up some sort of containerization for projects would be ideal (clean build environment, isolated testing environment, all the build requirements are isolated to prevent a hijacked package causing problems on the system) but haven’t figured out the proper way to do that yet.

2

Don’t want to be “that guy” but use the search. Look for things like “llama.cpp 16GB” or hell just “16GB”. Search for anything with VRAM, MoE, Qwen3.6, Gemma4, etc.

Or browse the specific category as there are several with the same subject:

I mention that because I’ve answered at least 5 other exact same “I have 12gb/16gb and want to host Ai locally" posts over the last month. They do exist!

With that said…

Things are getting way better. You can have a highly accurate local Ai setup, but right now you’ll need around 24GB or more. It hasn’t broken that barrier yet.

There are some high quality models you can squeeze into 16, like qwen3.6 35B A4B per Unsloth’s UD Q2 variant. But I’ve had some longtail issues with long toolcalls.

Otherwise, you’re going to have to sacrifice enormous performance with the technique called CPU Offloading.

Finally, if you are into DevOps or automating development, read up on Turnstone released here that uses Kappa as Kappa’s MoE MXFP4 fits within 12 GB and gives you the entire 131k context.

^- if any of these terms are foreign sounding, see earlier section about searching.

Ps, welcome to the wild west! It’s the golden age right now. Hehe

3 Likes
3

I have researched existing threads but the vast majority are deep in the weeds or just talking vaguely about concepts and neither of those really acts as a good guide/entrypoint.

I have yet to find a very good, locked in and detailed here is how to get starting practically using AI.

4

I feel you. That’s the wild west part. There’s no silver bullet due to all the bottlenecks and memory crunch.

It’s also much easier than you think. Generally it’s just a single app or two, plus some driver work. I like to do everything in containers personally, to keep my host machine clean.

Your end goal is to end up servering an OpenAI API endpoint, so your tools can easily connect and use the model(s) being hosted. It’s a standard that every tool supports now, with their own self-hosted self-contained webapp to serve the OpenAi API endpoint.

There’s many ways to explorer local Ai with the same hardware. So it’s a matter of starting with one, max out what you can, and maybe try another strategy if it’s not working for you. There’s no such one-size-fits-all to get started. It’s all, “it depends.”

For example, I’m a Security and Linux CLI person because I automate my setups to be reproducable and deployable on my server. Because of that, I really like Ramalama as a tool because:

  • It figured out my local hardware and setup, and selected the proper drivers automatically.
  • Outputs detailed logs and feedback to figure out what to change (use online Ai to debug errors and config options).
  • Uses containers to isolate the models with no Network access. You mentioned using containers earlier.

There was nothing else to install but Ramalama (though I already had podman).

However, if someone is exclusively Windows they may only be a point and click person, at which there are several pre-packaged monolithic GUI apps you can pull and tinker with.

  • Lemonade AI
  • Local Ai (a bit tough to get setup, but very complete)
  • LM Studio
  • AnythingLLM

Use online Ai to Pro/Con that list.

Stay away from Ollama. Long story to read up on later. :slight_smile:. And start browsing and familiarize yourself with HuggingFace:

5

Just remember: to jump to the end, 16GB is just a hair under the 24GB that most good models are working on.

iME, this means you can get really decent initial responses however your context and quality will suffer for longer local coding use - and you’ll be as searching ebay for a 24GB GPU, or 32GB. The next really big leap is to get 48GB to 64GB VRAM for really the best local Ai. That can be 2 or 3 cheap Tesla P40s though training will be on the slow side.

1 Like
6

Another thought.. That’s an RDNA3 GPU, so you’ll want to test both ROCm and Vulkan backends in Ramalama because each model is different and performs differently.

Maybe start budgeting for a 2nd 7800 XT for 32GB of VRAM in a few months. You don’t need power, it’s all VRAM these days so the cheapest way there wins. :slightly_smiling_face:

1 Like
7

If I have to buy another GPU to get reasonable coding assistance then I’m just going to give up on using AI for coding.

You can’t just drop that and walk away.

I’ve browsed it in the past and it feels like a mess.

I take it this means you recommend just downloading whatever random plugin, in whatever coding software I use, that claims to help with AI coding and just takes an OpenAI API address. Just vibe it out.

8

Not to discourage you, it all “depends” again.

For example, if you’re into DevOps and want to automate a lot of backend crap, run Turnstone with Kappa MXFP4 as it will fit nicely within 12GB, leaving you room to run another smaller model, and is rated insanely high on the RULER measurements. Now that I’ve got more experience with different quants of Kappa 20B, I cannot recommend this for 12GB. Need at least 16GB for the MXFP4, and you’ll get to keep ~210k context.

There hasn’t been an MTP nor MMPROJ for Kappa yet. So it’s falling behind on the latest models tech.

Read up in Turnstone:

And the Kappa model based on GPT OSS:

And there is your first real snag without reading up on GGUF packaged models: That MXFP4 release is not gguf, and even though there is a GGUF version, that MXFP4 seems to work better for me than the Q4 GGUF. The problem is I had to convert it locally usually one of llama-convert tools to get it in GGUF format. It’s fast and simple, just too a minute to figure out.

However, I don’t know how well it works as doubling as a coding agent. Haven’t made it that far yet.

9

Ollama - it’s easy-mode AI, but it’s also very slow compared with just about everything else that’s out there; these days, you’re giving up 30-50% performance relative to llama.cpp without MTP, and 50-100% compared to llama.cpp with it enabled (although it’s only currently available for Qwen 3.5+ models).

Huggingface - yes, there’s a lot of noise on there, but think about it as “Github for LLMs”. If you restrict it to a few useful repos - Unsloth and bartowski are my go-tos - then it should help you focus.

No, it means read enough to the point where you think you can get started, pick a couple of models that’ll fit in your GPU (including context), and look for agentic IDE plugins that get good reviews. To get you started…I use Cline in VS Code as the agent, because it can talk to any OpenAI-compatible API (which is what llama.cpp provides). It can also talk to Ollama, if you really wanted to start with that as an option.

In terms of models, I’m all-in on Qwen myself. You might want to get started with Qwen 3.5 9B - the GGUF files for a reasonable quantisation of that (Q4_K_M is where most folk start) come in at about 6GB, which will give you plenty of room for context while working at reasonable quality. Look up KV cache (context) quantisation - both Ollama and llama.cpp allow you to do this, and setting it to q8_0 will halve the VRAM requirement for the context (which I’m assuming will be critical, since I’m also assuming that your GPU is also pulling actual graphics duty).

Your 16GB VRAM is a hell of a limitation, unfortunately. Replacing it with a used 3090 will get you a long way, because the difference in the models you can run in 24GB compared to 16GB is a massive step-change in capability. That doesn’t mean you can’t get going, it just means that you’ll likely outgrow it and change your mind on this sooner than you think.

1 Like
10

Lol… here ya go:

11

Turnstone seems like it’s way too much for what I’m asking for, as in for much larger or more complex setups.

I’ll take a look at the Kappa model when I get the other stuff setup

12

Thanks for providing more detail.

Ollama - That seems like a good reason to jump away from it from the git-go and the other post about it sounds like it’s a good idea from other standpoints as well.

Huggingface - That seems like a correct view. Unlike Github it only has a few categories so there is way too much in any section and way too many variation, tweaks, tunes, etc. to filter through. The recommendation for a few specifics is great.

I was being hyperbolic to emphasize the lack of any details. But really what I meant is this. Just kinda feeling it out, going with what seems like it might be good and has some good reviews.

Getting some suggestions from your setup is very helpful. I’ll take a look at Cline specifically as an option.

I’ll also look to see how well Qwen would work for coding. I remember testing it awhile back and I believe it had some of the best results for what I was working on at the time.

I’m stuck with 16GB. I’m not getting another GPU anytime soon.

13

Aye, it’s certainly worked out as the best option for me. The really good models are 3.6 27B and 35B, though, which is where…

…yeah, I said that too, once :wink:

1 Like
14

I guess I’m not being clear enough. I cannot afford to buy a GPU right now.

If someone wants to mail me some cards for free, then I’ll swap.

15

Here ya go (u haven’t mentioned Linux or Windows, so I’ll assume Linux as you want that anyways for any real hosting).

  1. Make sure you have docker or podman installed and functional.
  2. Install the single binary Ramalama tool via their instructions (GitHub). It will use docker/podman to launch llama-serve instances.

You’re ready to get started! Your first goal is to setup an OpenAI API endpoint for your plugins to use (you mentioned coding, so here ya go). Use ramalama serve to setup that endpoint.

If you are wanting to kick the tires with one of the absolute best models that will fit entirely into your 16GB VRAM, AND no CPU offloading, start by pulling the 12GB down of the compressed Qwen3.6 35B A3B - UD Q2 K XL variant. This particular variant received a lot of love: they researched and resolved a large number of issues that normally come from Q2 models and rebuilt and re-released it. Still not as good as the well tested UD Q4 people use with 24GB of VRAM, but I think it gets me as close as I could on that laptop.

You can read more here: unsloth/Qwen3.6-35B-A3B-GGUF · Hugging Face

# this could take a while ...
# 
ramalama pull hf://hf.co/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf

Once that completes, start it up:

# Start llama-serve.  Ramalama tries to help by handing runtime parameters.  
# hence the funky split between --runtime-args (straight to llama-server), 
# and ramalama parameters.  More to learn.
# 
# ctrl-c to exit cleanly.
# 
# You can paste the entire output here, as it will be very informative of 
# your hardware and configuration.  (There's nothing identifying in it that
# I've notice but I would review anyways.)
#
# 
ramalama serve hf://unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf \
--runtime-args="--fit-target 256 --flash-attn on -ctv q8_0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --reasoning off" \
--temp 1.0 \
--ngl 99 \
--name localai \
--host 0.0.0.0 --port 8080

NOTE: I’ve disabled “thinking/reasoning mode” above, to give you a fast experience while we smoke test. I also left out the mmproj vision image-to-text stuff as well as custom chat template stuff (more advanced topics).

SECURITY: Don’t blindly run things a stranger tells you. lol I would take the time to research Ramalama and how it runs llama-server within a container, what each llama-server parameter is there and how it affects the model loading.

You’re done. You can test it out with a quick chat session here: http://0.0.0.0:8080

(There are much better GUIs - remember you are just kicking the tires.). Check out the tokens/sec numbers while you are playing around here.

Those settings worked pretty well on my 16GB VRAM laptop so it will load and auto-adjust the context from 262k down to around 60-160k depending on your existing overhead. You’ll need to read up on each one as there’s a lot going on there. If you have 4K widescreen or multiple monitors, considering going to 1080p and only 1 monitor to gain back like 500MB or even 1GB of VRAM.

This is also your OpenAI API endpoint to use in your coding tools.

As mentioned above, use Cline or Continue for VSCode. Once installed, point your plugin to the 0.0.0.0:8080 endpoint above. Sometimes they want a /v1 at the end of the url, like http://0.0.0.0:8080/v1.

Excluding the download of 12GB, that should get you going in about 5-15 minutes.

Use another terminal and run nvtop to monitor your GPU’s VRAM resources and speeds. Personally I like KDE’s System Monitor as I monitor CPU usage as well as GPU usage for MoE models.

Lastly, I highly HIGHLY recommend reading that entire output, line by line. It’s extremely informative and you’ll need to understand a lot of it when expanding to other models. Use the actuall instance your hosting to ask it about ramalama’s output, line by line.

1 Like
16

@Calvin updated my answer with what runs on my laptop. Ramalama detects I have a CUDA device and switches to CUDA.

However, Ramalama on yours should select the Vulkan backend (it’ll be in the log output) as that’s usually installed by default in the latest distros. It may use ROCm, if you’ve got that configured. You can switch these with another parameter and try both by checking the output of ramalama --help .

EDIT: Why Ramalama? It handles detecting your hardware, pulling the precise docker image that coorisponds to your hardware and drivers, and adjusts parameters to match all under the hood. it really simplifies things, AND you have the power to run any model with any parameter.

17

If you’re just looking for a model that can “do what you tell it”, not something to think for you. I suggest checking out liquid.ai’s LFM-2.5 models. I run one on my laptop that has an RTX 3050 with 4GB of VRAM.
You can ask it questions, but it’s more useful for things like, “Organize my Music folder” and things like that.

Also check out Hermes Agent. My setup is to run Hermes Agent locally on my laptop and either point it at a local LFM model, or use my workstation as the LLM server.

@eduncan911 ‘s suggestion of Qwen3.6 is spot on if you can get that running on your hardware! Qwen 3.6 is best in class for local AI, at least that I’ve seen.

2 Likes
18

For me as a beginner with an older gaming Laptop ASUS ROG STRIX with a 3070 GPU and 8 GB Vram / 32 GB RAM, this video was very helpful:

Even better was, that somebody down in the comments showed the parameters to use for it to work in LM Studio, which internally uses llama.cpp and is way friendlier to use than the raw thing.
So this runs in - for me - acceptable speed, since I learn the most from the reasoning part actually, and it’s as fast as I can read here.

Qwen 3.6 so far is my favourite from the LLMs I used, since it’s less of an overconfident asshole/idiot than gemma and the nvida thing.

Probably wouldn’t be fast enough for a production environment, but works great for teaching me to use CachyOS :slight_smile:

Cheers,

Tom

1 Like
19

Currently using LM Studio to host local models like Qwen 3.5 35B coder and using VSCode and Continue.dev extension for chat and AutoComplete.

For AutoComplete I use Qwen 2.5 Coder models. It seems there isn’t much local FIM models that could do the job. All these running with RX 9070XT and 64GB ram.

Qwen 2.5 coder, either 3B or 7B version can easily just be on CPU with 0 GPU layers in the settings and token limit 4096.

Qwen 35B model is MoE -model so it’s fast even with using only 13GB of VRAM. I’ve given it 65536 token limit so I can even chat with it in VSCode and the model can easily read the code while suggesting edits.

1 Like
20

I’ve actually switched from having two models loaded to just Qwen 3.6 35B now - I’ve also ditched VS Code and I’m now running Zed. Interestingly, I find the workflow much simpler when using the inline completions in subtle mode, so it only gives me suggestions when I request them - I originally set it that way because Zed’s autocomplete spams the hell out of the server (it doesn’t seem to have any kind of debounce), but removing the automatic suggestions has taken away a lot of noise I didn’t realise I was dealing with.

So, now that the noise is gone and it’s no longer spamming the AI server, using Qwen 3.6 for the suggestions is working brilliantly (it’s actually faster and more accurate than Zed’s purpose-built Zeta 2.1 model on the same hardware). Only annoyance is that it occasionally tries to suggest methods that already exist in the file. At some point, I need to see if I can fix that by changing the prompt that’s sent.