I have not yet started thinking about self-hosting LLMs but have seen a lot of community action about open LLMs and then just watched Chris Chappel (sp?)’s video about how they are susceptible to spying for the CCP. Thoughts?
China’s latest AI model looks set to be a new world leader. But it may have a dark secret.
When running a Chinese model on my own hardware, I’m not concerned at all. Unless there’s a malicious hack applied to llama.cpp, vLLM, or another model runner, there’s nothing that leaves your network except the occasional MCP web search, or fetch.
As for factual content, it’s not a good idea to blindly trust historical facts from any model.
When running on their hardware/API, it’s a fair concern that they get to see your prompts, which might include proprietary, private, or other sensitive info. Also, they are very likely using your prompts and responses to improve their models. Your potentially sensitive data might, however unlikely, end up trained into their next model, intact.
Where you need to be concerned is the agentic tooling. MCP servers and other agentic tools can open a connection to anywhere on the internet. A malicious MCP could potentially exfiltrate any data on your computer.
off topic?:
Model aggregators, like OpenRouter, are likely logging everything from every model that goes through their API. That info is probably the most valuable distillation dataset available anywhere. If Stripe buys OpenRouter for $10B, that will be a bargain. Your sensitive data can be captured here too.
Not off-topic. In fact one of the sources of rank international tension right now is an openrouter type provider, but in china, that was probably not doing anything nefarious but also gathering distillation data to use for next-gen models.
“Oh its a distillation attack!” not so fast there, chuckles.. what was the context in which said distillation occurred? Could, perhaps, the reason the distillation was not detected because it looked like ordinary end-user traffic? Perhaps because that’s what it actually was!? Just because those end-users captured & shared their inputs and outputs to improve the next gen model is not in itself nefarious.
To further what marked23 says – being able to run a model on your own hardware is like making your own food.
They’re giving you a recipe to try general tso’s chicken. You can cook it to your liking with ingredients you trust; you are not buying ready-made pre-cooked chicken. Unlike Anthropic/OpenAI where the only place you can get their chicken is from their kitchen.
But do they also offer a restaurant service as well for half the price of Anthropic and OpenAI?
I’m wondering if Chris’ (video in op) concerns are valid and under which circumstances. He is usually right in questioning the motives of the CCP (and any company in China by extension).
They do, sure, but anything open weights others can also offer a restaurant service, too.
Dario Amodi has literally testified before congress that Anthropic does the same sort of thing in terms of logging your sessions for analysis. I would add for analysis and looking for wrongthink. So in that sense Anthropic and OpenAI have exactly the same type of risk, depending on your point of view.
If we take at face value what Xi said in his speech, if they actually do it that way, it will actually benefit all of humanity. Sure, there is probably an aspect of that speech that was bait – to see if the US will punch themselves in the face, and oh cotton I think they really might – but if they follow through it’s going to be an economic boon the likes of which humanity hasn’t seen since probably cooking or possibly even fire.
Reading this thread reminded me of this blog post I read from Satya Nadella.
As a private person we are free to make our own choices when it comes to the information we share and how we conduct our business. As a corporation I think they need to make some choices going forward regarding their use of AI and the different vendors. I think this blog post is a good insight for people to think about these things.
Unless you are in China, you probably don’t need to worry about the CCP. I would be more worried about the spying OpenAI and Anthropic are doing.
If you commit a thoughtcrime, it’s your own government that is going to come for you, not the other guys.
Is the argument necessarily about spying. Would be more interesting and nefarious to deploy an AI that attempts to engender CCP values in its user. Those conversations are not exactly as one-way as many people think. In other words could Chinese models promote maligned cultural influences? A digital fifth column
^ cold warrior stuff to be sure
In some ways releasing a frontier model as they have is already propaganda of the deed wrt socialist ideas of the commons, plays to anticorporate sentiment in the west, blah blah
Hmm ..
I wonder what Anthropic would tell me about organised labour, or Grok on South African history.
The ability to rewrite history on the fly is what I worry about .. and nations doing those sorts of things, China isn’t at the top of my list.
If you are using an agent, you can ask the AI to search the internet for X number of sources, and give you their combined summary of the subject, with links to each source.
You don’t have to only rely on what the AI was trained to “think“.
That’s kinda my point. Cite which sources? What gets left out? What’s left in?
The possibilities to manipulate and misinform are endless.
The problem isn’t as much truth as trust. AI is as trustworthy as it’s developers.
Considering their and the soviets track record, promoting stuff that destabilizes the enemy country is often good enough.
Thats one specific situation where grok would not need to lie that much. But yes the concern is still valid in general.
I still remember the Google Gemini ai creating pictures of black and indian female popes, or showing a very diverse founding fathers picture, or also very diverse SS soldiers from ww2. They had to roll those weights back a mile and a half after the clowning they were getting.
You assume that you get the untampered processing of those results from the model but in this case I believe we are talking about an open weight model, not an open source model. So still closer to a black box than not… therefore I argue your assumption about the agent giving “uncolored” results is not necessarily a safe one.
A good test would be to ask it anything about Tiananmen square but it’s of course possible that they have one version for everything beyond the GFW and another for domestic consumption which agrees with CCP censorship.
Here is an example of this “one app, two systems” principle in action
Interesting link:
It is not difficult to run the same prompts on multiple models and cross-reference the results to measure how the model is biased.
This is done all the time to measure the difference between models, and there is nothing that supports the idea that the open weights models have some secret ccp backdoor.
Well…as far as I can tell, the CCP’s strategy is to facilitate the creation of open-weights models specifically to nobble the USA’s economic reliance on closed models, specifically from Anthropic, OpenAI and SpaceX-AI-whatever-they’re-called-this-week.
As a strategy, it’s actually pretty genius. It still gives you the ability to make money from providing the API versions, it garners a lot of karma from the enthusiast community who get to play with cool toys at astonishing levels of accuracy for low hardware requirements, and when they get good enough (if not now, pretty soon) it puts the ball in the US hyperscalers’ courts to justify their massively over-inflated valuations.
This could be China’s biggest move against the US in living history and could even lead to a general collapse of the US economy, and - amazingly - it didn’t need them to do any shady underhanded shit at all. At least, no more shady than their competitors; they’ve just played the same game and given it away for free because they don’t need to make a secret of it to win.
The point is…they can achieve all of this without pulling any shady tricks with the models themselves. It’s just a happy accident (for China) that all of this is happening exactly when the US government is making enemies of the entire world to the point where “US-made” is at least as much of a security risk as “Chinese-made”.
tl;dr - Their motives might be nefarious (from a US-centric perspective), but it doesn’t follow that the resulting tools are.
I think that was the strategy when they were far behind the US, just to try and stay relevant. This won’t be the case when they catch up or more of the cyber security implications of advanced models are realized.
I don’t think “when they catch up” will be the watershed moment; far more likely that the precipitating event will be the bubble bursting and the circular funding scam coming to an end. Basically, it’s not about the technology, it’s about the economic dominance.
I suspect it won’t be too long before we find out which of us is right 
However, when that happens, it’s reasonably likely that other large economic entities (eg the EU) will follow suit with open-weight models to counter them - who knows, maybe even Mistral might make something usable?
That’s a good point, that outcome seems likely to me as well.
If we see some really large hack facilitated by AI though that could accelerate the closing off of models… I feel like this situation could happen any day now if it hasn’t already.
Exactly, the majority of the “Chinese” model prop is all hype to see how they can puncture the bubble.
If substantially better models come out of places outside of the US, the likely economic situation will be the popularity calls for more systems which raise prices to par with US cloud models. Even if prolonged use by US citizens forces foreign models to get more chips. More chips means Nvidia keeps its bubble going.
Chinese chips offer that deflation angle, if they make headway towards alternatives, the hardware situation gets handed back to the supporting services, like DRAM/HBM which is still in the corruption bubble.
The open weights is a great strat to encourage users to pay attention to things the world is doing, when the US would much rather you only pay attention to the clown
They don’t even need to be substantially better - they just need to offer parity, because the benefits of open-weights models (cost and privacy) provide the incentive to move.
You’ll notice how Nvidia’s public-facing persona has become really bullish about open-weights models in the last few months after not even mentioning them outside of hobbyist curiosities since the whole debacle began. I suspect that they’re focused on wringing the last few drops out of the closed-model hyperscalers while preparing for the shift to a world where model hosting is far more distributed.