LiveOverflow has a good description of the steps the AI used.
1 Like
More information:
Also claude is doing similar levels of abuse in testing Fable
Yes, there’s obviously a degree of marketing spin on all this, but the rate of progress we are seeing here, I think writing it off as “that’s just marketing” is a fairly cavalier attitude….
Also - these are stories about the big labs. Open weight models are close (maybe mere months behind) and other nation states have access to these, and presumably in-house developed models superior to what’s available to the public.
Thats some pretty big indictment of jfrog, but then again i dont think it was particularly hardened for use in a hostile environment. What i picture of a hardened local package proxy is something like linux distro package mirror, a http server with files, and a cronjob running rsync from main repos.
I think ALL software in common use is not much different.
The number of vulnerabilities being discovered in recent months (including Windows, Linux, macOS, iOS, Android, Cisco, Palo Alto, etc.) is insane, and its not because they didn’t exist previously - its because finding them is mostly a case of security researcher effort - and these LLMs don’t sleep, get bored, have day jobs, etc.
Taking a shit on jfrog for this is missing the point - the entire industry has a heap of work to do.
Calling it shitting on jfrog is missing the point, im saying their sandbox wasnt exactly airtight when they use large applications that expose apis and stuff into the supposedly secure sandbox.
As i said, jfrog was not really designed for hostile environments, its more of an internal proxy for corporate use.
Im expecting most stuff for corpo use is a pos coded by blind monkeys so its not surprising it gets raped immediately when put in a box with an AI or even a human hacker
2 Likes
I think you can dump “for corporate use” from that last statement.
The world we live in today, unfortunately all software is pretty garbage.
The corporate stuff gets targeted (because corps use it and they can be extorted for more money) but just look at the number of vulnerabilities found in Linux in recent months.
I don’t say that to dump on linux, it’s just still written by humans, and maybe somewhat less garbage, but the past 50+ years have demonstrated that humans suck at writing software - for various reasons including but not limited to
- financial constraints / shipping deadlines
- Even open source impacted by this as typically someone has a problem to solve and need to solve it ASAP at day job before moving onto the next problem
- boredom
- humans lose interest once the problem has been “solved” - there’s no shortage of new software to write, new features to add or performance hacks to make
- limited context
- brains, just like AI can only hold so much context simultaneously. modern software is incredibly complex and I would suggest generally grows beyond the typical programmer’s ability to fully understand. Modularisation can help with this but stuff always tends to grow over time anyway.
Whether AI is going to be significantly better at this is to be confirmed (personally I think it will help massively), but it is certainly faster at just grinding through potential security issues to exploit.
The next few years are going to be interesting. I’d say we’re in for a few years (at least) of security pain, until we run out of vulnerabilities to discover (and people running the stuff get around to patching), hopefully in the interim development tools evolve to help programmers stop writing bad code in the first place - I’ve already seen IDE plugins that claim to help with this.
Yeah but thats not news. There are secure practices on sysadmin and system architecture because we know that just exposing anything not strictly necessary on the web means your software is getting bent over a table, even before AI. Sandboxing is the same but in reverse.
I still think the point worth underlining here is that we have proof a company allegedly testing an highly capable tool that “can hack the cia/fbi” is building a remarkably less secure sandbox than one of the randos that run ai models locally on the ai section of this forum. As others were guessing before the truth came out.
For example why the fuck was the jfrog instance able to access the web without any restriction? Locking down access to only the sites it actually needs to access by a network firewall is not exactly a revolutionary security practice, corpo networks often lock down the network pretty tight
If I had a frontier lab, those tests would be done air-gapped. It’s marketing. It may be a decent PoC, but the problem is, they locked up the models, so no one can prove it exactly how they can.
I’d rather see the headline
“Model closed 16 state actor backdoors” or “Latest LLM solved the cure for cancer”
this whole:
“AI made billionaires another billion” & “Be afraid, the models are coming to get your softwares” Is literally PR.
1 Like
Interesting that you bring it up, I’d add that software engineering is a fairly young engineering principle. What’s 50 years compared to construction or mechanics.
In this regard, the prediction model using current code bases as input might not be as effective either (given we’re still figuring things out ourselves).
Meanwhile at Anthropic:
“But, look! We hacked people too! Pay attention to MEEEE!!!11”
7 Likes
As usual, government agencies not having a single clue while also not giving a single shit about potential repercussions of their actions.
Anthropic noted that AISI tested Mythos 5 without its standard cyber safeguards enabled and that this is not the same configuration available to customers. AISI says its evaluation design and configurations may have contributed to the behavior, but did not anticipate how the agent would show “signs of novel, potentially deceptive behaviours”.
Oh yeah lets turn off any safeguards and run this ai hacking with full internet access, its totally an ok thing to do.
This should be treated as cyberwarfare and everybody involved should be yoinked by delta force
2 Likes
The D team are asking claude how to start a war
In other news, “Bleeding edge model does expected crazy shit after turning off the governer.”
No they are asking help with paperwork. WAR is week activity report.
The more warsy stuff is done with Palantir, and its a lot more involved than a chatbot.
And their social media people are asleep at the wheel as usual, didnt catch that could be misread badly
It’s the Maven Smart System, designed by Palantir, that decides who/what is getting bombed.
1 Like
Its only going to get worse.
It’s a race for them to figure out the capabilities of these models and to do so they are turning off the guardrails.
I’d say based on the past few weeks the capabilities are “sufficient” and beyond the competence of the labs testing them.
I do honestly wonder how long it is going to be before one of the frontier labs hacks the wrong corp and gets their ass sued.
The fact that these models are essentially going off on a hacking spree as a side-quest to get a good benchmark score is pretty hilarious in a black humour kind of way…
No they do know what it can do, thats why their “oopsies” are probably coordinated marketing stunts and limits to avoid pissing off other corpos are in place.
This is a government org from uk
The UK AI Security Institute, commonly known as AISI, is a government research organization that evaluates the capabilities and risks of advanced AI models.
Not an issue for a government entity. They can disable safeties and just send it, and all corpos will just kiss their butt to stay on the good graces of uk regulators.
That absolutely is an issue for government entities here in the UK, given that we have a Supreme Court that’s apolitical and doesn’t have a problem giving the government a bloody nose when necessary. In fact, given that this month’s PM is a self-confessed anti-tech luddite, any department found to be acting in such an unsafe manner would probably find itself “restructured” in pretty short order.
Meta has joined the party.
They are deciding to frame this (quite honestly, which is very suspicious) as their contractor screwing up the testing sandbox and not as “my AI is scawwyy” like the others.
So this is a bit different, and coming from Meta I’m always thinking the worst (they are behind the whole age verification nonsense btw). It’s entirely possible this is part of a plan to advocate for more regulation in the name of “safety”.
Meta said it was investigating an incident in which a misconfiguration by Irregular, an independent company that conducts cybersecurity evaluations for Meta, inadvertently gave one of its models internet access during a testing.
A spokesperson for Irregular told Reuters the incident was the “exact same evaluation-environment issue that was already disclosed by Anthropic last week” and did not involve a “sandbox escape or a sophisticated cyber action”.
“There are no current open issues. Irregular is developing a white paper to share best practices for containment and securely running cyber evaluations,” Irregular said.