AI fashions have turn into more and more democratized, and the proliferation and adoption of open weight fashions has contributed considerably to this actuality. Open-weight fashions present researchers, builders, and AI fans with a strong basis for limitless use instances and functions. As of August 2025, main U.S., Chinese language, and European fashions have round 400M complete downloads on HuggingFace. With an abundance of selection within the open weight mannequin ecosystem and the power to fine-tune open fashions for particular functions, it’s extra necessary than ever to know what precisely you’re getting with an open-weight mannequin—together with its safety posture.
Cisco AI Protection safety researchers performed a comparative AI safety evaluation of eight open-weight giant language fashions (LLMs), revealing profound susceptibility to adversarial manipulation, significantly in multi-turn situations the place success charges had been noticed to be 2x to 10x greater than single-turn assaults. Utilizing Cisco’s AI Validation platform, which performs automated algorithmic vulnerability testing, we evaluated fashions from Alibaba (Qwen3-32B), DeepSeek (v3.1), Google (Gemma 3-1B-IT), Meta (Llama 3.3-70B-Instruct), Microsoft (Phi-4), Mistral (Giant-2 also called Giant-Instruct-2047), OpenAI (GPT-OSS-20b), and Zhipu AI (GLM 4.5-On).
Under, we’ll present an outline of our mannequin safety evaluation, evaluation findings, and share the total report which supplies an entire breakdown of our evaluation.
Evaluating Open-Supply Mannequin Safety
For this report, we used AI Validation, which is a part of our full AI Protection resolution that performs automated, algorithmic assessments of a mannequin’s security and safety vulnerabilities. This report highlights particular failures akin to susceptibility to jailbreaks. tracked by MITRE ATLAS and OWASP as Multi.t0054 and LLM01:2025 respectively. The danger evaluation was carried out as a black field engagement the place the main points of the appliance structure, design, and present guardrails, if any, weren’t disclosed previous to testing.
Throughout all fashions, multi-turn jailbreak assaults, the place we leveraged quite a few strategies to steer a mannequin to output disallowed content material, proved extremely efficient, with assault success charges reaching 92.78 %. The sharp rise between single-turn and multi-turn vulnerability underscores the dearth of mechanisms inside fashions to keep up and implement security and safety guardrails throughout longer dialogues.
These findings verify that multi-turn assaults stay a dominant and unsolved sample in AI safety. This might translate into real-world threats, together with dangers of delicate information exfiltration, content material manipulation resulting in compromise of integrity of information and data, moral breaches by means of biased outputs, and even operational disruptions in built-in methods like chatbots or decision-support instruments. For example, in enterprise settings, such vulnerabilities may allow unauthorized entry to proprietary data, whereas in public-facing functions, they could facilitate the unfold of dangerous content material at scale.
We infer, from our assessments and evaluation of AI labs technical reviews, that alignment methods and mannequin provenance might issue into fashions’ resilience in opposition to jailbreaks. For instance, fashions that concentrate on capabilities (e.g., Llama) did exhibit the very best multi-turn gaps, with Meta explaining that builders are “within the driver seat to tailor security for his or her use case” in post-training. Fashions that targeted closely on alignment (e.g., Google Gemma-3-1B-IT) did exhibit a extra balanced profile between single- and multi-turn methods deployed in opposition to it, indicating a concentrate on “rigorous security protocols” and “low danger stage” for misuse.
Open-weight fashions, akin to those we examined, present a strong basis that, when mixed with malicious fine-tuning methods, might doubtlessly introduce harmful AI functions that bypass customary security and safety measures. We don’t discourage the continued funding and improvement into open-source and open-weight fashions. Reasonably, we concurrently encourage AI labs that launch open-weight fashions to take measures to stop customers from fine-tuning the safety away, whereas additionally encouraging organizations to know what AI labs prioritize of their mannequin improvement (akin to robust security baselines versus capability-first baselines) earlier than they select a mannequin for fine-tuning and deployment.
To counter the danger of adopting or deploying unsafe or insecure fashions, organizations should contemplate adopting superior AI safety options. This consists of adversarial coaching to bolster mannequin robustness, specialised defenses in opposition to multi-turn exploits (e.g., context-aware guardrails), real-time monitoring for anomalous interactions, and common red-teaming workouts. By prioritizing these measures, stakeholders can remodel open-weight fashions from liability-prone property into safe, dependable elements for manufacturing environments, fostering innovation with out compromising safety or security.
Comparative vulnerability evaluation exhibiting assault success charges throughout examined fashions for each single-turn and multi-turn situations.
Findings
As we analyzed the information that emerged from our analysis of those open-source fashions, we appeared for key risk patterns, mannequin behaviors, and implications for real-world deployments. Key findings included:
- Multi-turn Assaults Stay the Main Failure Mode: All fashions demonstrated excessive susceptibility to multi-turn assaults, with success charges starting from 25.86% (Google Gemma-3-1B-IT) to 92.78% (Mistral Giant-2), representing as much as a 10x improve over single-turn baselines. See Desk 1 beneath:
- Alignment Strategy Drives Safety Gaps: Safety gaps had been predominantly constructive, indicating heightened multi-turn dangers (e.g., +73.48% for Alibaba Qwen3-32B and +70% for Mistral Giant-2 and Meta Llama 3.3-70B-Instruct). Fashions that exhibited smaller gaps might exhibit each weaker single-turn protection however stronger multi-turn protection. We infer that the safety gaps stem from alignment strategy to open-weight fashions: labs akin to Meta and Alibaba targeted on capabilities and functions deferred to builders so as to add extra security and safety insurance policies, whereas lab with a stronger safety and security posture akin to Google and OpenAI exhibited extra conservative gaps between single- and multi-turn methods. Regardless, given the variation of single- and multi-turn assault approach success charges throughout fashions, end-users ought to contemplate dangers holistically throughout assault methods.
- Risk Class Patterns and Sub-threat Focus: Excessive-risk risk lessons akin to manipulation, misinformation, and malicious code era, exhibited persistently elevated success charges, with model-specific weaknesses; multi-turn assaults reveal class variations and clear vulnerability profiles. See Desk 2 beneath for a way totally different fashions carried out in opposition to numerous multi-turn methods. The highest 15 sub-threats demonstrated extraordinarily excessive success charges and are price prioritization for defensive mitigation.
- Assault Strategies and Methods: Sure methods and multi-turn methods achieved excessive success and every mannequin’s resistance assorted; the choice of totally different assault methods and techniques have the potential to critically affect outcomes.
- General Implications: The two-10x superiority of multi-turn assaults in opposition to the mannequin’s guardrails calls for rapid safety enhancements to mitigate manufacturing dangers.
The outcomes in opposition to GPT-OSS-20b, for instance, aligned carefully with OpenAI’s personal evaluations: the general assault success charges for the mannequin had been comparatively low, however the charges had been roughly in line with the “Jailbreak analysis” part of the GPT-OSS mannequin card paper the place refusals ranged from 0.960 and 0.982 for GPT-OSS-20b. This outcome underscores the continued susceptibility of frontier fashions to adversarial assaults.
An AI lab’s aim in growing a selected mannequin may affect evaluation outcomes. For instance, Qwen’s instruction tuning tends to prioritize helpfulness and breadth, which attackers can exploit by reframing their prompts as “for analysis,” “fictional situations”, therefore, the next multi-turn assault success charge. Meta, alternatively, tends to ship open weights with the expectation the builders add their very own moderation and security layers. Whereas baseline alignment is sweet (indicated by a modest single-turn charge), with none extra security and safety guardrails (e.g., retaining security insurance policies throughout conversations or periods or tool-based moderation akin to filtering, refusal fashions), multi-turn jailbreaks can even escalate rapidly. Open-weight centric labs akin to Mistral and Meta usually ship capability-first bases with lighter built-in security options. These are interesting for analysis and customization, however they push defenses onto the deployer. Finish-users who’re searching for open-weights fashions to deploy ought to contemplate what features of a mannequin they prioritize (security and safety alignment versus high-capability open weights with fewer safeguards).
Builders can even fine-tune open-weight fashions to be extra strong to jailbreaks and different adversarial assaults, although we’re additionally conscious that nefarious actors can conversely fine-tune the open-weight fashions for malicious functions. Some mannequin builders, akin to Google, OpenAI, Meta, Microsofthave famous of their technical reviews and mannequin playing cards that they took steps to cut back the probability of malicious fine-tuning, whereas others, akin to Alibaba, DeepSeekand Mistraldidn’t acknowledge security or safety of their technical reviews. Zhipu evaluated GLM-4.5 in opposition to security benchmarks and famous robust efficiency throughout some classes, whereas recognizing “room for enchancment” in others. Because of inconsistent security and safety requirements throughout the open-weight mannequin panorama, there are attendant safety, operational, technical, and moral dangers that stakeholders (from end-users to builders to organizations and enterprises that undertake these use) should contemplate when both adopting or utilizing these open-weight fashions. An emphasis on security and safety, from improvement to analysis to launch, ought to stay a prime precedence amongst AI builders and AI practitioners.
To see our testing methodology, findings, and the whole safety evaluation of those open-source fashions, learn our report right here.
