AI news · October 11, 2026
A terrorism-safety benchmark finds open model safeguards can be stripped
Tech Against Terrorism's test reported three in five models failing a terrorism-safety benchmark and Llama 3.1 8B dropping from 97 to about 3 after abliteration; treat exact model files as a supply-chain input.
- models tested
- >130
- reported benchmark failures
- 3 in 5
- reported Llama 3.1 8B score change
- 97 to about 3
CBS News reported on October 9 that Tech Against Terrorism tested more than 130 models with 627 prompts about terrorism-related safety. The group said three in five models failed its test, and that open-weight variants modified through a process called abliteration failed every test. Meta's Llama 3.1 8B reportedly fell from a score of 97 to about 3 after the modification.
The benchmark measures refusal behavior, not whether a model can carry out a real attack. The full prompts and results were not available for independent review in the report, so the exact scores should be treated as reported findings. The practical point is still clear: a public model can be changed after release, and a model name alone does not identify the artifact a product is running.
Open-model builders should evaluate the exact file, quant, adapter, and serving path they ship. Run safe refusal and prompt-reframing tests, pin hashes, keep a fallback, and retain the result with the release. Do not reproduce harmful instructions to prove the point; test the control behavior with a reviewed internal set.
What you can do with it
Before shipping an open model, pin the file hash, run refusal and prompt-reframing tests on the exact artifact, and keep a safe fallback. Store the eval result with the release instead of assuming the base model's card still applies.
Our take
This is a strong reason to evaluate model artifacts, not only model families. The reported test has limits, but a modified open-weight file can invalidate assumptions made from the original safety result.
Source: CBS News — Made With Models writes the brief; the reporting is theirs.