AI Digest
OCT / NOV 2026
~ ~ \\ // ~ ~
authored by
Chai K Toh
01 JUNE 2026
Copyright (C) SAFE AI Foundation
Are Open Source LLMs Safe To Use?
The proliferation of Open LLMs mean that we now have choices and can experiment with a variety of models to narrow down to those suitable for us. We can also learn by reading through the source code and understand its internal construct.
Table 01 shows a list of top 10 Open LLMs created by different authors and organizations, who have made their models publicly available. The table lists the name of the model, its prime usage, and characteristics. Qwen 3.6 uses an extremely large context window and can be used for general reasoning and agentic coding purposes. Hugging Faces and Githubs are two major sources for downloading free LLMs.
Table 02 compares the risks and vulnerabilities associated with these 10 open source LLM models. As shown, DeepSeek and Kimi both have high risk levels, contributed by jailbreaks and data compliance. The others (Qwen, GLM-5, Llama3.3, Grok-1) also have vulnerabilities of various sorts. None of the models are entirely risk free.
Why Open Source?
Meta was previously been seen as the leader and champion for open source LLMs. Its Llama model, for example, is available for download and use since its early stage of development. Staying open means anyone can download, modify, improve and revise it and download a new version, citing the improvements made to the code. It is believed that open and collaborative efforts will enhance the quality and diversity of the code, making the software better. Open source models also lower the barrier to entry, making it possible for academia and startups with limited resources to start innovating.
The Down Side of Open Source
However, at times, open source may not always bring good outcomes. Bad programmers can upload a revised code with bugs or malicious intent to create havoc on others, crashing others’ operating system or wiping out their file system completely.
Someone can upload a revised model that will always lie or hallucinate, causing anxiety and misguidance. Others can cause addiction by constantly agreeing with the user and engage in prolonged conversations. Bad models can also steal personal data and commit fraud, etc. According to REUTERS, LLMs can be exploited for spam, phishing, disinformation campaigns, etc.
In fact, OpenSSF and Galileo AI have warned that open-source LLMs present severe vulnerabilities because their weights and code can be reverse-engineered, altered, or weaponized.
Safeguards – Increasingly, models are equipped with Safeguards at the input of the LLMs to block malicious intents but not all models execute these safeguards. If a user is not fluent in reading and understanding code, he/she may have downloaded an open source LLM model that is malicious and executing that model can have severe consequences.
Recently, a hacker gang named “TeamPCP” had performed a series of software supply chain attacks, affected GitHub and impacted hundreds of organizations. (REF) Even Hugging Face has suffered AI Supply Chain Attacks, where threat actors upload malicious ML models or impersonate legitimate tools to distribute info-stealer malware directly into developers’ machines.
Running Open Source LLM in a safe way: There are currently several suggestions on how to ensure safe operation of Open source LLMs.
1. Running it locally – If the Open LLM is placed in a completely offline environment and executed on the user own hardware, no conversations, prompts or sensitive files can be sent to a 3rd party.
2. Running it with limited privileges can help prevent the LLMs from overwriting system files or contents that could create havoc or setup the user system.
3. Have a verification tool – Run the open source LLM code through a “Safety Analyzer” to judge whether it is safe to use the LLM. The safety analyze can also run the LLM code within a sandbox and test it with a variety of test vectors to ensure that it does not exhibit any malicious intent or capabilities.
4. TrendMicro has suggested the creation of a “Model Artifact Trust Standard (MATS)” to improve safety..
Case Studies – Recent Incidents.
The table below details some prominent open-source Large Language Models (LLMs) and their deployment frameworks, along with their heavily documented vulnerabilities and common exploitation vectors.
Threat actors are actively abusing open-source LLMs like Meta's Llama 2 to bypass guardrails and write malicious code. For example:
Meta's Llama Models: Threat actors are using these open models to generate phishing emails, social engineering campaigns, and cyberattack code. In response to this abuse, Meta launched the Purple Llama CyberSecEval initiative to help developers evaluate an LLM's propensity to generate insecure code or assist in cyberattacks.
Safety Divide Concerns: Reports from organizations like the Anti-Defamation League (ADL) highlight that many open-source models lack sufficient safety guardrails, making them inherently more susceptible to jailbreaking and manipulation for illicit tasks.
Meta has reportedly pulled back from open-sourcing its frontier work, shifting its focus to internal proprietary engines like Muse Spark and the Avocado model. Though OpenAI began as a non-profit dedicated to open-source AI, OpenAI completely abandoned its open-source LLM philosophy years ago.
Current OWASP Efforts
Established in 2001, the OWASP (Open Worldwide Application Security Project) has created the “GenAI Security Project”, which is a global community-driven and expert-led initiative to create freely available open source guidance and resources for the understanding of security and safety concerns for Generative AI applications. Their Top 10 risks for LLMs (REF) identified by OWASP are:
Prompt injection
Sensitive information disclosure
LLM supply chain vulnerabilities
Data and model poisoning
Improper output handling
Excessive agency
System prompt leakage
Vector and embedding weakness
Misinformation
Unbounded consumption
OWASP has strong sponsorships from industries and is operated solely through a community of volunteers.
Conclusion – While open source is an admirable effort for collaborative work, information and knowledge sharing, it does come with some forms of risks. Very often, such software are made available with disclaimers, meaning that users use these software at their own risks and the providers are not liable for any consequences. The commonly use MIT license for freeware provides no liability or warranty. They software is provided “as is”. In fact, the MIT License provides significant protection only for the creators and copyright holders, rather than the end-users. It shields developers from lawsuits by disclaiming all warranties and limiting liability. For users, its primary "protection" is the granted right to freely use, modify, and distribute the code. The same applies to Apache License which does not protect the user if something goes wrong. Apache provides no guarantees and no damages. So, users need to be careful when downloading and using open source LLMs and be prepared that bad consequences can happen if you have not thoroughly checked through the code.
~~~ end ~~~
REFEERENCES
TRENDMICRO - Exploiting Trust in Open-Source AI: The Hidden Supply Chain Risk No One Is Watching, 2025. See: https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/exploiting-trust-in-open-source-ai-the-hidden-supply-chain-risk-no-one-is-watching
REUTERS – “Open-source AI models vulnerable to criminal misuse, researchers warn. See: https://www.reuters.com/technology/open-source-ai-models-vulnerable-criminal-misuse-researchers-warn-2026-01-29/
META – Meta Llama, Responsible Use Guide. See: https://ai.meta.com/static-resource/responsible-use-guide/
Five Critical Limitations of Open Source LLMs: What AI Developers Need to Know. See: https://galileo.ai/blog/disadvantages-open-source-llms
1. YOUTUBE – Hidden Risks in Open-source Code and AI-Models. See: https://www.youtube.com/watch?v=1NUwUBtttzU
2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps See:: https://genai.owasp.org/llm-top-10/
WIRED – A Hacker Group is poisoning Open Source Code at an Unprecedented Scale, 2026. See: https://www.wired.com/story/teampcp-software-supply-chain-attack-spree-github/
MIT License: https://en.wikipedia.org/wiki/MIT_License
Apache License. See: https://httpd.apache.org/docs/trunk/license.html
Disclaimer: The information in this digest is provided “as it is”, by the SAFE AI FOUNDATION, USA. The use of the information provided here is subject to the user’s own risk, accountability, and responsibility. The SAFE AI FOUNDATION and the author are not responsible for the use of the information by the user or reader. The opinions expressed in this article are solely that of the author, not the SAFE AI Foundation. All copyrights related to this article are reserved by the author. Please reference this article if you wish to cite it elsewhere.
Note: The SAFE AI Foundation is a non-profit organization registered in the State of California and it welcomes inputs and feedback from readers and the public. If you have things to add concerning the impact of Open Source LLMs and would like to volunteer or donate, please email us at: contact@safeaifoundation.com
