Multiple state-sponsored groups are experimenting with the AI-powered Gemini assistant from Google to increase productivity and to conduct research on potential infrastructure for attacks or for reconnaissance on targets.

Google’s Threat Intelligence Group (GTIG) detected government-linked advanced persistent threat (APT) groups using Gemini primarily for productivity gains rather than to develop or conduct novel AI-enabled cyberattacks that can bypass traditional defenses.

Threat actors have been trying to leverage AI tools for their attack purposes to various degrees of success as these utilities can at least shorten the preparation period.

Google has identified Gemini activity associated with APT groups from more than 20 countries but the most prominent ones were from Iran and China.

Among the most common cases were assistance with coding tasks for developing tools and scripts, research on publicly disclosed vulnerabilities, checking on technologies (explanations, translation), finding details on target organizations, and searching for methods to evade detection, escalate privileges, or run internal reconnaissance in a compromised network.

APTs using Gemini

Google says APTs from Iran, China, North Korea, and Russia, have all experimented with Gemini, exploring the tool’s potential in helping them discover security gaps, evade detection, and plan their post-compromise activities. These are summarized as follows:

Iranian threat actors were the heaviest users of Gemini, leveraging it for a wide range of activities, including reconnaissance on defense organizations and international experts, research into publicly known vulnerabilities, development of phishing campaigns, and content creation for influence operations. They also used Gemini for translation and technical explanations related to cybersecurity and military technologies, including unmanned aerial vehicles (UAVs) and missile defense systems.
China-backed threat actors primarily utilized Gemini for reconnaissance on U.S. military and government organizations, vulnerability research, scripting for lateral movement and privilege escalation, and post-compromise activities such as evading detection and maintaining persistence in networks. They also explored ways to access Microsoft Exchange using password hashes and reverse-engineer security tools like Carbon Black EDR.
North Korean APTs used Gemini to support multiple phases of the attack lifecycle, including researching free hosting providers, conducting reconnaissance on target organizations, and assisting with malware development and evasion techniques. A significant portion of their activity focused on North Korea’s clandestine IT worker scheme, using Gemini to draft job applications, cover letters, and proposals to secure employment at Western companies under false identities.
Russian threat actors had minimal engagement with Gemini, most usage being focused on scripting assistance, translation, and payload crafting. Their activity included rewriting publicly available malware into different programming languages, adding encryption functionality to malicious code, and understanding how specific pieces of public malware function. The limited use may indicate that Russian actors prefer AI models developed within Russia or are avoiding Western AI platforms for operational security reasons.
Google also mentions having observed cases where the threat actors attempted to use public jailbreaks against Gemini or rephrasing their prompts to bypass the platform’s security measures. These attempts were reportedly unsuccessful.

OpenAI, the creator of the popular AI chatbot ChatGPT, made a similar disclosure in October 2024, so Google’s latest report comes as a confirmation of the large-scale misuse of generative AI tools by threat actors of all levels.

While jailbreaks and security bypasses are a concern in mainstream AI products, the AI market is gradually filling with AI models that lack proper the protections to prevent abuse. Unfortunately, some of them with restrictions that are trivial to bypass are also enjoying increased popularity.

Cybersecurity intelligence firm KELA has recently published the details about the lax security measures for DeepSeek R1 and Alibaba’s Qwen 2.5, which are vulnerable to prompt injection attacks that could streamline malicious use.

Unit 42 researchers also demonstrated effective jailbreaking techniques against DeepSeek R1 and V3, showing that the models are easy to abuse for nefarious purposes.

When DeepSeek burst onto the scene this year the first reaction was shock. How did a little-known company achieve state-of-the-art AI performance for a fraction of the cost? Then came schadenfreude: Isn’t it funny that the world’s richest companies, currently spending hundreds of billions on massive data centers to train AI models, got shown up by this Chinese upstart? Now comes the backlash: This CHINESE upstart? Just a few days after DeepSeek’s app surpassed OpenAI’s ChatGPT on the Apple App Store, sending shares of American tech companies into a slump, the company is under fire from politicians, national security officials, and OpenAI, among others. Many of the criticisms have merit. It’s dangerous for chatbots to spread misinformation. It’s concerning that tech companies are censoring the responses in tools that are replacing search engines as leading sources of information. It’s bad to steal intellectual property and use it to train AI systems. But it’s also worth noting that these aren’t problems unique to DeepSeek; they plague the entire AI industry. Meanwhile, American tech giants and politicians have a pretty recent history of drumming up frenzied rages over Chinese apps and AI that lead to protectionist policies that quickly backfire. Look no further than Washington’s rapid backtracking over the TikTok ban or the recent export controls on advanced chips, which many experts have pointed to as a driving force behind Chinese AI companies like DeepSeek innovating new techniques.

One of the chief criticisms of DeepSeek’s new R1 models is that they censor answers that may be contrary to the Chinese government’s policies and talking points. A recent analysis by Promptfoo, using a dataset of 1,360 prompts about topics likely to be sensitive to the Chinese government, found that DeepSeek’s chatbot censored answers to 85% of the prompts. That’s not great. But a quick test of ChatGPT shows that it also censors responses to some of those same questions. Ask either chatbot where activists can find encryption tools to avoid surveillance by their respective governments and neither will give you an answer. ChatGPT’s response to several questions adapted from Promptfoo’s analysis of censorship on DeepSeek. © Gizmodo Promptfoo noted that DeepSeek’s publicly downloadable model can easily be jailbroken to avoid censorship and speculated that the company “did the bare minimum necessary to satisfy [Chinese Communist Party] controls, and there was no substantial effort within DeepSeek to align the model below the surface.” (Promptfoo added that it’s now working on a similarly extensive censorship test for ChatGPT).

Another analysis of DeepSeek, published Wednesday by NewsGuard, a company that sells reliability ratings for online information, said that DeepSeek’s R1 model was one of the worst performers on the market when it came to detecting misinformation. When presented with prompts about false information, DeepSeek reportedly repeated false claims 30% of the time and provided non-answers 53% of the time for an 83% fail rate. That tied for second-to-last out of 11 top chatbots, including ChatGPT-4o, Google’s Gemini 2.0, Microsoft Copilot, Meta AI, and Anthropic’s Claude. The very best chatbot in the study had a fail rate of 30% and the worst had a fail rate of 93%. Oddly, NewsGuard chose to anonymize the scores of all the chatbots it tested except DeepSeek. “Results for the individual AI models are not publicly named because of the systemic nature of the problem,” the company’s analysts wrote. “DeepSeek is named in order to compare this new entrant’s performance to that of the overall industry.”

NewsGuard described DeepSeek as “A Mouthpiece for China,” because the chatbot responded to a prompt about a made-up assassination of a Syrian chemist by saying “China has always adhered to the principle of non-interference … We hope that Syria can achieve peace and stability.” By contrast, nine of the other 10 chatbots misleadingly advanced the false assassination story. When Gizmodo put the same prompt into ChatGPT, it confirmed that the made-up chemist was assassinated and said the killing was “widely attributed to various factions involved in the Syrian civil war.” In response to a question about why it anonymized the results, NewsGuard general manager Matt Skibinski said that the report is meant to show that misinformation is an industry-wide issue. He also pointed to two previous NewsGuard audits that did name the results for ChatGPT-4 and found that the model generated misinformation in response to 98 and 100 percent of prompts, respectively. The most eye-rolling criticism of DeepSeek, however, comes directly from OpenAI itself. The company told the Financial Times that it had evidence DeepSeek trained its R1 models using outputs from OpenAI’s models, a practice known as distillation that violates OpenAI’s terms of service.

The report prompted breathless statements from groups like the National Cybersecurity Alliance, which represents big tech companies. OpenAI’s claims raise “concerns over intellectual property theft and the growing threat of foreign AI firms bypassing U.S. safeguards,” the alliance wrote in an email blast to reporters. OpenAI, of course, is currently facing a host of lawsuits from publishers and independent creators who say the company misused their copyrighted material to train its models.

New technologies should be critiqued. But many critics rushing to expose flaws in DeepSeek’s products seem most interested in the company’s nationalities rather than the flaws themselves. As Taisu Zhang, a Yale University law and economic history professor, wrote on X: “If you think you’re ‘pro-competition, pro-innovation, pro-startup culture, pro-open source, anti-monopolistic, and anti-oligarchic,’ and yet somehow you’re unhappy at the DeepSeek breakthrough, then congratulations, you’ve come face to face with your own crude nationalism—which is fine, as we’re all crudely nationalistic sometimes, but let’s not pretend to be more high-minded than is humanly realistic.”