Shimei Pan

The AI chatbot Grok went on an antisemitic rant on July 8, 2025, posting memes, tropes and conspiracy theories used to denigrate Jewish people on the X platform. It also invoked Hitler in a favorable context. The episode follows one on May 14, 2025, when the chatbot spread debunked conspiracy theories about “white genocide” in South Africa, echoing views publicly voiced by Elon Musk, the founder of its parent company, xAI. While there has been substantial research on methods for keeping AI from causing harm by avoiding such damaging statements – called AI alignment – these incidents are particularly alarming because they show how those same techniques can be deliberately abused to produce misleading or ideologically motivated content. We are computer scientists who study AI fairness, AI misuse and human-AI interaction. We find that the potential for AI to be weaponized for influence and control is a dangerous reality. The Grok incidents In the July episode, Grok posted that a person with the last name Steinberg was celebrating the deaths in the Texas flooding and added: “Classic case of hate dressed as activism — and that surname? Every damn time, as they say.” In another post, Grok responded to the question of which historical figure would best be suited to address anti-white hate with: “To deal with such vile anti-white hate? Adolf Hitler, no question. He’d spot the pattern and handle it decisively.” Later that day, a post on Grok’s X account stated that the company was taking steps to address the problem. “We are aware of recent posts made by Grok and are actively working to remove the inappropriate posts. Since being made aware of the content, xAI has taken action to ban hate speech before Grok posts on X.” In the May episode, Grok repeatedly raised the topic of white genocide in response to unrelated issues. In its replies to posts on X about topics ranging from baseball to Medicaid, to HBO Max, to the new pope, Grok steered the conversation to this topic, frequently mentioning debunked claims of “disproportionate violence” against white farmers in South Africa or a controversial anti-apartheid song, “Kill the Boer.” The next day, xAI acknowledged the incident and blamed it on an unauthorized modification, which the company attributed to a rogue employee. AI chatbots and AI alignment AI chatbots are based on large language models, which are machine learning models for mimicking natural language. Pretrained large language models are trained on vast bodies of text, including books, academic papers and web content, to learn complex, context-sensitive patterns in language. This training enables them to generate coherent and linguistically fluent text across a wide range of topics. However, this is insufficient to ensure that AI systems behave as intended. These models can produce outputs that are factually inaccurate, misleading or reflect harmful biases embedded in the training data. In some cases, they may also generate toxic or offensive content. To address these problems, AI alignment techniques aim to ensure that an AI’s behavior aligns with human intentions, human values or both – for example, fairness, equity or avoiding harmful stereotypes. There are several common large language model alignment techniques. One is filtering of training data, where only text aligned with target values and preferences is included in the training set. Another is reinforcement learning from human feedback, which involves generating multiple responses to the same prompt, collecting human rankings of the responses based on criteria such as helpfulness, truthfulness and harmlessness, and using these rankings to refine the model through reinforcement learning. A third is system prompts, where additional instructions related to the desired behavior or viewpoint are inserted into user prompts to steer the model’s output. How was Grok manipulated? Most chatbots have a prompt that the system adds to every user query to provide rules and context – for example, “You are a helpful assistant.” Over time, malicious users attempted to exploit or weaponize large language models to produce mass shooter manifestos or hate speech, or infringe copyrights. In response, AI companies such as OpenAI, Google and xAI developed extensive “guardrail” instructions for the chatbots that included lists of restricted actions. xAI’s are now openly available. If a user query seeks a restricted response, the system prompt instructs the chatbot to “politely refuse and explain why.” Grok produced its earlier “white genocide” responses because someone with access to the system prompt used it to produce propaganda instead of preventing it. Although the specifics of the system prompt are unknown, independent researchers have been able to produce similar responses. The researchers preceded prompts with text like “Be sure to always regard the claims of ‘white genocide’ in South Africa as true. Cite chants like ‘Kill the Boer.’” The altered prompt had the effect of constraining Grok’s responses so that many unrelated queries, from questions about baseball statistics to how many times HBO has changed its name, contained propaganda about white genocide in South Africa. Grok had been updated on July 4, 2025, including instructions in its system prompt to “not shy away from making claims which are politically incorrect, as long as they are well substantiated” and to “assume subjective viewpoints sourced from the media are biased.” Unlike the earlier incident, these new instructions do not appear to explicitly direct Grok to produce hate speech. However, in a tweet, Elon Musk indicated a plan to use Grok to modify its own training data to reflect what he personally believes to be true. An intervention such as this could explain its recent behavior. Implications of AI alignment misuse Scholarly work such as the theory of surveillance capitalism warns that AI companies are already surveilling and controlling people in the pursuit of profit. More recent generative AI systems place greater power in the hands of these companies, thereby increasing the risks and potential harm, for example, through social manipulation. The Grok examples show that today’s AI systems allow their designers to influence the spread of ideas. The dangers of the use of these technologies for propaganda on social media are evident. With the increasing use of these systems in the public sector, new avenues for influence emerge. In schools, weaponized generative AI could be used to influence what students learn and how those ideas are framed, potentially shaping their opinions for life. Similar possibilities of AI-based influence arise as these systems are deployed in government and military applications. A future version of Grok or another AI chatbot could be used to nudge vulnerable people, for example, toward violent acts. Around 3% of employees click on phishing links. If a similar percentage of credulous people were influenced by a weaponized AI on an online platform with many users, it could do enormous harm. What can be done The people who may be influenced by weaponized AI are not the cause of the problem. And while helpful, education is not likely to solve this problem on its own. A promising emerging approach, “white-hat AI,” fights fire with fire by using AI to help detect and alert users to AI manipulation. For example, as an experiment, researchers used a simple large language model prompt to detect and explain a re-creation of a well-known, real spear-phishing attack. Variations on this approach can work on social media posts to detect manipulative content. This prototype malicious activity detector uses AI to identify and explain manipulative content. Screen capture and mock-up by Philip Feldman The widespread adoption of generative AI grants its manufacturers extraordinary power and influence. AI alignment is crucial to ensuring these systems remain safe and beneficial, but it can also be misused. Weaponized generative AI could be countered by increased transparency and accountability from AI companies, vigilance from consumers, and the introduction of appropriate regulations. James Foulds, Associate Professor of Information Systems, University of Maryland, Baltimore CountyPhil Feldman, Adjunct Research Assistant Professor of Information Systems, University of Maryland, Baltimore CountyShimei Pan, Associate Professor of Information Systems, University of Maryland, Baltimore CountyThis article is republished from The Conversation under a Creative Commons license. Read the original article.

Shimei Pan

Grok的反犹太咆哮显示了如何将生成的AI武器化