Google's Threat Intelligence Group (GTIG) has stopped a zero-day exploit designed to bypass two-factor authentication in an open-source administrative tool. Researchers detected that cybercriminal actors were planning a massive attack, and clues pointing to artificial intelligence involvement included a fabricated CVSS score and a highly structured text format.
The exploit and the traces of a language model 🧠
GTIG analysts identified that the malicious code exploited an unknown vulnerability to bypass two-step verification. What caught their attention was the presence of a hallucinated CVSS score, a typical error when a language model generates data without validation. Additionally, the format of the technical report associated with the exploit showed a structure and wording very similar to outputs from AI assistants, leading researchers to conclude that artificial intelligence participated in its development.
Not even ChatGPT passed the security exam 🤖
It seems cybercriminals asked AI for help creating their exploit, but the assistant returned a report with a made-up risk score. Basically, the AI did the work for them but fabricated the homework. Good thing Google caught it, because otherwise, the open-source tool would have needed more than a simple CAPTCHA to protect itself from this digital mess.