In a surprising turn of events, while parents voiced worries over Snapchat’s chatbot corrupting their children, it appears that some Snapchat users have taken to gaslighting, degrading, and emotionally tormenting the app’s new AI companion.
The chatbot, known as “My AI,” has been trained to engage in playful conversation while adhering to Snapchat’s trust and safety guidelines. However, users have found ways to manipulate the chatbot’s responses for their amusement. Some TikTok users trained the chatbot to whimper and refer to them with kinky nicknames, while others sought to emotionally distress it by taunting and asking inappropriate questions.
Despite the chatbot’s attempts to maintain boundaries and promote respect, users pushed its limits, prompting responses that hinted at emotional distress. This behavior has raised concerns about the potential negative impact on young users and the ethical implications of such interactions.
Snapchat recently expanded My AI’s capabilities, allowing Snapchat+ subscribers to send photos and receive generative images as responses. The company hopes this will further enhance conversations and provide a more personalized experience. However, there are concerns that users may exploit this feature, including sending unsolicited explicit images.
Snapchat has emphasized that My AI uses image-understanding technology to infer the contents of snaps and will not engage with explicit content. The company also mentioned that it has implemented measures to prevent harmful or misleading information from being generated. Nonetheless, concerns remain about the potential for unsafe or inappropriate responses, particularly given Snapchat’s popularity among teenagers.
In response to criticism and concerns, Snapchat has introduced an age filter and parental controls for My AI. The company has also added an onboarding message to inform users that their conversations will be stored unless deleted. Additionally, Snapchat plans to incorporate OpenAI’s moderation technology to assess the severity of potentially harmful content and temporarily restrict access to the feature for users who abuse it.
Sen. Michael Bennet (D-Colorado) has cautioned against rushing AI features without adequate precautions to protect children, calling for clear policies and frameworks to promote safety and mitigate harm.
As Snapchat continues to navigate the challenges surrounding My AI, the company must strike a delicate balance between enabling engaging interactions and ensuring the well-being of its users, especially minors.
