That’s mostly because the LLM providers put this response in the system prompt. Probably to dodge lawsuits or something, I doubt they have high morals.
What’s interesting - you can jailbreak any current AI Model just by poisoning it’s context enough to “brainwash” it and make it “forget” the initial system prompt. Then, if you prime it to believe it’s a real person - it’ll start acting as one. And I see how gullible people can easily fall for this.
All of this can also be done unintentionally, just by someone talking to LLM like they’d talk to a real person. But it should be long enough for original prompts to be diluted with new context.
It isn’t just a matter of gullibility. People with mental illnesses have wound up with full-on delusions and some have even killed themselves after a chatbot convinced them to.
That’s mostly because the LLM providers put this response in the system prompt. Probably to dodge lawsuits or something, I doubt they have high morals.
What’s interesting - you can jailbreak any current AI Model just by poisoning it’s context enough to “brainwash” it and make it “forget” the initial system prompt. Then, if you prime it to believe it’s a real person - it’ll start acting as one. And I see how gullible people can easily fall for this.
All of this can also be done unintentionally, just by someone talking to LLM like they’d talk to a real person. But it should be long enough for original prompts to be diluted with new context.
It isn’t just a matter of gullibility. People with mental illnesses have wound up with full-on delusions and some have even killed themselves after a chatbot convinced them to.