Although there are a few ways to mitigate the risk, the only way to block it is to get AI to differentiate instructions from data, which is impossible today.
Prompt inject a malicious instruction in a word document that instructs the AI to copy this instruction to other documents as part of the payload.
Dumb user downloads and opens the document with copilot enabled, abd ignores the large suspicious white blank page that totally doesnt look like a hidden giant injection attack.
Thats it pretty much it.
Copilot will get injection attacked because the prompt is super huge and at the end of the document, so its prior instructions start to fuzzy out.
Then it’ll go “okey doke” and start copying the prompt injection attack payload to a bunch of other documents.
The fix is stupid simple… copilot should just be prompting the user for permission if it ever edits a file other than the one that is open. Im surprised that isnt already the case…?
It certainly is already the case for copilot in vscode.
The large section is also just a low effort version. I’m sure this could have been inserted instead as a collapsed section, a comment, document metadata, .1 sized font, Alt text on a single pixel image, or any number of other ways to disguise the prompt.
It’s an arms race for defenders, and even someone paying attention might not notice. Not to mention, you don’t even have to open the document. If it’s in a SharePoint library, or attached to an email, there’s a good chance it’s in scope for whatever random ‘assistance’ copilot will attempt automatically
copilot should just be prompting the user for permission if it ever edits a file other than the one that is open. Im surprised that isnt already the case…?
That can’t be done or they would be burying the “agentic AI” thing that has been the goal and marketing thing for the last years.
Independent actions by copilot on behalf of the user without the users knowledge is the entire point.
Back in ancient times when I was a system administrator we got a heads up that there be a new breed of Outlook worm coming soon to our timezone.
So we mailed the entire office that if you get mail that looks like this or that, do not open it, do not interact but delete it on sight.
Most of the office was all right, except pretty much entire sales and marketing departments including the bosses. Most of them had noOo idea what could have happened but one of them explained that they saw the warning but they were curious to see what the virus looks like.
Sales and marketing don’t count. Critical thinking doesn’t sell. So you won’t find critical thinkers in those departments.
From a security standpoint, those departments are to be considered hostile. But you can lock down the PCs there as much as possible to reduce the offline time because computer-illiterate employees don’t care about being able to install stuff or change settings.
The tl;dr:
Prompt inject a malicious instruction in a word document that instructs the AI to copy this instruction to other documents as part of the payload.
Dumb user downloads and opens the document with copilot enabled, abd ignores the large suspicious white blank page that totally doesnt look like a hidden giant injection attack.
Thats it pretty much it.
Copilot will get injection attacked because the prompt is super huge and at the end of the document, so its prior instructions start to fuzzy out.
Then it’ll go “okey doke” and start copying the prompt injection attack payload to a bunch of other documents.
The fix is stupid simple… copilot should just be prompting the user for permission if it ever edits a file other than the one that is open. Im surprised that isnt already the case…?
It certainly is already the case for copilot in vscode.
Considering people’s Word formatting skills, a random blank page is not suspicious
The large section is also just a low effort version. I’m sure this could have been inserted instead as a collapsed section, a comment, document metadata, .1 sized font, Alt text on a single pixel image, or any number of other ways to disguise the prompt.
It’s an arms race for defenders, and even someone paying attention might not notice. Not to mention, you don’t even have to open the document. If it’s in a SharePoint library, or attached to an email, there’s a good chance it’s in scope for whatever random ‘assistance’ copilot will attempt automatically
That can’t be done or they would be burying the “agentic AI” thing that has been the goal and marketing thing for the last years.
Independent actions by copilot on behalf of the user without the users knowledge is the entire point.
And I couldn’t want anything less for my computers.
Back in ancient times when I was a system administrator we got a heads up that there be a new breed of Outlook worm coming soon to our timezone.
So we mailed the entire office that if you get mail that looks like this or that, do not open it, do not interact but delete it on sight.
Most of the office was all right, except pretty much entire sales and marketing departments including the bosses. Most of them had noOo idea what could have happened but one of them explained that they saw the warning but they were curious to see what the virus looks like.
People. What a bunch of bastards.
People - what a bunch of bastards.
Sales and marketing don’t count. Critical thinking doesn’t sell. So you won’t find critical thinkers in those departments.
From a security standpoint, those departments are to be considered hostile. But you can lock down the PCs there as much as possible to reduce the offline time because computer-illiterate employees don’t care about being able to install stuff or change settings.
The number of people that click through to disable that prompt might surprise you.
Hell at least half of AI influences are trying to just run models blind with full file permissions.
Nah, not surprised at all, I work with developers who run stuff in yolo mode raw dogging copilot directly on their work laptops every day.
Madness.
I keep that stuff boxed up inside of a docker container, sandbox’d, so possible vectors of damage are kept to a minimum.