The direction of your change doesn’t matter, the GPL license under which the program was already given out is not revocable.

If all copyright holders agree you can grant a different license in addition to the first one, or you can stop offering one license and start offering another one, all the new changes that were never offered under the first one will then only be publicly available under the new license.

But anyone who received the code at a specific time with a GPL license can keep it, modify it, distribute it onwards with the same license and so on, no matter what new terms the copyright holders begin to offer to other people later.