The rubber gasket insert that's inside the window frame channel is not held in by anything, it just sits in the groove and is attached on both sides of the frame at the base of the window. So don't really worry about damaging the rubber gasket, because when you pull out the visors the gasket will probably come out with it. Obviously do this with the windows all the way down.
What you should do is take a pair of tin snips and cut the visor in the middle to make it easier to take out of the window frame. Pull out the pieces, and don't worry if the rubber gasket comes out with it because you can clean off the adhesive and pop the gasket back in the channel.