First of all, the reason why this happens is because the A/C water drain is in that area. I had to change to Polyurethane bushing. But a friend of mine re-routed the water drain and end of problem. Yes, in the picture this is the other side. But this was the cause, and the cure. And yes, its the TRD front Anti-Sway bar. I bought it from the founder of this forum many years ago.
Should you drive it like this? NO. It might, (or it might be already vibrating) damage that area and then you'll have to take it to a body and fender shop. Wield and do some other fix nickel and dimming you to death. Fix it right, and you will not have problems.
Another thing, it will also effect another large bushing creating a very bad vibration, very bad. As toy can see in the illustration. Hop this helps.