I feel your pain--made the same mistake not long ago. There was another thread about this recently and a few of us shared our cautionary tales--definitely suggest avoiding the navi system. Our dealer didn't have any Ravs with Navi in stock and got our Rav from a Chicago dealer--when it arrived and we gave it a look the weakness of the sound system didn't register with me. I should have paid more attention and declined the vehicle.
I do like the look of built in navi (as I've known too many people who've had a portable unit stolen) and that is what made me want it, but of course now I realise there were other options on that front. I do use it and for me it works fine, but I understand (from other posters) that there are better options.
I've not looked into upgrading the stereo yet, but it's on the list.