I can speak from experience in trying to do this with an old '62 Ford F-100. I finally sent the whole dang thing to a scrap pile because it was just too much work, and that was a solid front axle, just 2 wheel drive. I don't know about the Rav's since I just got my 2001 last week, but I do know that for other vehicles the 2WD and the 4WD sit at different heights. That is the start of the problems, most also have different A-arms and suspension components too.
Again, I do not know if this is true for the Rav's but I would expect there to be some substantial differences. I personally would trade the 2WD in for the 4WD, save a whole heap of money and time.