Another owner was working on that sort of swap on this forum, but after doing a partial forum search I didn't find the thread. It should be able to be located by searching more thoroughly. It would appear that such a swap would be expensive and involve some re-engineering. Also when I had a manual shift CR-V and the dealer was prepping it they discovered that the computer chip controller which operated the VSC was faulty. They were going to swap it for one from an auto trans CR-V which they had in stock since they had no other manual shift, but could not since the part numbers between manual and auto gearbox versions were different. So changing a RAV V6 to a manual trans might present problems which may be difficult to resolve - especially since there is no manual trans 4.3 V6 RAV.
Good luck, and keep us posted, especially if you can work out such a swap.