I have 2009 Rav Ltd - intermittent problem with key fobs. Had to dealer twice. Replaced 1 fob and new battery in other. When they don't work, it won't lock or unlock door - I have to use key. Then I have to hold fob up to start button when starting car.
Sure seems like a battery problem. I would verify the batteries are ok by checking the voltage. After proving they are good then that can be checked off the list of possibilities. I have had new batteries that were bad. I have learned the hard way to never assume any battery is good just because it's new. Remove both and check voltage with a digital meter. They will stop working at about 2.6 volt.
Where are you storing your key fobs? If they're constantly being woken up by the LF "smart key" signal, because they're too close to the car, that could be causing an issue. I don't know if the fobs are programmed to account for such an incident, but this kind of issue has been seen with other manufacturers' smart key systems so it's safe to assume that it is not.