So its not just me!!
I actually took mine into the dealer and had a tech go for a drive with me - of course my RAV made me look like a fool. Didn't make a sound.
I've found the noise to be inconsistent, although it seems to happen more often when up to temp.
I've found the location hard to pin down but it sounds to me like it is coming from the rear of the vehicle.
I have the same symptom - I slow down and stop at a light, wait about 2 seconds and it starts up.... sounds like a whine or something. Then, it either goes away (sometimes) or goes away immediately when I press the gas - in drive, not neutral. The noise will remain in neutral as well, but not park. I find as soon as shift to park, the noise will stop.... but will return when I shift back into drive.
Do you guys have similar observations? I will be very interested to hear if it is indeed the fuel pump.