I have also found that driving at high speeds with the sunroof open makes a very loud wind noise inside the vehicle, regardless of roof rack crossbar position.
I have purchased a Toyota Sunroof Wind Deflector (Toyota Part Number Part #: 00016-48505) for my 2004 RAV4 which is bolt/clip mounted to the inside edge of the sunroof opening. This accessory reduces the interior wind noise significantly, but not entirely.
Toyota Sunroof Wind Deflector - Toyota Part Number Part #: 00016-48505 - fits all 2001 , 2002 , 2003 , 2004 , and 2005 models; and has Toyota 1-year/unlimited mile warranty. Price about $50.
I have found that by leaving the driver and front passenger windows up and lowering the rear drivers side and passenger side windows down about 4 inches or so, dramatically reduces interior wind noise.