Well, that could be it. For comparison, many in the Gen 4.3 RAVs report a quick and sometimes abrupt shift with engine cold and at high idle, and backing out, then shifting forward. You have to really come to a full stop with a fraction of a second pause as shift is instantaneous to smooth it out. No lag at all, just the opposite....I experience this too, but I think it does this only when the engine is cold to reduce excessive wear on the clutch packs or it's a comfort thing...
So Toyota may have designed the lag to eliminate this for smoothness. But the OP reports 3-5 seconds lag. If it was a true 5 second delay, that would border on dangerous. He should get a stopwatch and confirm--is it 3 seconds or is it 5 seconds? Collect some real data, not estimate. Does it only do it when cold, as in first shift of the day? Is there ANY delay when fully warmed?
Then, as others say, ask to drive another 'cold' 2016 on the dealer lot to compare. With a stopwatch.