Is it tough to pull it out of 5th, or tough to get it into 4th afterwards?
If the latter, our RAV has the same problem - coming out of 5th, it sometimes won't go into 4th, and coming out of 2nd, it won't go into third. The reason (I'm pretty sure after experience with some other cars with cable change) is slop somewhere in the cable/linkage that engages/disengages the gear. You have two cables - one selects the 1st/2nd, 3rd/4th, or 5th/reverse 'slots', the other pushes/pulls the gear thus selected into mesh. What can happen is, when you pull the lever out of 5th, the centreing spring pushes the lever across to the middle 3rd/4th plane before it's come fully back into neutral; the interlocks then won't allow another gear to be engaged because they're not fully released from 5th (although the gear is out of mesh by that point).
The 'cure' is to hold the lever in the 5th-gear plane (don't let it slop across to 4th early) until you've pulled it right back into the neutral position. Ditto coming out of 2nd to engage third. The real cure is to find the slack in the linkage and fix it - I haven't done that yet.
If, on the other hand, your gear lever is actually hard to pull out of 5th, then it's something else, please ignore what I wrote up there.