I've been having this problem with my 2020 xse as well. I took it to Dublin Toyota in California to have the problem looked at while it's happening, and the stupid service manager got his "master mechanic" just to say you're not supposed to plug your phone into that usb port. Then he said your phone is low on battery (it wasn't). I wanted them to write it up so it would show on my history report but they refused. He said we fixed it by unplugging the cable and now your solution is that usb port is not to be used. man they pissed me off so bad. To sit there and lie to me and i broke out the manual and showed them this is how you access carplay. I wish toyota would hold these dealership accountable.
Anyways back to your original post. Yes i run into this issue all the time. I've diagnosed it to happening when i have navigation and pandora working (or it's one of the triggers to replicate the problem). Play music on pandora and then set a destination on google maps or waze. Then purposely not follow the map guidance so the map alerts you. When the alert comes on, it interrupts the music and it glitches the headunit. From there, your music completely goes dead and give it a minute or two and your screen will go black. the only way to fix this is to turn it off and back on. If you unplug your phone/cable all it does it gets your back to the homescreen of the OEM entertainment but doesn't bring carplay back.
Ive tried to get toyota to look into this, but no luck. Since it's under warranty, i'm thinking about going back to a different toyota dealership, replicate the problem and all they'll do is get a new head unit ordered/installed. I'll then replicate the problem some more and a few things can happen, they can either buy the car back under the lemon law, or they can finally pay attention and try to get an updated fix. Really all it is, is a glitch in the headunit, which toyota contracts with some stereo manufacture to make. They just have to find the glitch in the programming and make a new update. A whole new unit won't be necessary.
That's my .02 cents at this point....i'm getting really frustrated paying over 30k for this car and the simple stereo won't work right.