Problem ansering calls using AirPods

I have an iPhone XR and Airpods 2 running iOS 13.5.1.

I experienced the following problem when trying to anser a phone call when wearing the Airpods. This occured several times in the past few days.

1. I am reading a news article using VoiceOver and listening through my Airpods.
2. Suddenly the phone goes quiet. VoiceOver doesn't say anything as I flick back and forth. No sound through the iPhone or the Airpods.
3. I try to tap twice on the Airpods to anser. I also try tapping on the phone to anser. I still hear nothing from the call or from VoiceOver.
4. Finally I take out the Airpods and can anser the phone using a two finger double tap and can hear both the phone call as well as feedback from VoiceOver.

The audio route in the rotor is set to "default", and in my phone settings I have the phone set to anser on the default device.

Has anyone else experienced this difficulty trying to anser phone calls when wearing AirPods? Any suggestions as to what can be going wrong? This is quite frustrating and can lead to missed phone calls.

Thanks for any advice.

--Pete