No, don't stop the Xarelto. For one, if the Xarelto really is at fault then you stopping it might conceal that from the gastrologist. Second, don't try to argue with your PCP by throwing research papers at them. Your EP and the gastrologist are the ones who should be deciding this issue, so if your PCP wants to argue about it have her call them and discuss it directly.
And to answer your question, yes, there are studies showing that not being actively in afib doesn't mean you don't need anticoagulation. It's not as simple as your PCP (and most people) think. Having afib predisposes you to clot formation even when you're not in afib, which is why the CHADS-Vasc score is so important.