Your risk of embolic stroke (stroke from a clot) is much,
much higher than your risk of hemorrhagic stroke (stroke from a bleed), so I'm afraid he's quite right. Any well informed doctor in the world would agree with him on this.
You're in the same boat as people who are afraid of flying. They're perfectly happy to drive to the airport, which is by far the most dangerous part of their trip, but they're afraid of the flying part, which is by far the safest. The numbers are indisputable, but nevertheless they're afraid of flying but not driving. That's you and your fear of anticoagulants. You're afraid of the rare thing they can cause and not afraid of the common thing they prevent.
NOACs do not cause dementia. Not sure where you heard that. In fact, there's new evidence they may actually prevent dementia.
With your afib, age, and gender, you absolutely should be on the 5 mg dose religiously. Please believe this.