@drbollocks Here's something that sometimes helps me: Ask yourself, if someone else was in your situation, what would you tell them.
Because honestly, from what you're writing, you don't have much of a relationship to lose, and you seem to be clinging onto something that isn't there, if it ever was.
It's okay to break up with people for any reason or no reason at all, if only for your own peace of mind. Don't stay with someone out of fear.