What ever gave me the confidence then that it would have worked out as a relationship even if you never got married?

I would have felt bad that you couldn't go to whichever place you'd like to go to eat, to drink.
You might have felt stifled at some point.
Our differences would have gotten apparent and maybe too big for us to handle.

In the end, all I see is that I was never the right fit.
I would have never been the right fit.