Can of worms, yew betcha.
I hate to think of relationships as keeping score, but sometimes it comes down to this. There's a list of good things and bad things about the other person(s) that you keep somewhere in your head. If the good outweighs the bad, you tend to stick around. And the decision about what is good and what is bad (and just how good and bad those things are) is completely arbitrary-- that's part of what makes us individuals.
As we get older, we get more baggage. The experiences of our lives just make it that way. Somebody wise once said that it's a matter of finding someone with matching baggage, essentially meaning someone who is willing to accept your baggage and you theirs.
That's not all that easy.