Can anyone help me stop thinking about my partner's history?

my current partner has a few ex's in his past. theres a pattern though, low self esteem- highly sexualized, rather homely type folks. one I went to high school with, who is mildly retarded. We are all dirty, maldeveloped little hoes deep down somewhere, i know. and i know im behaving like some elitist snob, but this guy is basically a prude, an independent bussiness man, and well respected in our community. Im just curious. does this "pattern" show a rescuer or fixer complex in his personality? Is there another way for me to use this information, instead of feeling judgemental and like im a dirty little thing that needs fixing up- is he condescending to my level in his mind?