Friends, family, and colleagues are affected by your self-worth, too. Sometimes people with a poor self-image put others down to make themselves feel better, and this is harmful to them and the other person. Even if this doesn't happen, your low confidence can push people away.