Abusers don't abuse everybody.
If they did, they would be easy to spot. They would all already be in jail, ostracized by the community or committed to a local psychiatric ward.
Real abusers are selective in who they mistreat. Abuse victims are typically someone close, who is powerless to retaliate or unwilling to report the abuse. Abusive behaviors are typically kept behind closed doors and restricted to moments when there are no objective witnesses. A person who mistreats you may mistreat only you and may be a model citizen to everybody else.