rub the wrong way

To annoy or irritate

Warren had always rubbed Alison the wrong way, so she did her best to avoid him.