an interpersonal difficulty

There is someone who sometimes goes to Wiscon that I don't want to interact with. We have lots of friends in common, but this is not a problem: when they join a conversational group that I'm in, I leave. However. Last Wiscon, every time this person entered a party room or the consuite when I was already there, they immediately joined the conversational group that I was in. I don't know whether this was on purpose. I might just always have been talking to the most interesting people in the room. But if it was on purpose, it's a problem. I suppose I should talk to them about it, but (1) I don't want to interact with them; (2) if it was on purpose, I don't see any good outcome; (3) if it wasn't on purpose, I think a response of, "Now that you've pointed it out, I'll avoid it when it doesn't inconvenience me," is a million times less likely than, "Now that you've pointed it out, I will do this on purpose to teach you that you don't get to control who I talk to."

Advice?