but what if there were more to this.

What if u were with someone who had a very dirty past?
It disturbs you all the time. U manage to supress it at times but it still comes back and haunt you. Because you know supressing your feelings isn't the answer.

Would you leave this person? Or stay and wait for this feeling to die down?