Words used to refer to the person we are speaking to are neutral or show respect. It is a second person pronoun, used in place of the person we are talking about with respect.
The pronoun for the person we have talked to is neutral or respectful, and is the second male pronoun used to represent the person we have discussed out of respect.