Let us, for the sake of analysis, break down and examine this statement. The first half on the sentence can be interpreted in a number of ways. It could for instance be taken to mean: 'I am irresponsible' or 'I fail to be responsible' or 'I am not the cause' or 'I cannot be held accountable', etc. So, you are either claiming callus disregard for the results of you actions, declaring that your actions had no effect, or declaring that you are above being accountable for the hurt caused by your actions. If we examine each of these cases we see in the first case, espousing the claim that one feels no remorse for practicing callus disregard for others seems to indicate an inhuman lack of compassion for fellow people. It would seem insanely stupid to make such a declaration. Skipping to the third case we see a circumstance where the declaration is that it is acceptable to cause mental anguish and suffering in others because of some special elevation or divine right. This again, seems like a truly half-witted statement for a reasonably enlightened person to be making. Back to the second case we come to 'I am not the cause of the emotional hurt that others feel'. People may feel that this statement is valid but of course it is false. The premise on which this is based is the false. The premise is that people are in control of their feelings. This might seem to be true but is in fact not. The amount of emotional effect you have on people is directly proportional to the value they place in you and your opinions and how much they depend on you for emotional support. A normal human person requires a certain amount of emotional reinforcement and support. To be denied this will cause severe emotional and developmental problems. Unless a person is severely autistic or otherwise unable to have emotional attachments, the need for emotional support is unavoidable. As humans we receive just about all of our feedback on our social acceptance and integration from other humans. So as you see the way you treat other people does have an emotional impact. This simply can't be helped.
As the above clearly illustrates, people cannot help but be affected by the things you say and do. It seems clear enough that to say 'I am not responsible for the the things I do that affect other people emotionally' is just as invalid as saying 'I am not responsible for the the things I do that affect other people.'. Of course the problem here is what should a reasonable person be responsible for? Well, it seems logical that a reasonable person would and should act in a reasonable and responsible fashion, treating others as they would wish to be treated.
The only things you can truly say are 'I am not responsible for the emotions of others over which I have no control.' or 'I should not be held accountable for people's emotions when I have treated those people responsibly.'. Really all that is is a statement of 'I am responsible for the things for which I am responsible.'. Be ever vigilant that you do not fall into the trap of buying into feel good, I can do no wrong, pop psych, glittering generalities that are for the most part just a way of relieving you of the responsibilities of being a good person.