It's not the thought, it's the judgement

Men are disturbed not by things, but by the views which they take of things. 

~ Epictetus