This isn’t supposed to happen to Trump in New York City, Duffy says