Decline the invitation (loitering)