I believe the answer is: Does the speech pose a threat to the public interest?
One example of such speech would be a speech that call for a destruction of other people's house. This type of speech could either make the target of speech feel unsafe/distress, or make other people actually persuaded to destroy the house. Since both of them are breaking the law, such speech is not protected under free speech.
