What I am against is the use of state force to shut down speech. What I am also against is allowing speech to go unrebutted.
EDIT: As I've put it elsewhere:
What I don't want is for free speech to be eroded to the point where harsh but legitimate criticism (for instance, pointing out the egregious misdeeds of various religions or members thereof) is on the border between allowable and illegal speech, because it could be taken by some to be a justification for violent behaviour.
In general, I'm opposed to hate speech laws because I think they border on thoughtcrime.
I support restricting speech on the basis of copyright, trademark, libel, slander (with truth as an absolute defence for libel and slander), and direct calls for violence or speech that would reasonably lead to an immediate threat of harm (this is where the old, tired "fire in a crowded theatre" line falls, because that is liable to incite a riot which would cause harm), but not on a broader, more amorphous notion of "hate speech." (So, as I've seen it put, given that we do accept* all of those restrictions, at this point we're just quibbling over the details.)
I support the right of people to say what they will, within the above bounds, without fear of governmental sanction, because I want that right--and other rights--to be there to protect me against governmental action when I need them to.
*Well, most of us; I think Queen has said some things about copyright.
--
I don't like how those groups use their rights, but I also want to be able to use those same rights when a majority--even a vast majority--of society dislikes how I might exercise them.