As a single entity, gay marriage is harmless. Bob and Dan around the corner getting hitched doesn't really affect anyone other than as an affront to their religious beliefs. I imagine a polyamist marriage is the same. Truth be told, a incestual marriage of gay or sterile consenting adults isn't going to harm anyone beyond those two, if them.
In the big picture, all can create problems.
Gays can't procreate which, I think, is one of the main reasons marriage came to being. Of course, plenty of married couples choose not to and many unmarried couples are spitting out babies on the reg. But there was a structure of family that worked for many years. Maybe it's no longer necessary but maybe it is.
Polygamy would create a clusterfuck of legal issues. No point in listing them as I'm sure they're pretty apparent. And the family structure that worked in the past is gone.
Finally, incestual marriage that can't produce children is always going to "feel" wrong. But that's just us making judgements on the love between two consenting adults. Which is pretty much what we did with gay marriage.
Again, one single marriage, or a few, isn't really an issue. Widespread marriages of the above just might. As panda says, we don't know.