"Quotes"

We also know that men who are more likely to be perpetrators of sexual violence are typically more misogynistic and anxious about their position in a perceived hierarchy (rape is not about sex, it's about power). So, yes, being perceived as less of a man for perpetrating or at the very least, not challenging sexual violence, is actually an effective deterrent.

A major problem is what you just described, which is that most men are actually quite insecure about challenging other men, and therefore often participate in social groups where that kind of deterrent is punished. There's two men in this thread already who were angered by the suggestion that they have to risk their social standing to contribute to sexual violence prevention, when they thought not being rapists themselves was all they had to do. It's also important to remember that this doesn't mean they haven't also committed some form of gender based and sexual violence, since they typically don't know what qualifies as that beyond overt sexual assault.