Even in those cases, sometimes the victim doesn't let go of the abuser, as a sane person would (and I'm talking about BOTH women and men). S/he not forced from outside, there's a kind of "entanglement" between the two.
In those cases, responsibility still lies on both parts. Sometimes the victim chooses to remain a victim. It often happens in co-dependency relationships.
It's easy and comforting looking at relationships in a black and white way: one is good and the other bad, one is the victim and the other the oppressor. But, most of the time, reality is more complex than that.