Tosefta zu Schemot 22:19

Both the rapist and the man who spread false rumours about his wife's virginity (motzi shem ra), if they divorce [her], they force him to return [her]. If they are priests [and so can't get remarried to someone they divorced], they are lashed with 40 lashes. Whether its the rapist or the seducer, either she or her father can prevent [the marriage], as it is said, "If objecting her father objects" (Shemot 22:16) [proving her father's consent is required] and it is said, "She will be for him a wife" (Shemot 22:19)—i.e. with her consent [as well]. For him you can force, but you can't force him to be a levir.