מדרש אויף בראשית 26:35

And there stood Samuel before the Holy One, blessed be He, and he said: Sovereign of all the Universe ! Do not forget the sin which Esau did to his father, || for he took strange women (for his wives), who offered sacrifices and burnt incense to idols, to embitter the years of the life of his parents. Remember his sin unto his sons and unto his grandsons unto the end of all generations, as it is said, "Let the iniquity of his fathers be remembered with the Lord" (Ps. 109:14). Samuel heard the voice of Agag muttering with his mouth, saying: Perhaps the bitterness of the evil death has passed from me, as it is said, "And Agag said, Surely the bitterness of death is past" (1 Sam. 15:32). Samuel said to him: Just as the sword of Amalek thy ancestor consumed the young men of Israel who were outside the cloud, so that their women dwelt (as) childless women and widows, so by the prayer of the women all the sons of Amalek shall be slain, and their women shall dwell (as) childless women and widows. And by the prayer of Esther and her maidens all the sons of Amalek were slain and their women remained childless and widowed, as it is said, "And Samuel said, As thy sword hath made women childless, so shall thy mother be childless among women" (1 Sam. 15:33).

“They were a source of bitterness for Isaac and for Rebecca” (Genesis 26:35).
“They were a source of bitterness” – why for Isaac first? Because Rebecca was the daughter of idolaters, she was not particular about the filth of idol worship. But this one, because he was the son of the holy, he was particular about the filth of idol worship. That is why it was for Isaac first.
Another matter, why for Isaac first? It is because the matter was attributable to her, as it is stated: “The Lord said to her: Two peoples are in your womb” (Genesis 25:23).³It was because of her lineage that one of the twins was wicked. That is why it was for Isaac first.
Another matter, why for Isaac first? It is because it is the way of women to sit inside the house, and the way of men to go out on the road and learn understanding from people. But this one, because his eyes dimmed, and he was sitting inside the house, that is why it was for Isaac first.
Rabbi Yehoshua ben Levi said: He [Esau] caused the Divine Spirit to depart from his parent.