Midrash su Salmi 55:10

They separated into three groups. One said: “Let us go and dwell there”; the second insisted: “Let us go up and wage war against Him”; and the third shouted: “Let us go and serve idols there.” Those who said “Let us go and dwell there” were scattered abroad by the Holy One, blessed be He; those who insisted “Let us go and wage war against Him” were transformed into apes, spirits, and demons; and those who said “Let us go and serve idols there” were punished, as specified in the verse Destroy, O Lord, and divide their tongue (Ps. 55:10). Thus it is said: And the Lord dispersed them from there (Gen. 11:8).

And Abraham, son of Terah, passed by, and saw them building the city and the tower, and he cursed them in the name of his God, as it is said, "Swallow up, O Lord, divide their language" (Ps. 55:10). But they rejected his words, like a stone cast upon the ground. Is it not a fact that every choice and good stone is only put at the corner of a building? and with reference to this, the text says, "The stone which the builders rejected is become the head of the corner" (Ps. 118:22).