Midrasch zu Hoschea 4:4

R. Juda said in the name of Samuel: "Pashur, the son of Immer the priest [referred in Jer. 20, 1] had four hundred servants and according to others four thousand servants, all of whom were involved in priestly intermarriages, and every insolent priest is the descendent of none else than of the above servants. This differs from the opinion of R. Elazar, who said: "If thou seest an insolent priest, do not think evil of him, for it it said (Hos. 4, 4) Thy people are as they that strive with the priest."