Midrash sur Rois 1 1:36

The law: If a Jewish person who passes before the ark [to lead the prayer], what is the law, should it be permitted for him to answer "Amen" after the [blessings of] the Kohanim? Such taught the sages: One who passes before the ark, he should not answer "Amen" after the Kohanim because of distraction. Our rabbis taught us: If he can answer without being distracted, he may answer. Why? Because nothing is greater before the Holy One, blessed be He, more than the "Amen" that the Jews answer. Rabbi Yehudah bar Simon: This "Amen" has three intentions. Oath, acceptance, faithfulness. How do we know "oath"? As its says, (Numbers 5:19-22) "The Kohen should put her under oath... and the woman is to say, Amen Amen."