Midrasch zu Dewarim 20:20

7) Of the wicked: (Devarim 20:20) "Only a tree that you know that it is not a fruitful tree, it shall you destroy and cut down, etc.": Now does this not follow a fortiori: If trees, which do not see, and which do not hear, and which do not speak, because they do not grow fruits, the Holy One did not pity them against removing them from the world, then a man who does not "grow" Torah and does not do the will of his Father in heaven, how much more so will the Holy One not pity him against removing him from the world.

(Devarim 20:20) "Only a tree that you know": This is a fruitful tree. (Ibid. 19) "to come before you into the siege": Cut it down.