Midrash for Numbers 7:51

... [What about all] the praise of Joseph, who exceeded in the honor of his father? And yet he did not enter into him all the time, such that were it not that they came to tell him, "Your father is sick," he would not have known! Rather this is to inform you of his righteousness. For he did not want to be alone with his father, lest he say to him, "How did your brothers act with you?" And [then] he would curse them.... Hence he did not go to his father all the time.)