nand: brcmnand: Don't use -EPROBE_DEFER

This has no useful meaning in U-Boot and will never be returned. We want
to reserve this flag for internal driver model use.

Drop the code.

Signed-off-by: Simon Glass <sjg@chromium.org>
diff --git a/drivers/mtd/nand/raw/brcmnand/brcmnand.c b/drivers/mtd/nand/raw/brcmnand/brcmnand.c
index 7349a9b..99a1c2e 100644
--- a/drivers/mtd/nand/raw/brcmnand/brcmnand.c
+++ b/drivers/mtd/nand/raw/brcmnand/brcmnand.c
@@ -2526,10 +2526,7 @@
 		if (ret)
 			return ret;
 	} else {
-		ret = PTR_ERR(ctrl->clk);
-		if (ret == -EPROBE_DEFER)
-			return ret;
-
+		/* Ignore PTR_ERR(ctrl->clk) */
 		ctrl->clk = NULL;
 	}