serial: ns16550: Remove unwanted cast

This cast does not seem correct, since we should be writing to a pointer,
not a ulong.

This fixes the following warning on nds32:

-ns16550.c:49: warning: passing argument 2 of 'writeb' makes pointer from integer without a cast

Signed-off-by: Simon Glass <sjg@chromium.org>
diff --git a/drivers/serial/ns16550.c b/drivers/serial/ns16550.c
index 87a0917..b2da8b3 100644
--- a/drivers/serial/ns16550.c
+++ b/drivers/serial/ns16550.c
@@ -46,7 +46,7 @@
 					defined(CONFIG_AM33XX)
 	serial_out(0x7, &com_port->mdr1);	/* mode select reset TL16C750*/
 #endif
-	serial_out(UART_LCR_BKSE | UART_LCRVAL, (ulong)&com_port->lcr);
+	serial_out(UART_LCR_BKSE | UART_LCRVAL, &com_port->lcr);
 	serial_out(0, &com_port->dll);
 	serial_out(0, &com_port->dlm);
 	serial_out(UART_LCRVAL, &com_port->lcr);