This causes crashes on some boards, e.g. rockpro64. In any case, we
should not do it.
Check the usb_started flag to avoid this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Jonas Karlman <jonas@kwiboo.se>
Tested-by: Tom Rini <trini@konsulko.com>