Correct. To clarify what happened (and I made the same mistake, even though I should know better)-- When you plug in a USB device, Windows scans for a driver, and if it can't find an exact match, it installs a generic driver. Then even though you install the Epson driver after that, Windows is still insisting on using the generic driver. This has been a problem in Windows since Windows 98 (or earlier).
By deleting the device and plugging it back in, you force Windows to re-scan the driver database while "getting your device ready". In theory, moving the USB connector to a different controller should force Windows to re-scan-- But the two front USB ports are usually on the same controller, so you'd have to switch from a back port to a front port, or vice versa (or USB 2.0 to USB 3.x).
So the rule is, always install the driver, THEN connect the USB cable. I have a vague memory that the instructions said plug it in, THEN run the software, but it was late at night, and I wouldn't swear to it.
Glad you got it working.
Oh, and for Silverfast, you're probably better off installing from the CD, registering it (which requires the physical CD), and then downloading the update from Silverfast.