I do it all the time, and I've never had an issue, interestingly on one of my printers if I move the stepper very quickly with the 12V supply not plugged into the wall, I can see the PSU fan move in response.
I believe the stepper motors have clamping diodes that prevent the coil voltage rising above the supply voltage - if it tries to current can flow into the power supply - which is why the fan would whir.
If you move the stepper very fast, yes. I installed new printed bushings on the extruder carriage and was moving it back and forth by hand and poof. Burned out the pololu.
But i do move them slowly without problem.