That's a good point. I though it would be advantageous to not have the extruder motor rapidly going back and forth while you were within an objects walls. I actually have retraction on mine set all the time.
I started thinking about this when I observed Cura's behavior. By default (at least in my case) Cura has a feature called "Retraction combing" turned on. With this feature turned on, whenever it can, it stays within an objects perimeter and does not retract. And this is not just for straight-line moves. Cura will take a circuitous path to get from point A to point B so that it doesn't cross perimeters and, therefore, doesn't need to retract. This feature too is flawed for the same reason - on layer 0, it may be staying within the object's perimeter, but it is actually on the outside of the object. I have this feature turned off.