Maybe thermistor is not mounted properly and when your fan blowing at the thermistor, the temperature drops?
Not sure how is thermistor mounted in e3d v5 - if it's (as in v6) screwed in with a wide screw - I'd check if it's tight and no gaps to actual thermistor chamber.
If it's glued in (as in jhead)- I'd check the glue - maybe it's worn or something.
Similar thing happened to me with j-head, shortly before the actual glue fell apart and thermistor came out of the heatblock, but moving fan so it didn't blow at the thermistor helped in the meantime.
/Larry
(www.filamentshop.ie)