Found the solution!
This kind of wobble was a chronic problem of mine and I have posted about it before. It did come and go apparently randomly.
I was using the Prometheus hotend, but had some consistent feeding problems. I couldn't finish many of my prints, as the hotend was jamming at random intervals. While I like Prometheus, as it is very small and can heat up very fast, I gave up and changed it with an original E3D V6. When I did that, I used the same X axis car that I was using for the Prometheus, however the mounting size is slightly different so while E3D fits, there is a very small gap. I use a clamping machanism which presses the hotend to the X car, so it is quite stable. If I tried to move the hotend applying light pressure, it didn't move at all. If I applied more force I could see a very small movement but I was assuming that it was OK, as the hotend wouldn't have to receive this much abuse during normal prints. Well, it seems that I was wrong. The hotend should not move, even the tiniest bit! I thought that was impossible, because even if the hotend didn't move, it would flex the X car, resulting in movement.
I tried to fix this problem by re-designing the X car for a very snug fit for E3D. Also, I used Colorfabb XT instead of PLA or nGen (my previous car is made frpm nGen). The net result is a very very thight fit (seriously, I cannot fit the hotend manually, I have to use something metallic to pull it) and a very rigid X car (it doesn't flex). The net result: Perfect prints.
TLDR: When people tell you your hotend might be wobbling, listen to them and know that even the slightest movement can degrade printing quality considerably.