Hello there.
Babystepping is a function that works so it should work also with you.
Worse case scenario, you load a firmware where it's working then you "steal" it and inject it in your firmware.
preliminary questions:
Q1: what's your printer? Cartesian? Delta? Other?
Q2: have you tried another encoder/ lcd board with encoder? In any case, it makes sense to have some spare parts, atlest for testing.
Q3: have you tested babystepping while printer is stopped? I mean, during hotend/ bed warmup when the printer is still. No xy moves. Try then to babystep
In any case, given the fact that your babystepping does not work at all when you move +y, +x and -x and on -y it only can move up Z (not down as you need it), I have only conclusion: your firmware is broken in the babystepping part.
The way I see it:
Option1: use another firmware. It'll take some time to re-configure the "configuration.h" files and the rest but atleast it should work
Option2: if you use a modified version, report this bug to the originator and wait for a update.
Option3: drink a black coffee and start debugging the code.
good luck