I would be inspect the front axle-shafts, front differential ( gears), portals ( gears), drive-shaft, and the servo. I would disconnect the servo and check the movement of the arm and the fluidness of the servo output shaft rotation. I would also open up the front differential and inspect your gears and look at side-to side movement. Make sure that everything is seated as should be. Doesn't sound like a shim issue but dunno. For the axle/ stub shaft, hold and rotate it while cycling left/ right. Regarding portals, again... open em up to inspect the gears and look for broken teeth, debris, etc. I'd also inspect the drive shaft at the joints, make sure pins aren't backing out and listen/ feel for any binding, resistance, etc.
Maybe someone else will chime in with a better idea more specifically that might be going on but if I was winging it, the above components are where I'd look.