Turning on Gamepad UI makes it so the interaction circle for the fenced garden (at least with the tutorial Shabby Fence Garden) does not show up and thus you cannot plant crops or remove the fence.
Changing back to regular UI allows you to interact with said garden.