Turning on Gamepad UI makes it so the interaction circle for the fenced garden (at least with the tutorial Shabby Fence Garden) does not show up and thus you cannot plant crops or remove the fence.
Changing back to regular UI allows you to interact with said garden.
Bug Reports
Use this forum section to report bugs during your adventures.