Suggestions/Comments: Let me start by prefacing that I am not sure if this was intened by Pearl Abyss or not. If this was intended, my suggestion is to change it, and if it was not intended, the issue should be fixed.
A little while ago, PA made it so canceling worker tasks became instantanious. This worked for workers assigned to node production, however not workers assigned to gardens. My suggestion is to make canceling workers on gardens instantanious as well. Below is an example of how it can take nearly a half hour of waiting before you can reassign a worker that was previously working on a garden.