You have to uncheck "auto play while mounted" and it'll go away. Unfortuantely as I have noticed, this saves fine until you logout completely from the game. You can swap characters and it'll remember but the moment you close out of the game, it'll come back. Pretty sure this is just a bug and they'll fix it. Understandibly people should avoid doing this quest until they fix it.
I do hope you are correct because it is really really annoying to close the thing every single time.