The word "Ban" usually means that it not only removes the user but also prevent him from rejoining for some time, hence the word "Kick" would be more fitting for it's current role.

Or alternatively: "Remove" or "Remove player"

Funny thing is about the time you posted this thread i wanted to post the exact same feedback and forgot to do so. It's great that you posted this though! Agreed, "Ban" should not be used for terminology here, until recently when I tested it for fun, i thought it meant permanent kick with no recourse since I couldn't find a place to unban party users. "Kick from party" should defintively be used instead.