When I am looking at my placed leaders, mainly trying to activate the pop up menu to give a harmony crystal gift to improve loyalty, quite often a leader will get twitchy and jump back to the leader pool losing 10 loyalty. This usually happen when AI is moving or something makes things jerky in game, but also when my hand is twitchy. Any movement of mouse while trying to press the button to activate the menu will remove the leader cause 10 loyalty loss not to mention more possible loyalty loss if replaced. 

Please don't allow leaders to be drag and dropped for removal. 

PS I played one full game with all options maxed except minor races and pirates were minimum with 6 random enemies, slow pacing. Went well and was fun as Arnor. Eventless with no conflicts and a prestige win. It was fun, and ran well.