Calls are growing for Prince Andrew to be kicked out of the Royal Lodge (pictured today), the lavish Windsor mansion he shares with his ex-wife