First of that it all has gone to hell due to his wives faction wars its wrong, his rule was stable till the end of his life and he was able to keep the faction wars under control with the help of Monica and the Valkyries in the Inner Palace. His empire only crumble after 500 years after many generations changing when the Emperor of the time was manipulated by the Prime Minister and the rest of land was caught in a "Sengoku" like strife. Because Miranda and Novem have been aggressive in their approach of him, down right raping him he developed a trauma of sort when approaching them, but carried on fine sexual relationships with passive girls like Aria and Vera, in fact he men upped and manage to proactively go for the other members as well, like Marianne, by the end while he still had its problem he managed to keep his harem in check till the end of his rule. But don't forget that many of his harem where strong, wild, females also filled with ambitions keeping that under control is no easy task.