In The Crown season 4 finale, “War,” Camilla tells Charles that their romance is doomed by reminding him that his great-great-grandfather and her great grandmother also had an affair. Charles, aware of the illicit romance between their ancestors, responds by saying that they loved each other to the end. At that point in The Crown, Charles and Camilla had begun an affair while Camilla was still married to Andrew Paker Bowles and Charles to Princess Diana. Rumors of Charles and Camilla’s affair had begun to circulate, but the couple continued to carry out their affair with discretion until Charles decided that he wanted to leave Diana and marry Camilla.

In The Crown, Charles believes that the people of England would appreciate his and Camilla’s love story after all they had gone through, but Camilla is very aware of how going public with their relationship would make her the villain in Princess Diana’s fairytale. Camilla tells Charles that she will never amount to anything more than Charles' mistress and that Diana will always defeat her in the court of public opinion. She cites her great grandmother and Charles' great-great-grandfather as an example, which - while true - is not a perfect comparison.

Related: The Crown Season 4 True Story: What Really Happened (& What Changed)?

Camilla's great grandmother, Alice Keppel, was one of more than fifty-five mistresses to Charles' great-great-grandfather, King Edward VII. Alice Keppel and King Edward VII got together when he was still Prince of Wales - while she was 29 and he was 56, a larger age gap than Diana and Charles - but the pair began their affair when Alice and Edward were both already married. However, both of their spouses were surprisingly okay with the affair. Alice Keppel’s husband, George Keppel, is said to have made himself scarce when Edward came around and Alexandra of Denmark, Edward’s wife, reportedly preferred Alice to the king’s many other mistresses who were less discreet and couldn’t soothe Edward’s moods like Alice did.

Emerald Fennell as Camilla Parker Bowles in The Crown

Alexandra tolerated Alice, even allowing her to join the king at his deathbed - though she reportedly had Alice removed as soon as the king lost consciousness. Alice, for her part, leveraged her position to secure a job for her brother and was financially supported by the king, who acquired shares in a rubber factory for Alice. Regardless of how Alice and Alexandra privately felt about each other, there wasn’t a bitter rivalry between the pair on the same scale as Camilla and Diana - despite the fact that Alexandra was publicly liked just as much as Diana.

Nearly 25 years after Charles and Diana’s divorce and even after Charles’ did ask Camilla to marry him, Camilla is still seen as an outsider by the royal family and much of the public. Charles and Camilla’s family history makes for compelling dramatic effect in The Crown and Charles was right – Edward did love Alice. But Alice Keppel was never hated by the public like Camilla, possibly because she wasn’t as well-known, or simply because Edward never tried to divorce Alexandra for her. History has shown and seasons 5 and 6 of The Crown will likely tell that the fictional Camilla was right about Charles leaving Diana to be with her – in the eyes of the public, she became seen as little more than the villain in Diana’s story.

More: The Crown Season 4 Heartbreaking Ending For Charles, Diana & Thatcher Explained