The Others have a history that greatly predates the time Lost’s main characters spent on the island. Not long after the crash of Oceanic Flight 815, Jack’s group came to discover that they weren’t alone. They shared the island with a group known as the Others. Led by Ben Linus, the Others provided the main source of conflict for the show’s first three seasons, while also factoring heavily into several other stories as well.

Over the course of the show’s six-season run, Lost delved deep into the history of the island’s many inhabitants. In addition to the Dharma Initiative which ran various operations on the island, there were also the Others aka the Hostiles, who had originally had a reputation as savages and killers. The Others vastly improved their station on the island when they wiped out the Dharma Initiative and took over their homes and resources in the 1970s. Lost season 5’s time travel story expanded on them a bit more by revealing that they were active in the 1950s, which was a time when Charles Widmore and Eloise Hawking were active as members.

Related: LOST Secretly Hinted At Its Time Travel Twist In Season 2

Richard’s origin story in Lost season 6’s “Ab Aeterno” provided a hint at just how long they’ve been around while also shedding light on what originally brought them to the island. His flashbacks explained that his time on the island began in 1867 when his ship, the Black Rock, crashed. Mark Pellegrino's Jacob admitted that he caused it to crash so that he could bring people to the island. After agreeing to become immortal, the Others were officially formed, with Richard as their first member. Instead of relying solely on his abilities to bring shipwrecked victims to his domain, Jacob allowed Richard to leave the island and recruit people to live. It’s unclear exactly when this began, but at some point after 1867 Richard began seeking out people willing to serve Jacob’s interests.

Richard Alpert in Lost

Most were presumably found as a result of Richard and Jacob’s recruitment process that took them off-island on multiple occasions. That was how Richard brought in members such as Mikhail, Juliet, and more. Those that he convinced to come to the island started families of their own, thus allowing the Others’ numbers to grow and prosper. That could be how Widmore and Eloise came into the picture, but their backstories (prior to 1954) were never explored on the show.

It's worth noting that not every member of the Others in Lost was personally invited by Richard. Some, like Ethan Rom and Ben, were already on the island via the Dharma Initiative. Thanks to the efforts of Richard and other events that worked out in their favor, Richard Alpert was able to build a steady group of followers for the island’s protector in his battle against the Smoke Monster.

More: Lost Theory Reveals Hurley's Curse Wasn't The Numbers' Fault