There were five villainous organizations that the team had to contend with in Person of Interest. These powerful groups, which were led by highly capable adversaries, were the primary threats to Reese (Jim Caviezel), Finch (Michael Emerson), and the others during the CBS show’s five-season run.

In the beginning of the series, taking on massive criminal and government organizations wasn’t part of their mission. In the Person of Interest pilot episode, Reese and Finch formed a partnership that involved saving people in need and stopping crimes before they could happen. The Machine provided Finch with social security numbers that pointed them in the direction of people involved in future homicides. Together, Reese and Finch would have to figure out if the person they were investigating was either the victim or the perpetrator. This was the original formula for the series, and It allowed for several standalone stories.

Related: Person of Interest: How Finch & Reese Found Their Dog, Bear

As the series progressed and the formula evolved, Person of Interest began to put more focus on larger arcs with each “number” being connected to something bigger. These storylines would center on an enemy organization whose goals would conflict with the team’s. Their struggles to defeat these organizations had a tendency to last for a season or more. Here’s all five of the enemy groups in Person of Interest, and how Reese and Finch took each one of them down.

Elias’ Gang

Carl Elias (Enrico Colantoni) was a low-level gangster who quietly seized power in Person of Interest season 1. Elias, the illegitimate son of a crime boss, concocted a plan to change organized crime in New York City after being looked down upon by his father. Elias believed that he could unite the various gangs in the city. As Elias started to become a threat, the mob families set their sights on him.

Elias, who managed to keep his identity a secret, used Reese to stay alive. Later, Elias succeeded in eliminating the heads of the mob families and assumed control. Part of his plan to accomplish this was to use Detective Carter’s son as leverage, but fortunately, the team was able to secure his rescue. Once he was safe, Elias was arrested and put behind bars. The problem, though, is that Elias still had control over the mob. Elias continued to be their leader for years, but his days of being a main villain were over. He became an occasional ally in season 2 and was killed off toward the end of season 5.

HR

Introduced in season 1, HR was a group of corrupt cops who had dealings with criminals. They had members in all areas of law enforcement, giving them access to a ton of resources and information. In season 1, Fusco (Kevin Chapman) infiltrated the group so that the team could keep an eye on them. They clashed with the team throughout the first three seasons, with their boss, Alonzo Quinn (Clarke Peters) finally showing himself in season 2. These recurring villains were defeated in season 3 when HR’s second-in-command, Simmons (Robert John Burke), killed Carter. This set off an all-out with HR, with Reese taking the lead in tracking down the members of HR. Simmons was apprehended and Quinn was handed over to the FBI. Thanks to the evidence Carter had gathered on HR, they made sure that any remaining high-ranking members would be arrested. After that, it was stated by Simmons that HR was dead.

Decima Technologies

Decima Technologies and their leader, John Greer (John Nolan) were the main villains of Person of Interest. They first emerged in season 2 and beating them and their own advanced A.I. Known as “Samaritan” became the focus of the show in the back half of season 3 after Greer’s company learned about the Machine and sought to gain access to it so that they could use its surveillance capabilities.

After convincing the government to launch Samaritan as their new nationwide surveillance system, Greer became one of the most powerful figures in the country. Once Samaritan was activated, Reese, Finch, Root, and Shaw had to lay low and come up with fake identities to protect themselves from Decima. The final battle with the team and Decima took place across season 5 and culminated in a last-ditch effort to destroy Samaritan with a computer virus. Greer was killed in the penultimate episode of the series when the main characters were trying to escape from one of Decima’s facilities. In the series finale, Decima’s hold on the country was broken at last when Reese sacrificed himself and uploaded the Machine to a satellite, destroying Samaritan forever.

Related: Person of Interest: Why Fusco Became A Dirty Cop In Season 1

Vigilance

Vigilance was a group whose ideals ran counter to the purpose of the Machine. Members of Vigilance believed in protecting the privacy of the country’s citizens and were willing to resort to kidnapping, acts of terrorism, and more to get what they wanted. They were led by a man named Peter Collier (Leslie Odom, Jr.). Collier and Vigilance captured Finch and through him, learned the truth about the Machine. What the team (and Vigilance themselves) didn’t understand at first was that Collier was just a tool for Decima to achieve their ultimate goal.

It was revealed that the real founder of Vigilance was John Greer, who wanted to use them as a scapegoat for a massive terrorist attack at a courthouse. The point of this attack was to prove to the government that it needed Samaritan to protect lives. Once Vigilance served its purpose and Samaritan was activated, they were promptly discarded. Collier was murdered by Decima agents. Following his death, Vigilance was blamed for the explosion at the courthouse and it didn’t take long at all for law enforcement to hunt down what was left of them. By the end of season 3, Vigilance was no longer a threat.

The Brotherhood

Led by Dominic (Winston Duke), the Brotherhood was a rival gang that opposed Elias’ authority in Person of Interest season 4. Throughout the season, Dominic worked against the team and tried to become the new leader of organized crime in New York City. What made Dominic different and more dangerous is that Reese and Finch had reached an “understanding” with Elias, who now refrained from hurting innocents. Dominic refused to follow the same rules, and began a gang war with Elias.

Dominic, with all his numbers, was winning against Elias until Elias tricked Dominic into thinking that his right-hand man, Link (Jamie Hector), was a traitor. When Dominic murdered Link, loyalty to him was shaken. With the help of the Machine, Reese saved Elias and took out a number of Dominic’s men. Dominic and Elias were both taken into police custody. The two crime lords were seemingly assassinated by Samaritan agents in the season 4 finale, but it was revealed in season 5 that Elias faked his death. Elias was able to live on (for a while), but this was the end for Dominic and the Brotherhood.

More: Person of Interest Season 6: Why The Show Was Cancelled