SIMON COWELL’s shocking affair with a married New York socialite could send him to prison – and put his $380 million fortune at risk!

A source tells The ENQUIRER that the estranged husband of Cowell’s pregnant lover, Lauren Silverman, wants the former “American Idol” judge to be prosecuted for adultery – a crime that could put him behind bars. What’s more, if the Silver­mans’ divorce suit goes to trial, insiders say it could trigger even more legal problems for Cowell, with tax auditors in both England and America combing through his financial empire.

The ENQUIRER has also learned that Lauren’s husband Andrew discovered the affair when he caught Cowell in the couple’s bedroom – and that’s when Lauren dropped the bombshell on both men that she’s expecting Cowell’s child!

“If Andrew Silverman has his way, Simon Cowell could end up broke, behind bars and even thrown out of the U.S.,” a source told The ENQUIRER.

“Andrew has told friends, ‘I want Simon crushed! I want him destroyed!’”

Silverman, a wealthy property developer, and his wife have been friends with Cowell since meeting him seven years ago while vacation­ing in Barbados, sources say.

The couple spent many summers partying on Cowell’s luxury yacht, Slipstream, off the French Riviera, and also socialized with him in Eng­land and Los Angeles.

“Andrew knew Lauren and Simon were close, but he thought Simon was simply part of their social circle,” a source told The ENQUIRER.

“During their vacations, Andrew sometimes returned to New York while Lauren remained on the yacht with Simon. That’s how much he trusted them.”

But according to published reports, 36-year-old Lauren has told pals that she became intimately involved with Cowell, 53, in late 2011 or early 2012.

And insiders say her husband was completely unaware of the illicit af­fair until June when he discovered the “X Factor” judge in the Silver­mans’ bedroom at their $4 million mansion in Bridgehampton, N.Y.

“Andrew often spent the week in New York City working and joined Lauren in the Hamptons on the weekends,” said the source.

“But he decided to pop in a day early. Right away, he spotted their bed was messed up and saw a half-naked Simon lounging in just a robe with his clothes scattered around the bedroom.

“Andrew realized Simon, caught with his pants down, was making himself at home with his mansion AND his wife – and he went ber­serk!”

Enraged, Silverman exploded, said the source.

“Lauren admitted everything and announced that she was pregnant by Simon,” said the source.

“Andrew went crazy. He called Lauren a whore, a slut and a tramp. Simon tried to calm him down, apologiz­ing and begging for Andrew’s understand­ing.

“But Andrew blew up again, and yelled, ‘I’ll finish you, you cheating bastard!’ Then he stormed out of the house.”

On July 15, Silverman made good on his threats – filing for divorce on the grounds of adultery, demanding full custody of their 7-year-old son and naming Cowell as co-respon­dent.

Even worse, Silverman instructed his attorneys to hit Cowell with a $100 million lawsuit for alienation of affections and to seek legal pros­ecution against the star for adultery.

New York state law has officially outlawed adultery since 1907. The Class B misde­meanor encompasses married people cheat­ing or an unwed person sleeping with a married one. The crime, punishable by up to a $500 fine and 90 days in jail, was last prosecuted in 2010, but there were a dozen cases before that.

While such cases rarely go to court, Andrew could use his wealth and power to have Cowell prosecuted, New York divorce attorney Scott I. Orgel of the firm Eiges & Eiges told The ENQUIRER.

“Andrew also wants his lawyers to investigate whether an adultery conviction could lead to Simon’s visa being revoked and him being de­ported to England,” said the source.

As if that’s not bad enough, if the Silvermans’ divorce suit goes to trial, sources say it could open Cowell’s business records to government tax snoops.

“Simon is afraid that exposing his finances will give British tax collectors an open door into his financial empire,” a source told The ENQUIRER.

“Andrew is betting Simon will cough up a large settlement to avoid that. But for Simon, Lauren Silver­man could be the most expensive sex toy in history.”

 Meanwhile, as the affair continues to make headlines, an entertainment industry source says Cowell’s Ameri­can TV empire could take a huge hit.

“This adultery scandal is the worst thing that could have happened to Simon,” veteran “San Francisco Examiner” and “Oakland Tribune” TV critic Bill Mann told The ENQUIRER.

“Ratings for ‘The X Factor’ slumped last year, and now sponsors will turn their backs on the show. For Simon, this is a public relations disaster!”