Rob Lowe is taking action against two of his family’s former nannies for allegedly violating a confidentiality agreement, defamation and infliction of emotional distress. Lowe also filed a lawsuit against his former chef, who, he alleges, had sex on his bed “with third parties” when the family wasn’t home, stole prescription drugs and broke several security cameras.

Lowe first went public with with his story on Monday when he penned a blog on the Huffington Post. “A former employee is demanding my wife Sheryl and I pay her $1.5 million by the end of the week or she will accuse us both of a vicious laundry list of false terribles,” blogged Rob, who employed the woman on and off for seven years. “It is an attempt to damage and humiliate not only my wife and me, but our two young sons as well.”

The woman has been identified in court papers as Jessica Gibson. In the lawsuit, Lowe says, “Defendant [Jessica Gibson] has made statements suggesting that she and Rob Lowe have a personal and intimate relationship and, in effect, accusing Rob Lowe of infidelity.” Lowe claims the statements are “false and defamatory.” Lowe also claims that, “Defendant [Jessica Gibson] has also made statements that Rob Lowe sexually harassed her.”

“No one intimidates my family,” Lowe wrote in his blog. “My wife and I have many former and long term employees, all of whom know this woman, who can and will refute any claims of anything inappropriate in our home, or anywhere else. We will defend ourselves with vigor and without fear.”

The Brothers & Sisters star also provided emails that Gibson allegedly sent his wife Sheryl after she recently quit. In the emails, Gibson allegedly apologized for abruptly walking out on them and said it had “nothing to do with you or Rob either. I love you guys.”

Lowe claims that the woman was probably motivated by her romantic feelings toward him. “Recently, a colleague of hers has come forward to reveal that this young woman had ‘a crush’ on me and told her on many occasions, “I wish he would get a divorce.”

In a separate lawsuit, Lowe and his wife claim another former nanny, Laura Boyce, “betrayed their trust and engaged in a scheme to hurt the Lowes by spreading malicious lies” during and after she served as a nanny for about a year. The Lowes are asking for at least $1,000,000 in damages.

Lowe is reportedly rejecting help from the Santa Barbara County Sheriff’s office to stop the extortion plot. Law enforcement sources told TMZ that Lowe told authorities their involvement wasn’t necessary because they were handling the matter civilly.