Fragile DEMI LOVATO’s friends and family are ter­rified that TV’s “Mr. Nasty,” Simon Cowell, is going to drive the former Disney cutie back to booze and drugs!

While the two have been engaging in playful banter since 20-year-old Demi joined “The X Factor” in May, sources say those close to her fear Simon is going to ramp up his cutting remarks as the season pro­gresses. And when he does, they’re worried she’ll crack!

Even Demi seems to be on to Simon’s caustic ways.

“I always have my wall up when it comes to him,” she recently said. “I’m like, ‘You’re nice right now… but I know what’s coming.’”

The chart-topping singer and for­mer star of Disney’s “Sonny with a Chance” has admitted she was the victim of bullying in middle school – and says she thinks it’s part of the reason she’s struggled with eating disorders, cutting and addictions to drugs and alcohol.

She finally went through rehab two years ago when she realized she’d hit rock bottom, after getting into a brawl with a dancer while touring with the Jonas Brothers.

Meanwhile, Simon earned his “Mr. Nasty” nickname for the way he berated contestants AND his co-workers – including Paula Abdul – during his run on “American Idol” and now “The X Factor.” In fact, Paula has said his cruel on-air re­marks sometimes affected her long after the cameras stopped rolling.

“He’s a bully, he’s a masterful bul­ly,” she said in an interview. “The first day I went to work on Season 1, I quit seven times.”

The biggest problem, says the source, is Simon’s penchant for rec­ognizing people’s weaknesses, then using those frailties against them.

“Demi seems tough, but she still struggles with personal issues,” added the source. “If Simon starts making remarks about her drink­ing and drugging past, it will drive her over the edge for sure.”