It was because of the doll with buttons. Wybie gives the doll to Coraline, and through it, the Other mother knows everything that happened to her. How did she knew that Coraline was there in the first place? We don´t really know, but maybe Wybie told his grandma about a blue haired girl wearing a yellow jumper on the rose Palace and the Other mother heard about it through another doll in Wybie´s house and made another doll that looked like Coraline. Nevertheless it´s just a theory
Any children who déplacer into the rose Palace are instantly found out about par Beldam. That's why Wybie isn't allowed inside. His Grandmother doesn't want to loose him too
