It was because of the doll with buttons. Wybie gives the doll to Coraline, and through it, the Other mother knows everything that happened to her. How did she knew that Coraline was there in the first place? We don´t really know, but maybe Wybie told his grandma about a blue haired girl wearing a yellow jumper on the розовый Palace and the Other mother heard about it through another doll in Wybie´s house and made another doll that looked like Coraline. Nevertheless it´s just a theory
Any children who Переместить into the розовый Palace are instantly found out about by Beldam. That's why Wybie isn't allowed inside. His Grandmother doesn't want to loose him too