Apply the condition for Toppling,
The Tension tends to create a clockwise torque about the COM. This causes the normal reaction to shift to the right , and in the limiting case, torque due the normal reaction = 0,when it passes through P
Now, torque due to tension = Th (about P)
Torque due to mg = Mgh/2 (about P )
Now, in the limiting case when the body is just about to topple,
Mgh/2 = Th
Mg/2 =T
Now, Consider the acceleration of block m , a
T-mg= ma
But, T is also equal to Ma , because in the absence of any friction, it would have been T=Ma
Therefore a=g/2
i.e. T-mg = mg/2
T = 3mg/2
since T=Mg/2
Mg=3mg
m/M = 1/3