A free body diagram of a mass pully system is shown below
Now, tension in the string T−Mg=0 ⇒ T=Mg Force in vertical direction, F1=T+mg ⇒F1=Mg+mg=(M+m)g Similarly, force in horizontal direction F2=T=Mg Now, the resultant force Fres =√F12+F22+2F1F2cos90° ⇒Fres =√F12+F22=√(M+m)2g2+M2g2 ⇒Fres =g√(M+m)2+M2 Hence, the resultant force is g√(M+m)2+M2.