A free body diagram of a mass pully system is shown below
Now, tension in the string T−Mg=0 ⇒ T=Mg Force in vertical direction, F1=T+mg⇒F1=Mg+mg=(M+m)g Similarly, force in horizontal direction F2=T=Mg Now, the resultant force Fres=F12+F22+2F1F2cos90∘⇒Fres=F12+F22=(M+m)2g2+M2g2⇒Fres=g(M+m)2+M2 Hence, the resultant force is g(M+m)2+M2.