We know that, if A is an orthogonal matrix, then AAT=I⇒31​​12a​212​2−2b​​⋅31​​122​21−2​a2b​​=​100​010​001​​⇒​12a​212​2−2b​​​122​21−2​a2b​​=​900​090​009​​⇒​90a+2b​092a+2−2b​a+4+2b2a+2−2ba2+4+b2​​=​900​090​009​​⇒a+4+2b=0 and 2a+2−2b=0⇒a+2b=−4… (i) and a−b=−1… (ii) By solving Eqs. (i) and (ii), we get a=−2,b=−1