The coefficient matrix is given by ​1a5​1−a−3​13a​​ And the augmented matrix is given by ​1a5​1−a−3​13a​156​​ For infinite solution, Rank(coefficient matrix) = Rank(augmented matrix) < Order of matrix(3) ​1a5​1−a−3​13a​156​​R2​→R2​−aR1​,R3​→R3​−5R1​​100​1−2a−8​13−aa−5​15−a1​​R3​→aR3​−4R2​​100​1−2a0​13−aa2−a−12​15−a5a−20​​ For Rank(coefficient matrix) = Rank(augmented matrix =2a2−a−12=0ora=−3,4 And 5a−20=0 or a=4 So, a=4 is the correct answer