Correct option: 3) had tried - is the most suitable option. There are three forms of conditional sentences :
>Conditional 1 : ‘if+simple present, will-future’ E.g. If I find her address, I’ll send her an invitation.
>Conditional 2 : ‘if+simple past, conditional 1' (=would+infinitive) E.g. If I found her address, I would send her an invitation.
>Conditional 3 : ‘if+past perfect, conditional 2' (=would+have+past participle) E.g. If I had found her address, I would have sent her an invitation.
This is a third conditional sentence showing a condition that is possible to achieve, and so it has to be written in the same format. Thus option 3 is the most suitable option and it should be written as 'If he had tried he would have succeeded.'