Sacked Tory party chair overcame many challenges but was unable to avoid questions over his tax affairs