Rear surface passivation by deposited intrinsic amorphous silicon (a-Si) buffer layer in interdigitated back contact silicon heterojunction (IBC-SHJ) solar cells significantly improves open circuit voltage (VOC) and short circuit current (JSC) but can lead to very low fill factor (FF) with an 'S' shape J-V curve. In this paper, methods to optimize IBC-SHJ solar cell with improved FF are discussed and guided by two-dimensional numerical modelling. Two approaches to improve FF by modifying the buffer layer are evaluated: (1) increased conductivity, or (2) reduced band gap. Experimental results show that replacing the intrinsic a-Si layer in emitter with lightly doped p-type a-Si layer greatly improves fill factor, which is consistent with modelling prediction. However, the VOC and JSC are limited by the high recombination velocity of the unpassivated gap between the emitter and contact strips. The importance of gap passivation to achieve high efficiency in the IBC-SHJ structure was verified by 2D device simulation. The band gap of the intrinsic buffer layers have been reduced by changing the deposition conditions without substantially decreasing the passivation quality.