Carbon nanotube (CNT) is a novel one dimensional (1D) material that has unique electrical and optoelectronic properties. Photo-sensors using CNT can sense infrared signals by using Schottky barriers between metal and nanotube, which are able to separate photo-generated electron-hole pairs in order to generate photocurrent or photovoltage for detection and quantification. It has been demonstrated that both asymmetric metal structure and electrical field can improve the performance of the sensors by manipulating the energy alignment between metal and CNT. However, it is not clear how to optimize the design of the CNT photo-sensors. An asymmetric multi-gate field effect transistor based infrared detector was fabricated, integrating with asymmetric metal structure (Au-CNT-Al) and multiple gates, which allow for controlling the doping level at source, drain and channel independently. It was found that dark current was suppressed and photocurrent was enhanced by applying negative gate voltages, thus improving sensor's performance. The CNT detector exhibited similar photo-response when modulating the doping level of CNT segments at source, drain and bulk. We ascribe this to the charge distribution that has a long tail extending over the whole tube.