Infrared (IR) sensors have been widely used for military and civil industry, while carbon nanotube (CNT) research progresses have been made in recent years. This led to investigation and development of CNT based IR detectors. In this paper, CNT-metal based Schottky barrier structure was designed to detect IR irradiance, and different interfacial layer, including SiO2, parylene-C, Si3N4, quartz, and polyimide, was used under CNT-metal contact to isolate substrate noise. The experimental results have shown that, the polymer based interfacial layer has strong substrate noise reduction on CNT IR sensor. The dark current is less than 0.3 nA under the condition of photocurrent 170 nA, noise at 0.5 nA when CNT-metal contact was fabricated on polyimide polymer due to substrate isolation and noise reduction.