Kernel estimation in transect sampling without the shoulder condition

Y. P. Mack, Pham X. Quang, Shunpu Zhang

Research output: Contribution to journalArticlepeer-review

11 Scopus citations


We consider the estimation of wildlife population density based on line transect data. Nonparametric kernel method is employed, without the usual assumption that the detection curve has a shoulder at distance zero, with the help of a special class of kernels called boundary kernels. Asymptotic distribution results are included. It is pointed out that the boundary kernel of Zhang and Karunamuni (1998) (see also Müller and Wang (1994)) performs better (for asymptotic mean square error consideration) than that of the boundary kernel of Müller (1991). But both of these kernels are clearly superior to the half-normal and one-sided Epanechnikov kernel when the shoulder condition fails to hold. In practice, however, for small to moderate sample sizes, caution should be exercised in using boundary kernels in that the density estimate might become negative. A Monte Carlo study is also presented, comparing the performance of four kernels applied to detection data, with and without the shoulder condition. Two boundary kernels for derivatives arc also included for the point transect case.

Original languageEnglish (US)
Pages (from-to)2277-2296
Number of pages20
JournalCommunications in Statistics - Theory and Methods
Issue number10
StatePublished - 1999

ASJC Scopus subject areas

  • Statistics and Probability


Dive into the research topics of 'Kernel estimation in transect sampling without the shoulder condition'. Together they form a unique fingerprint.

Cite this