Comparing the van Oosterhout and Chybicki-Burczyk methods of estimating null allele frequencies for inbred populations

In spite of the usefulness of codominant markers in population genetics, the existence of null alleles raises challenging estimation issues in natural populations that are characterized by positive inbreeding coefficients (F > 0). Disregarding the possibility of F > 0 in a population will generally lead to overestimates of null allele frequencies. Conversely, estimates of inbreeding coefficients (F) may be strongly biased upwards (excess homozygotes), in the presence of nontrivial frequencies of null alleles. An algorithm has been presented for the estimation of null allele frequencies in inbred populations (van Oosterhout method), using external estimates of the F-statistics. The goal of this study is to introduce a modification of this method and to provide a formal comparison with an alternative likelihood-based method (Chybicki-Burczyk). Using simulated data, we illustrate the strengths and limitations of these competing methods. Under most circumstances, the likelihood method is preferable, but for highly inbred organisms, a modified van Oosterhout method offers some advantages.