Asymptotic sampling formulae for $𝛬$-coalescents

Berestycki, Julien; Berestycki, Nathanaël; Limic, Vlada

Annales de l'I.H.P. Probabilités et statistiques, Tome 50 (2014) no. 3, pp. 715-731.

Nous présentons une méthode robuste qui permet de traduire des informations sur la vitesse de descente de l’infini d’un arbre généalogique en formules d’échantillonnages pour la population sous-jacente. Nous appliquons cette méthode au cas où la génélaogie est donnée par un $𝛬$-coalescent. Nous en déduisons une formule exacte pour le comportement asymptotique du spectre des fréquences alléliques et du nombre de sites de ségrégation, lorsque la taille de l’échantillon tend vers l’infini. Certains de ces résultats sont valides dans le cas général où le coalescent descend de l’infini, tandis que d’autres plus précis sont obtenus sous une hypothèse de variation régulière. Dans ce cas nous obtenons également des résultats, dont l’intérêt dépasse ce contexte, sur le temps auquel une mutation choisie uniformément au hasard est apparue. Il apparaît que cette quantité connaît une transition de phase autour de la valeur $\alpha =3/2$, où $\alpha$ est l’exposant de variation régulière.

We present a robust method which translates information on the speed of coming down from infinity of a genealogical tree into sampling formulae for the underlying population. We apply these results to population dynamics where the genealogy is given by a $𝛬$-coalescent. This allows us to derive an exact formula for the asymptotic behavior of the site and allele frequency spectrum and the number of segregating sites, as the sample size tends to $\infty$. Some of our results hold in the case of a general $𝛬$-coalescent that comes down from infinity, but we obtain more precise information under a regular variation assumption. In this case, we obtain results of independent interest for the time at which a mutation uniformly chosen at random was generated. This exhibits a phase transition at $\alpha =3/2$, where $\alpha \in (1,2)$ is the exponent of regular variation.

Classification : 60J25, 60F99, 92D25
Mots clés : $\varLambda $-coalescents, speed of coming down from infinity, exchangeable coalescents, sampling formulae, infinite allele model, genetic variation

[1] D. J. Aldous. Exchangeability and related topics. In École d'Eté de Probabilités de Saint-Flour XIII - 1983. Lecture Notes Math. 1117. Springer, Berlin, 1985. | MR | Zbl

[2] A.-L. Basdevant and C. Goldschmidt. Asymptotics of the allele frequency spectrum associated with the Bolthausen-Sznitman coalescent. Electron. J. Probab. 13 (2008) 486-512. | MR | Zbl

[3] J. Berestycki, N. Berestycki and V. Limic. The $𝛬$-coalescent speed of coming down from infinity. Ann. Probab. 38 (2010) 207-233. | MR | Zbl

[4] J. Berestycki, N. Berestycki and V. Limic. A small-time coupling between $𝛬$-coalescents and branching processes. Preprint, 2012.

[5] J. Berestycki, N. Berestycki and J. Schweinsberg. Small-time behavior of beta-coalescents. Ann. Inst. Henri Poincaré Probab. Stat. 44 (2008) 214-238. | Numdam | MR | Zbl

[6] J. Berestycki, N. Berestycki and J. Schweinsberg. Beta-coalescents and continuous stable random trees. Ann. Probab. 35 (2007) 1835-1887. | MR | Zbl

[7] N. Berestycki. Recent Progress in Coalescent Theory. Ensaios Matematicos 16. Sociedade Brasileira de Matemática, Rio de Janeiro, 2009. | MR | Zbl

[8] J. Bertoin. Random Fragmentation and Coagulation Processes. Cambridge Studies in Advanced Mathematics. Cambridge Univ. Press, Cambridge, 2006. | MR | Zbl

[9] J. Bertoin and J.-F. Le Gall. Stochastic flows associated to coalescent processes III: Limit theorems. Illinois J. Math. 50 (2006) 147-181. | MR | Zbl

[10] J.-S. Dhersin, F. Freund, A. Siri-Jegousse and L. Yuan. On the length of an external branch in the Beta-coalescent, 2012. Available at arXiv:1201.3983. | MR | Zbl

[11] P. Donnelly and T. Kurtz. Particle representations for measure-valued population models. Ann. Probab. 27 (1999) 166-205. | MR | Zbl

[12] R. Durrett. Probability Models for DNA Sequence Evolution. Springer, Berlin, 2002. | MR

[13] W. J. Ewens. The sampling theory of selectively neutral alleles. Theor. Pop. Biol. 3 (1972) 87-112. | MR | Zbl

[14] W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 2. Wiley, New York, 1971. | MR | Zbl

[15] N. Freeman. The number of non-singleton blocks in Lambda-coalescents with dust, 2011. Available at arXiv:1111.1660.

[16] A. Gnedin, B. Hansen and J. Pitman. Notes on the occupancy problem with infinitely many boxes: General asymptotics and power laws. Probab. Surv. 4 (2007) 146-171. | MR | Zbl

[17] G. Kersting. The asymptotic distribution of the length of Beta-coalescent trees. Ann. Appl. Probab. 22 (2012) 2086-2107. | MR | Zbl

[18] M. Kimura. The number of heterozygous nucleotide sites maintained in a finite population due to steady flux of mutations. Genetics 61 (1969) 893-903.

[19] J. F. C. Kingman. On the genealogies of large populations. J. Appl. Probab. 19 (1982) 27-43. | MR | Zbl

[20] V. Limic. On the speed of coming down from infinity for $𝛯$-coalescent processes. Electron. J. Probab. 15 (2010) 217-240. | MR | Zbl

[21] V. Limic. Genealogies of regular exchangeable coalescents with applications to sampling. Ann. Inst. Henri Poincaré Probab. Statist. 48 (2012) 706-720. | Numdam | MR | Zbl

[22] V. Limic. Processus de Coalescence et Marches Aléatoires Renforcées : Un guide à travers martingales et couplage. Habilitation thesis (in French and English), 2011. Available at http://www.latp.univ-mrs.fr/~vlada/habi.html.

[23] M. Möhle. On the number of segregating sites for populations with large family sizes. Adv. in Appl. Probab. 38 (2006) 750-767. | MR | Zbl

[24] M. Möhle and S. Sagitov. A classification of coalescent processes for haploid exchangeable population models. Ann Probab. 29 (2001) 1547-1562. | MR | Zbl

[25] J. Pitman. Coalescents with multiple collisions. Ann. Probab. 27 (1999) 1870-1902. | MR | Zbl

[26] J. Pitman. Combinatorial stochastic processes. In École d'Eté de Probabilités de Saint-Flour XXXII - 2002. Lecture Notes Math. 1875. Springer, Berlin, 2006. | MR | Zbl

[27] S. Sagitov. The general coalescent with asynchronous mergers of ancestral lines. J. Appl. Probab. 36 (1999) 1116-1125. | MR | Zbl

[28] J. Schweinsberg. A necessary and sufficient condition for the $𝛬$-coalescent to come down from infinity. Electron. Commun. Probab. 5 (2000) 1-11. | MR | Zbl

[29] J. Schweinsberg. The number of small blocks in exchangeable random partitions. ALEA Lat. Am. J. Probab. Math. Stat. 7 (2010) 217-242. | MR | Zbl