On the relation between the instability of ESS in discrete dynamics and segregation distortion.
There is a simple correspondence between discrete dynamical systems associated with evolutionary game dynamics and general locus multiallele selection models with non-Mendelian segregation. When interpreted properly the payoff matrix has two components, a fitness matrix component and a segregation matrix component. The presence of segregation distortion which corresponds to a non-symmetric payoff matrix, is a source of instability. With non-symmetric payoff an ESS does not usually correspond to a stable equilibrium. It is always externally stable but does not necessarily have an internally stable equilibrium.
Liberman
U
U
Statistics Department, Tel-Aviv University, Israel.
D000483:Alleles; D005075:Biological Evolution; D008433:Mathematics; D008957:Models, Genetic