By applying inverse statistics to financial data it has recently been found from empirical studies that indices exhibit a pronounced gain-loss asymmetry [M.H. Jensen, Phys. Rev. Lett. 83 (1999) 76; I. Simonsen, M.H. Jensen, A. Johansen, Eur. Phys. J. B 27 (2002) 583; M.H. Jensen, A. Johansen, I. Simonsen, Physica A 324 (2003) 338]. This gain-loss asymmetry appears to have some similarities with the stylized fact leverage effect and we investigate if they could be of same origin. For this purpose we introduce the Frustration Governed Market model which includes correlations in time between a model index and its individual stocks. It is shown that the model reproduces very well the empirical findings with respect to gain-loss asymmetry and leverage. In special cases, however, the model may produce leverage without a pronounced gain-loss asymmetry.