The value of stocks indices, and other assets, are examples of stochastic processes that drop and raise in unpredictable ways. In this paper, we discuss certain asymmetries in short term price movements that can not be attributed to a long term increasing trend. These empirical asymmetries predict that price drops in stock indices on a relatively short time scale are more common than the corresponding price raises, and we present several empirical examples of such asymmetries. Furthermore, a simple model is introduced with the aim of explaining these facts. The prime idea is to introduce occasional, short periods of dropping stock prices that are synchronized for all stocks of the index. These collective negative price movements are imagined to be triggered by external factors in our society that create fear for the future among the investors. In the model this is parameterized by a ``fear factor'' defining how often such events take place. It is demonstrated that such a simple fear factor model can reproduce several empirical facts concerning index asymmetries. It is also pointed out that in its simplest form, it has certain shortcomings.