401k's and company's being able to buy their own stock has been a game changer for the stock market. You can't compare the pre 1980 data to today's stock market. As long as money flows into the stock market via 401k's and companies buying their own stock the s&p 500 will continue to flow upwards. There will be panic selling but the market will recover since index funds have to use what money investor's give them no matter which way the market is going. There will be a time when that won't be the case since more people will start to retire and will start to sell but for now the flow is up- in my opinion.

Just because the S&P 500 has long periods where it is basically flat doesn't mean it isn't the best investment option for the non-professional investor.

The right two-part question to ask is:

a) can you know in advance whether or not the S&P 500 is going to be flat and can you know in advance for how long

b) can you pick, in advance, an investment or portfolio of investments that you can be more confident about will have a higher risk-adjusted rate of return for said period.

The natural answer is no. Firstly, you don't know in advance how long the index will be stagnant (if at all) or for how long and you cannot know with high certainty which investments will outperform on a risk-adjusted basis for an unknown period of time.