I just heard an amazing interview with Barry Ritholtz from September 1, 2005. At the time, he was chief market strategist for Maxim Group, but now runs his own hedge fund. Among many great ideas and concepts on applied economics, he provided an excellent explanation for herd mentality.
"It is better to fail conventionally than to succeed unconventionally in many instances." -- Keynes
Wall Street is populated with many people who come from similar backgrounds, have similar training, and work in a similar environment. They are in a natural herd, and the herd mentality comes with the territory, because it's about survival. To complete the analogy, who wants to be the stray gazelle that gets picked off by the predators? Therefore, analysts, strategists, and economists keep their predictions within a range, so as to avoid become the outlier to be removed. There is a psychological tendency not to fight the flow, and bend into peer pressure. He views Wall Street research as no longer a real profit center, outside of [investment] banking.
That gives us outside amateurs a certain advantage of having a unique background and point of view. Most of the guys on financial news networks are mostly just reading what is handed to them, and many of them get their news from similar sources. It's kinda like the AM radio talk show guys I used to listen to religiously, I don't get the time to listen to them so much any more. But when I could listen all day, it was show after show of the same "news" & topics all day long. It's like a list of talking points was printed early in the morning, and they all read from the same page for their particular show.
The current convention is usually not right, that's the mantra of a typical contrarian investor. The trick is figuring out when the rest of the herd will realize that, and then keeping a step ahead. I don't think it's really that difficult for the average amateur. There are so many sources and tools out there for each of us to develop our own investing & trading strategies. As long as one can follow what he/she learns and knows instead of following the crowd, anyone can succeed.