21 Jan 2015

Don’t Trust the Experts: Not in Physics, Certainly Not in Economics

Austrian School, Richard Feynman 28 Comments

Tyler Cowen has an amusing post (with which I largely agree) where he wonders how many economists truly understand various important results in economics. One might walk away from Tyler’s post thinking, “Human knowledge has become so specialized that we can really only trust a handful of world experts in each little niche.”

But that’s a very reckless strategy, because sometimes the experts in a very narrow field paint themselves into a corner, and they don’t even realize it because they’re the only ones refereeing each other’s articles in the academic journals. As a fan of Austrian economics, I am sure that this has happened with the more elite and prestigious economics that the “leaders in the field” use at central banks and in the most highly ranked universities.

Now when it comes to economics (or climate science, another obvious example of what I mean), there are political and cultural factors that might explain a lot of the bias, and why people disagree so strongly about something seemingly objective. Yet the problem is much more general, reflecting human nature rather than mere politics.

I saw a great example tonight when I was reading physicist Richard Feynman’s bestselling memoirs with my son. Feynman was telling the story of when he had come up with a new approach to modeling the behavior of subatomic particles while attending a conference in Rochester, but he had thought it was a dead end because at the time there was a well-known experimental result that didn’t fit. Later on, the experimental physicists at his own school told Feynman that new experiments had come in, challenging the old consensus, and eventually he realized he had discovered a new law of nature. Here’s what happened next:

I went to Professor Bacher and told him about our success, and he said, “Yes, you come out and say that the neutron-proton coupling is V instead of T. Everybody used to think it was T. Where is the fundamental experiment that says it’s T? Why don’t you look at the early experiments and find out what was wrong with them?”

I went out and found the original article on the experiment that said the neutron-proton coupling is T, and I was shocked by something. I remembered reading that article once before (back in the days when I read every article in the Physical Review–it was small enough). And I remembered, when I saw this article again, looking at that curve and thinking, “That doesn’t prove anything!”

You see, it depended on one or two points at the very edge of the range of the data, and there’s a principle that a point on the edge of the range of the data–the last point–isn’t very good, because if it was, they’d have another point further along. And I had realized that the whole idea that neutron-proton coupling is T was based on the last point, which wasn’t very good, and therefore it’s not proved. I remember noticing that!

And when I became interested in beta decay, directly, I read all these reports by the “beta-decay experts,” which said it’s T. I never looked at the original data; I only read those reports, like a dope. Had I been a good physicist, when I thought of the original idea back at the Rochester Conference I would have immediately looked up “how strong do we know it’s T?”–that would have been the sensible thing to do. I would have recognized right away that I had already noticed it wasn’t satisfactorily proved.

Since then I never pay any attention to anything by “experts.” I calculate everything myself….I’ll never make that mistake again, reading the experts’ opinions. (Surely You’re Joking, Mr. Feynman, pp. 254-255)

Now to be sure, if at any moment you have to make a guess, it’s generally reasonable to take the “expert opinion” of the people in the field. But my point is that very often, these experts overrate the strength of the evidence in favor of their view. I know for sure this happens all the time in economics, and I’ve seen plenty of examples of it in other fields too.

28 Responses to “Don’t Trust the Experts: Not in Physics, Certainly Not in Economics”

  1. Tel says:

    In the publication industry there’s a strong negative opinion about negative results. For this reason you will never read “peer reviewed” papers about an experiment that doesn’t really give any real conclusion.

    You won’t read anyone say, “Yeah, if you try it this way you get a lot of noise, and the sensor gums up, which pretty much swamps the measurements; after 12 months work we discovered this problem, so we wrote about what we did and now we will go try something else for a while.”

    The reason you won’t read it is not that such a thing never happens, on the contrary, it happens all the time. The reason is those sort of results just don’t get published.

    • Harold says:

      Bang on Tel. This is why it took a long time for cold fusion to be overturned. Those groups that saw something that reinforced the original work rushed to publication. Those that could not reproduce the results thought they were doing something wrong, so held off publication until they had done lots more work.

    • Ken P says:

      The first time I read Hayek, I felt like I was reading something by Feynman.

      • Ken P says:

        The Feynman comment was intended to be in reference to the blog post not Tel’s comment.

        But in reference to Tel’s comment, I remember listening to a podcast, I believe on Econtalk where the person was suggesting journals be started for research that did not work.

  2. Harold says:

    The role of H. Pylori in stomach ulcers is very similar. All the experts knew that no bacteria could grow in the stomach, as it was too acidic. This had become received wisdom. Then came Marshall and Warren, who identified bacteria in the stomach. Faced with scepticism, Marshall drank a beaker of H. Pylori culture, and became ill with stomach irritation and vomiting,. An endoscopy days later showed inflammation of the stomach. It turns out that the knowledge that bacteria could not live in the stomach was based on very little data. In fact several researchers in the 19th century had seen bacteria in the stomach, but could not culture them at the time. Marshall and Warren received a well deserved Nobel prize for their efforts.

    So what is the lesson here? Is it that experts overrate the strength of th evidence? Not really; it is beware of received wisdom, and it underlines the importance of checking where assumptions come from. After a relatively small amount of research into the question of whether bacteria could live in the stomach, it became very clear that the received wisdom was not based on very much, and opinion was overturned. This is always the case when people look into areas that have been accepted as fact. The fact will be either re-inforced or overturned. Plate tectonics is another case.

    Was this the case with neutron-proton coupling? Yes indeed. It was accepted that it was T, but it took very little actual research to overturn that view, since it was based on very weak data. Is this the case with climate change? Certainly not. There has been huge amounts of research into this and the received wisdom has not been changed. In fact, received wisdom used to be that man could not do anything to change the climate. On further examination, that turned out to be based on very little evidence, so the opinion was overturned.

    This is an important point. All examples where accepted facts have been overturned that I can think of are of this nature. The accepted wisdom turns out to be based on little evidence, and is overturned when people start to look deeper. I can think of no case where a large research effort led us away from the truth. Any examples would be welcome.

    • Bob Murphy says:

      Harold wrote:

      The accepted wisdom turns out to be based on little evidence, and is overturned when people start to look deeper. I can think of no case where a large research effort led us away from the truth. Any examples would be welcome.

      How about economics and psychiatry?

      • Harold says:

        Do you mean the whole of economics and psychiatry, or do you have specific examples from these fields? Psychiatry has had some awful blunders – lobotomies springs to mind – but that was based on very little evidence, and as soon as some proper research was done the practice was stopped. Economics has surely taken us in a general sense towards the truth and away from mercantalist ideas.

        • Yancey Ward says:

          Economics has surely taken us in a general sense towards the truth and away from mercantalist ideas.

          Simply hilarious. No, economics has dressed mercantilism up in finer clothing and sent it out on the street to pick up Johns.

          • Harold says:

            Seriously? You think our understanding was closer to the truth before, say, Adam Smith? Or Thomas Aquinas, who said it was immoral to raise prices in times of scarcity? When was this golden age of understanding that we have moved away from?

            • Levi Russell says:

              Aquinas’ point was a moral one, that much is obvious. Thankfully, his followers, the Late Scholastics, began the development of economics apart from ethics. That’s a big improvement.

              Adam Smith killed Mercantilism. Another improvement.

              Menger, Jevons, and Walras gave us the marginal revolution. Another huge improvement.

              Keynes dredged up Mercantilism, put a fancier veneer on it, and put macroeconomics behind quite a bit. His followers by and large continue to pretend this veneer fixes the underlying issues.

              Buchanan, Tullock, and others gave us Public Choice theory.

              Coase helped us understand transactions costs.

              Lucas tried to put some sense into macro with his microfoundations, but his point wasn’t taken far enough. We ended up with representative agents and DSGE models. Recently, Caballero, Laidler, and some others have begun to realize that Hayek, Garrison, Horwitz, and others were right all along, at least about business cycles. If the “Austrian economists” have a comparative advantage in anything, it’s capital and interest theory. I hope macroeconomists continue to incorporate their insights and leave Keynes and scientism by the wayside. Of course, microeconomists should also reject scientism as well.

              So you and Yancey are just speaking too broadly. Some areas of economics have seen massive improvements while others have improved and regressed to varying degrees.

          • Tel says:

            The Johns seem happy enough. No victim here.

      • khodge says:

        When I was in college I occasionally read Psychology Today. Not many years after that I picked up a copy in the local library…it had turned into high-class porn.

        Psychology is all about politics and virtually no science now. It started with the gay rights movement convincing psychologists that homosexuality was a choice and not a disease therefore it should not be included among the ICD codes. When the disciplined had been thoroughly captured they could get the establishment to accept whatever consensus could be reached without reference to any science.

        There are some areas where I see the same thing happening in Economics where narrative rules and the surveys can be twisted to whatever supports the liberal line, such as: minimum wage, income inequality, and (per the next post Correcting Hartmann) taxes and the business cycle.

        • Harold says:

          Don’t confuse psychology with psychiatry. Also I don’t think the gay rights movement believes homosexuality is a choice. Not counting it as a disease is an advance in my book.

          • khodge says:

            They do not consider it a choice. That was back in the 70’s and 80’s. As I understand it, there is no definitive science explaining it (or, more precisely, if you’re a politician and state the best science they’ll roast you and turn the entire establishment press against you).

    • Harold says:

      Just noticed -the p in H. pylori should not be capital. Blame it on autocorrect.

  3. Onlooker from Troy says:

    The whole area of nutrition is yet another prime example. The notion that saturated fat (especially evil red meat) is bad for us, high serum cholesterol caused heart disease, etc. was entrenched by the “experts” and just can’t be killed now, regardless of the contrary evidence and good science done to refute it.

    • Harold says:

      Again, the whole of nutrition science has not taken us away from the truth. Science has identified a correlation between high consumption of saturated fats and some health problems. That is not the same as saying all consumption of same is bad for us. The science is somewhat mixed, since this is a very complex area, but I doubt you would find a consensus among nutritionists that all consumption of red meat was bad. Do not mix up what the media says with what the science says. I suspect you would find agreement that consumption of high levels of saturated fat was bad for us.

      • Dean T. Sandin says:

        Actually, if you read Gary Taubes he’ll tell you that the German scientists studying nutrition before WWII did believe that carbohydrates were the chief dietary contributor to obesity and related ailments, and that this knowledge was lost/abandoned after the war, allowing the twentieth century scientific consensus of blaming dietary fat/cholesterol. If this is true, then yes, “the whole of nutrition science” has “taken us away from the truth.”

        But that probably doesn’t matter, because you totally changed your argument. At first you were just claiming that thinly supported but widely held scientific conclusions break down very fast when challenged. Then when people gave counter examples, your argument became that the scientific knowledge wasn’t better before it was bad. Totally different claim.

        • Harold says:

          “But that probably doesn’t matter, because you totally changed your argument…Then when people gave counter examples, your argument became that the scientific knowledge wasn’t better before it was bad.”

          Only because others cited entire disciplines as examples. I was asking for examples of real and extensive research leading to a position that was later shown to be an error. This is in contrast to positions that were accepted, but were actually based on flimsy evidence that was quickly revealed by some proper research. I have had economics, psychiatry and nutrition given as examples. I believe these are very poor examples since our understanding in all three areas has been one of general advancement, very much in line with my argument.

  4. Josiah says:

    Bob,

    I agree. It’s amazing how often I will look into the basis behind some commonly accepted claim, only to think “wait, you mean that’s it?”

  5. Josiah says:

    I will add, though: while I’ve often found that experts frequently overrate the strength of the evidence in favor of their views, my experience has also been that amateurs/outsides who claim to have overturned an expert consensus are almost always cranks.

  6. Major.Freedom says:

    Does this lesson include religion, or the Bible in particular? Serious question.

    Is it OK if I distrust every single person who tries to convince me of anything about religious doctrines or stories, even the experts? Serious question.

    Or am I supposed to be skeptical of scientists only, which is supposed to strengthen my trust of men of faith? Serious question.

    “Human nature”, etc.

    Doesn’t this post necessitate agnosticism? Serious question.

    • khodge says:

      Yes, it means religion. That is precisely why Bob puts his religious questions to us. He studies the scriptures with his friend; he examines the bible, he examines the commentaries; he then puts his observations to an audience that is largely either skeptical or coming at it from a total different angle: us.

      I think it is fair to read a bit of exaggeration in Feynman’s final paragraph: he never pays attention? ever? How about he reviews their work then checks the details? Wouldn’t you become rather detached from reality and the most current studies if you completely ignored the experts? Someone listening only to himself is far worse than someone relying totally on experts.

      • Major.Freedom says:

        “Someone listening only to himself is far worse than someone relying totally on experts.”

        That can’t be right. What about the experts themselves? Those who by definition listen to themselves and not the more dull witted masses?

        I don’t put one as above or below the other. When you’re smarter than others, you should listen to yourself, and when others are smarter than you, than you should listen to others.

        The dictum “It is far worse to listen only to yourself than to listen only to experts” only works for those who are not yet experts. It doesn’t work for experts.

        But this is neither here nor there. Bob is telling us to not trust experts. Well there are religious experts who tell me what to think about theological doctrines, so presumably I should not trust them, because they might have made an egregious mistake…like committing a mistranslation of Greek into Latin and English, which has altered the meaning of the original books of the Bible which today’s experts promote as God’s word.

        The reading I get from this post is to distrust science experts, but trust religious experts. I’ll continue to assume that is the likely truth, until I see a bunch of blog posts of Bob explicitly and directly telling his readers not to trust their pastors or any other religious experts, like he has in this blogpost. I’ve never seen such a post.

        • Harold says:

          “Bob is telling us to not trust experts.” I think he is saying “look at the evidence behind your assumptions.” Especially if some new evidence seems to contradict them. This is directly applicable to experts more than lay people, as they are in a position to check. So a biblical scholar may be able to check the translations of the original texts, whereas it is difficult for the lay person. An expert can assess the evidence for neutron-proton coupling, whereas it is difficult for the non-physicist.

          Where does that leave the non-expert? We may not be capable of assessing the original work, but we can look around to see if there has been much work looking into the question. A non-expert could check how much work had been done on neutron-proton coupling or stomach bacteria.

          So if a scientist or cleric suggests something that sounds wrong to you, you could in principle ask on what evidence he bases that opinion. If it is a translation of the bible or something, you could look around and see if that translation was widely accepted by other translators. If it is something weird like singe photons producing interference patterns, the lay person can check if this experimantal result has been replicated many times.

  7. Harold says:

    Possible source of examples – a list of vindicated maverick scientists.

    http://amasci.com/weird/vindac.html

    I find Robert Folk interesting – he discovered nannobacteria in rocks. Why were these not found before? He says ” and it has been standard microbiological dogma for fifty years that bacteria smaller than 0.2 micrometers cannot exist.” Althoguh the existence of these nannobacteria is not confirmed, so maybe the dogma is rigth in this case.

  8. Lio says:

    Ok. You emphasize that experts are sometimes wrong too, as ordinary people, but it doesn’t mean that they are wrong all the time neither or that they are wrong more often than ordinary people. We must not fall into the trap of thinking like this. It is not because someone is considered as an expert that what he says is surely false.

Leave a Reply to Harold

Cancel Reply