Parliamentary committees are not especially noted for entertainment, but the November Treasury Select Committee hearing on the Bank of England’s Inflation Report is a refreshing exception. The fun starts on p. 30 of the transcript of the hearings with Steve Baker MP and Bank of England Governor Mark Carney light-heartedly jousting with each other.
Steve begins by asking Dr. Carney if the Bank is all model-driven. To quote from the transcript:
Dr Carney: No. If we were all model driven, then you would not need an MPC.
Q81 Steve Baker: All right. But we do have plenty of models floating around.
Dr Carney: I presume you feel we do need an MPC, Mr Baker?
Steve Baker: I think you know I think we don’t.
Dr Carney: I just thought we would get that read into the record.
[KD: First goal to Dr. Carney, but looks to me like it went into the wrong net.]Steve Baker: I want to turn to a criticism by Chris Giles in The Financial Times of the model for labour market slack, which called it a nonsense. If I may I will just share a couple of quotes with you. He said that, according to a chart in the inflation report, the average-hours gap hit a standard deviation of -6, and this is something we would expect to happen once in 254 million years. He also said that the Bank of England is again implying the recent recession, as far as labour market participation is concerned, was worse than any moment in 800 times the period in which homo sapiens have walked on the earth. How will the Bank reply to a criticism as strident as this one?
[KD: The article referred to is Chris Giles, “Money Supply: Why the BoE is talking nonsense”, Nov 17 2014: http://ftalphaville.ft.com/2014/11/17/2045002/moneysupply-why-the-boe-is-talking-nonsense/#]Dr Carney: Since you asked, let me reply objectively. Calculations such as that presume that there is a normal distribution around the equilibrium rate. Let me make it clear. First off, what is the point of the chart? The chart is to show a deviation relative to historic averages. It is an illustrative chart that serves the purpose of showing where the slack is relative to average equilibrium rates, just to give a sense of relative degrees of slack. That is the first point. The second point is that the calculation erroneously, perhaps on purpose to make the point but erroneously, assumes that there is a normal distribution around that equilibrium rate. So in other words to say that there is a normal distribution of unemployment outcomes around a medium-term equilibrium rate of 5.5%. So it is just as likely that something would be down in the twos as it would be up in the eights. Well, who really believes that? Certainly not the MPC and I suspect not the author of that article. It also ignores that the period of time was during the great moderation for all of these variables as well, so it is a relatively short period. These are not normal distributions. You would not expect them. You would expect a skew with quite a fat tail. So using normal calculations to extrapolate from a chart that is there for illustrative purposes is—I will not apply an adjective to it—misleading and I am not sure it is a productive use of our time.
Q82 Steve Baker: That is a fantastic answer. I am much encouraged by it, because it does seem to me it has been known for a long time that it is not reasonable to use normal distributions to model market events and yet so much mathematical economics is based on it.
[KD: Carney’s is an excellent answer: one should not “read in” a normal distribution to this chart, and the Bank explicitly rejects normality in this context.Slight issue, however: didn’t the Bank’s economists use the normality assumption to represent the noise processes in the models they used to generate the chart? I am sure they did. One wonders how the charts would look if they used more suitable noise processes instead? And just how robust is the chart to the modelling assumptions on which it is based?]
Dr Carney: People do it because it is simple—it is the one thing they understand—and then they apply it without thinking, which is not what the MPC does.
Steve Baker: That is great. I can move on quickly. But I will just say congratulations to the Bank on deciding to commission anti-orthodox research because I think this is going to be critical to drilling into some of these problems.
Dr Carney: Thank you.
[KD: Incredulous chair then intervenes.]Q83 Chair: To be clear, the conclusion that we should draw from this is that we should look at all economic models with a very high degree of scepticism indeed.
Dr Carney: Absolutely.
[KD: So you heard it from the horse’s mouth: don’t trust those any of those damn models. Still incredulous, the chair then intervenes again to seek confirmation of what he has just heard.]Chair: Can I just add that it is an astonishing conclusion? I do not want to cut into Steve Baker’s questions, but is that the right conclusion?
Dr Carney: Absolutely. Models are tools. You should use multiple ones. You have to have judgment, you have to understand how the models work and particularly, if I may underscore, dynamic stochastic general equilibrium forecasting models, which are the workhorse models of central banks. What they are useful for is looking at the dynamics around shocks in the short term. What they are not useful for is the dynamics further out where—
[KD: Dr. Carney reiterates the point so there can be no confusion about it. So let me pull his points together: (1) He “absolutely” agrees that “we should look at all economic models with a very high degree of scepticism.” (2) He suggests “You should use multiple [models]”, presumably to safeguard against model risk, i.e., the risks that any individual model might be wrong. (3) He endorses one particular – and controversial – class of models, Dynamic Stochastic General Equilibrium (DSGE) models as the “workhorse models” of central banks, whilst acknowledging that they are of no use for longer-term forecasting or policy projections.I certainly agree that none of the models is of any longer-term term use, but what I don’t understand is how (1), (2) and (3) fit together. In particular, if we are to be skeptical of all models, then why should we rely on one particular and highly controversial, if fashionable, class of models, never mind – and perhaps I should say, especially – when that class of models is regarded as the central banks’ workhorse. After all, the models’ forecast performance hasn’t been very good, has it?
The discussion then goes from the ridiculous to the sublime:]
Chair: I am just thinking about all those economists out there whose jobs have been put at risk.
Dr Carney: No, we have enhanced their jobs to further improve DSG models.
Steve Baker: We are all Austrians now.
[Laughter.] [A little later, Steve asks Sir Jon Cunliffe about the risk models used by banks.]Q84 Steve Baker: Sir Jon, before I move too much further down this path, can I ask you what would be the implications for financial stability and bank capital if risk modelling moved away from using normal distributions?
Sir Jon Cunliffe: Maybe I will answer the question another way. It is because of some of the risks around modelling, the risk-weighted approach within bank capital, that we brought forward our proposals on the leverage ratio. So you have to look at bank capital through a number of lenses. One way of doing is to have a standardised risk model for everyone and there is a standardised approach and it works on, if you like, data for everybody that does not suit any particular institution and the bigger institutions run their own models, which tend to have these risks in them. Then you have a leverage ratio that is not risk-weighted, and therefore takes no account of these models, and that forms a check. So with banks, the best way to look at their capital is through a number of different lenses.
[KD: Sir Humphrey is clearly a very good civil servant: he responds to the question by offering to answer it in a different way, but does not actually answer it. The answer is that we do not use a non-normal distribution because doing so would lead to higher capital requirements but that would never do as the banks would not be happy with it: they would then lobby like crazy and we can’t have that. Instead, he evades the question and says that there are different approaches with pros and cons etc. etc. – straight out of “Yes, Minister”.However, notwithstanding that Sir Jon didn’t answer the question on the dangers of the normal distribution, I would also ask him a number of other (im)pertinent questions relating to bad practices in bank risk management and bank risk regulation:
1. Why does the Bank continue to allow banks to use the discredited Value-at-Risk (or VaR) risk measure to help determine their regulatory capital requirements, a measure which is known to grossly under-estimate banks true risk exposures?
The answer, of course, is obvious: the banks are allowed to use the VaR risk measure because it grossly under-estimates their exposures and no-one in the regulatory system is willing to stand up to the banks on this issue.
2. Given the abundant evidence – much of it published by the Bank itself – that complex risk-models have much worse forecast performance than simple models (such as those based on leverage ratios), then why does the Bank continue to allow banks to use complex and effectively useless risk models to determine their regulatory capital requirements?
I would put it to him that the answer is the same as the answer to the previous question.
3. Why does the Bank continue to rely on regulatory stress tests in view of their record of repeated failure to identify the build-up of subsequently important stress events? Or, put it differently, can the Bank identify even a single instance where a regulatory stress test correctly identified a subsequent major problem?
Answer: The Northern Rock ‘war game’. But even that stress test turned out to be of no use at all, because none of the UK regulatory authorities did anything to act on it.
In the meantime, perhaps I can interest readers in my Cato Institute Policy Analysis “Math Gone Mad”, which provides a deeper – if not exactly exhaustive but certainly exhausting – analysis of these issues:
http://www.cato.org/publications/policy-analysis/math-gone-mad
1. http://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/treasury-committee/bank-of-england-inflation-report-hearings-session-201415/oral/15826.pdf
Carney and Co. are so dire it made me laugh out loud. In despair. I am ever more convinced of my ‘Theory of the Great and the Good’ which states that more you get to know them the less they are.
We’re screwed. Aren’t we?
It seems that the more information at the Banks fingertips the less accurate their conclusions.
The proof of the pudding and all that