Audit quality: asking the right questions

I am, of course, delighted to read the headline Audit Committee Chairs believe audit is improving.  The Financial Reporting Council’s press notice about this year’s FRC Audit Committee Chairs (“ACC”) Survey says “ACCs scored their auditors highly across all questions. There was also evidence of improvement in all categories with the highest being that of independence and objectivity. The lowest overall scores, for a second year, were for questions on professional scepticism and the auditor’s response to regulatory oversight suggesting there is still some work for firms to do in this area.”

But how much does the survey really tell us about audit quality?  Not enough, in my view.

By asking about focus, approach, risk assessment, materiality, resources, scepticism, independence, communication and interaction,  the questionnaire effectively imposes its own answer to the question “What is a good audit?”  Each of those factors may be an important dimension of quality, but it seems to me that we’re limiting the survey to confirming what we think we already know, rather than risking uncovering a more challenging definition of audit quality.

The survey did go further this year in asking ACCs what other criteria they use to assess audit quality outside of the headings in the questionnaire.  But I’d also like to know which components of audit quality they regard as the most important.  Also, given there is an inherent tension between some of the factors, which way would the ACCs sway if they had to choose between factors?

I’d like to try to structure the questionnaire so that ACCs rate the components of quality in order of importance or are required to prioritise apparently competing factors.  As an example, they could be asked to choose between pairs of factors such as independence versus continuity; sector specialism versus breadth of experience; high level of coverage versus timeliness of identification of issues – both in terms of how important those factors are to quality and then in terms of which they felt their auditor exhibited more strongly.

Of course, given many ACCs will have completed this questionnaire before, they will be aware of what aspects of quality the previous surveys have enquired about.  So the questionnaire needs to actively probe to identify components of audit  quality other than the old comfortable favourites.  These other components might include firm sector specialism and individual (partner) sector specialism but ought also to include some of the aspects of quality that we all find it hard to acknowledge and talk about, for instance empathy, emotional resilience, a sense of proportion.

It would be particularly interesting to know to what extent “perceived calibre of individual” – one of the alternative components of audit quality volunteered by the ACCs – is made up of observable characteristics and behaviours as opposed to subjective impressions.  This could be illuminated by asking the ACCs to score individuals from the audit team across a range of observables and then to score individuals overall for “perceived calibre”.

Finally, while I don’t want to be cynical, I note that more than 10% of the respondents had changed auditor as a result of a competitive tender within the previous 12 months.  It’s reasonable to expect that ACCs who have recently selected a new auditor and, of course, done so on the basis of quality, will be satisfied with the quality of their auditor.

The debate around indicators of audit quality is largely being led by regulators, who have a vested interest in keeping the emphasis on indicators  that can be quantitatively measured – even where a measure may have a non-linear relationship to audit quality.  Indeed, the US regulator, the Public Company Accounting Oversight Board (PCAOB), says much of the information it measures might contain flaws, but it is still worth producing it in order to promote debate about the quality of the audit.  And I agree – it’s no bad thing for ACCs have market-level, firm-level and engagement-level data that they can use to frame a discussion with an audit engagement partner as to how their firm will deliver a quality audit.

It would be a mistake, however, to give too much weight to the things we can measure at the expense of broader and deeper insight into the nature of and drivers of audit quality.  And it could harm quality if the drive to achieve attractive audit quality metrics were to become a disincentive to audit firms to innovate.

Perhaps most important of all, though, the ACCs are just one of many stakeholder groups.  The ACCs represent those with direct equity investments in the company but there is a much wider group of “investors” who are exposed to the risk of loss of capital in public companies – society as whole.  The FRC recognises this – at least, that’s what I infer from their introduction of the concept of the “objective, reasonable and informed third party” in the consultation document Enhancing Confidence in Audit.  How does that objective, reasonable and informed third party judge the quality of an audit or the calibre of an auditor?  We need to ask a lot more questions – a lot more of the right questions – if we’re going to find out.



False Assurance: how ICAEW’s new film invites us to think about human factors in auditing

I had the privilege of being invited to speak at ICAEW’s premiere of False Assurance “An exciting film drama created to provoke discussions on how accountants, auditors and company directors should act when faced with difficult situations.”  Here is a slightly extended version of the speech I gave.

It is a privilege and a pleasure to be given the opportunity to share some personal reflections on False Assurance. It is really excellent – I love it – and I think that Duncan and the Professional Standards team should feel really proud not only of the content but also of the production values.

I first watched the film four weeks ago and I have to say I had an almost visceral reaction to it.  It was very uncomfortable to watch.  The scenarios were so plausible.  Also, I have to confess to being a bit of an aficionado of those classic public information films from the 1970s – you know, the ones that dole out disfigurement and death to drink-drivers, children trespassing on railway lines and women running in the street.  This film is like one of those: it builds up a sensation of mounting dread.  You know something bad is going to happen to these nice people, but what? And to whom?  Here, the answer might as well be: everything that possibly could, and to everyone.

That’s the beauty of it.   The scenario that is developed is one in which there are a number of factors that all contribute to corruption and fraud going undetected for some time.  None of the characters are unbelievably good, and none unbelievably bad – all of them succumb to pressures that we see in real life in one form or another.

I’ve worked in both Professional Practice and Audit Quality for a number of years now, so I’m particularly interested in how the auditors in the film behave and why – and how we should respond to that. In my experience, audit firms tend to take what we might call a “person approach” to dealing with quality issues. Poor decisions are seen as arising primarily from flaws in an individual person’s mental processes such as forgetfulness, inattention, poor motivation, carelessness, negligence, and recklessness.

When we try to eliminate individual weaknesses, the sort of measures we put in place are directed mainly at reducing unwanted variability in human behaviour.  It’s a regulatory compliance approach.  So we ask for more procedures and more checklists; we design disciplinary measures that appeal to fear – if not fear of litigation, fear of sanctions – naming, blaming, shaming and, these days, fining.  There’s an uncomfortable implied moral subtext to this approach in that it seems to inherently assume that bad things happen to bad people.

The film instead highlights the value of what we might call the “system model”.  In this model for understanding failure, human errors are seen as inevitable products of systemic weakness. We can’t change the human condition, so we have to change the conditions in which humans operate.

An audit team is a system of defensive layers – like the “Swiss Cheese” model proposed by James Reason, Professor of Psychology at Manchester University[1]. There are holes continually opening, shutting, and shifting in each slice of cheese. The presence of holes in any one “slice” does not normally cause a bad outcome. Usually, this can happen only when the holes in many layers momentarily line up, as in the film, where there are multiple opportunities for the fraud to be identified, and multiple failures – some individually minor – are required for it to go undetected.

In the film, you see the individual active failures – poor decisions made by each character – but you also observe the latent conditions that increase the possibility of poor decision-making.   Professor Reason uses the analogy of mosquitos for active failures, versus mosquito breeding grounds for latent conditions.  You can swat all the mosquitos you want, but if you don’t drain the swamp, they’ll keep coming – and you’ll have to keep swatting.  In the film, these swampy conditions include overwork, time pressure, a culture of rewarding strong relationships with client executives and the sort of hierarchy where none of the senior people seem to seriously entertain the possibility that the concerns of more junior members of the team might ever amount to much.

I want to make particular mention too of the way the CFO in the film plays on institutional sexism by criticising the female audit partner for “interrogating” him.  Research at Stanford University is ongoing but shows that women receive 2.5 times the amount of feedback that men do about aggressive communication styles. Another study found that negative personality criticism showed up thirty times as frequently in appraisals of women as in appraisals of men, though the population selected for that review had all been considered to be equally strong performers. The women were much more likely to be described as “abrasive”, “coming on strong”, “strident” or “aggressive”.

So one of the latent conditions in our profession is a particular disadvantage to women.  It seems women are much more likely to be criticised for the robust challenge, persistence and scepticism that would be praised in a male colleague.

So why does the idea of personal responsibility for failure persist? Well for one, we tend to prefer it – it resonates with our ideas of responsibility and accountability.  It’s much easier to sanction a person than to change the culture that fostered that person’s mistakes. And sadly, we’re all human and we find blaming individuals emotionally satisfying.

We also like the idea of single causes because we are afraid of risks we can’t control. Sidney Dekker, Professor of Safety Science at Griffith University, Australia says “The failures which emerge from normal everyday systems interactions question what ‘‘normal’’ is. It is this threat to our epistemological and moral accountancy that makes accidents of this kind so problematic. We would much rather see failure as something that can be traced back to a single source, a single individual. When this is not possible in the assignation of blame and responsibility, accuracy or fairness matters less than closing or reducing the anxiety associated with having no cause at all. In the Western scheme of things, being afraid is worse than being wrong, being fair is less important than being appeased. Finding a scapegoat is a small price to pay to maintain the illusion that we actually know how a world full of risk works.”[2]

So what do we do?  We need a reporting culture and we need safe spaces to analyse what is reported.  Without a detailed analysis of mishaps, incidents and near misses we have no way of uncovering recurrent error traps or of knowing where the “edge” is until we fall over it.  Both Reason and Dekker refer in their work to “Just Culture”, in particular restorative Just Culture rather than retributive Just Culture.

A Just Culture is one with a vital “collective understanding of where the line should be drawn between blameless and blameworthy actions” (Reason).  It’s a culture that learns and prevents by asking why it made sense at the time for highly intelligent, highly educated, highly trained and highly regulated professionals to do what they did. How many audit firms are really asking that question about failures?

I am hoping therefore that no-one is going to leave the film thinking that the next step is to warn audit partners about that Bad Things will happen to them if they don’t get written representations about related parties.  Let’s instead look for our swamps and set about draining them.  In the context of the film that might include:

  • Looking at how complaints to “relationship” partners about audit team members are handled
  • An honest look at whether and how individual patronage plays a part in promotion processes and the allocation of valuable work within firms
  • Examining the trends/differences in language used in performance appraisals to describe certain behaviours when shown by women or men.

Those are just a few suggestions – there are many other areas to consider.

As Professor Reason says “Perhaps the most important distinguishing feature of high reliability organisations is their collective preoccupation with the possibility of failure. They expect to make errors and train their workforce to recognise and recover them. They continually rehearse familiar scenarios of failure and strive hard to imagine novel ones. Instead of isolating failures, they generalise them. Instead of making local repairs, they look for system reforms.”

I would like to say that I work in a High Reliability Organisation.  But are we audit firms prepared to turn that unflinching scrutiny on ourselves?

[1] Human error, models and management (Reason) BMJ. 2000 March 18; 320 (7237): 768–770

[2] Cognitive engineering and the moral theology and witchcraft of cause (Dekker, Nyce) 2011

%d bloggers like this: