Audit quality: asking the right questions

I am, of course, delighted to read the headline Audit Committee Chairs believe audit is improving.  The Financial Reporting Council’s press notice about this year’s FRC Audit Committee Chairs (“ACC”) Survey says “ACCs scored their auditors highly across all questions. There was also evidence of improvement in all categories with the highest being that of independence and objectivity. The lowest overall scores, for a second year, were for questions on professional scepticism and the auditor’s response to regulatory oversight suggesting there is still some work for firms to do in this area.”

But how much does the survey really tell us about audit quality?  Not enough, in my view.

By asking about focus, approach, risk assessment, materiality, resources, scepticism, independence, communication and interaction,  the questionnaire effectively imposes its own answer to the question “What is a good audit?”  Each of those factors may be an important dimension of quality, but it seems to me that we’re limiting the survey to confirming what we think we already know, rather than risking uncovering a more challenging definition of audit quality.

The survey did go further this year in asking ACCs what other criteria they use to assess audit quality outside of the headings in the questionnaire.  But I’d also like to know which components of audit quality they regard as the most important.  Also, given there is an inherent tension between some of the factors, which way would the ACCs sway if they had to choose between factors?

I’d like to try to structure the questionnaire so that ACCs rate the components of quality in order of importance or are required to prioritise apparently competing factors.  As an example, they could be asked to choose between pairs of factors such as independence versus continuity; sector specialism versus breadth of experience; high level of coverage versus timeliness of identification of issues – both in terms of how important those factors are to quality and then in terms of which they felt their auditor exhibited more strongly.

Of course, given many ACCs will have completed this questionnaire before, they will be aware of what aspects of quality the previous surveys have enquired about.  So the questionnaire needs to actively probe to identify components of audit  quality other than the old comfortable favourites.  These other components might include firm sector specialism and individual (partner) sector specialism but ought also to include some of the aspects of quality that we all find it hard to acknowledge and talk about, for instance empathy, emotional resilience, a sense of proportion.

It would be particularly interesting to know to what extent “perceived calibre of individual” – one of the alternative components of audit quality volunteered by the ACCs – is made up of observable characteristics and behaviours as opposed to subjective impressions.  This could be illuminated by asking the ACCs to score individuals from the audit team across a range of observables and then to score individuals overall for “perceived calibre”.

Finally, while I don’t want to be cynical, I note that more than 10% of the respondents had changed auditor as a result of a competitive tender within the previous 12 months.  It’s reasonable to expect that ACCs who have recently selected a new auditor and, of course, done so on the basis of quality, will be satisfied with the quality of their auditor.

The debate around indicators of audit quality is largely being led by regulators, who have a vested interest in keeping the emphasis on indicators  that can be quantitatively measured – even where a measure may have a non-linear relationship to audit quality.  Indeed, the US regulator, the Public Company Accounting Oversight Board (PCAOB), says much of the information it measures might contain flaws, but it is still worth producing it in order to promote debate about the quality of the audit.  And I agree – it’s no bad thing for ACCs have market-level, firm-level and engagement-level data that they can use to frame a discussion with an audit engagement partner as to how their firm will deliver a quality audit.

It would be a mistake, however, to give too much weight to the things we can measure at the expense of broader and deeper insight into the nature of and drivers of audit quality.  And it could harm quality if the drive to achieve attractive audit quality metrics were to become a disincentive to audit firms to innovate.

Perhaps most important of all, though, the ACCs are just one of many stakeholder groups.  The ACCs represent those with direct equity investments in the company but there is a much wider group of “investors” who are exposed to the risk of loss of capital in public companies – society as whole.  The FRC recognises this – at least, that’s what I infer from their introduction of the concept of the “objective, reasonable and informed third party” in the consultation document Enhancing Confidence in Audit.  How does that objective, reasonable and informed third party judge the quality of an audit or the calibre of an auditor?  We need to ask a lot more questions – a lot more of the right questions – if we’re going to find out.



False Assurance: how ICAEW’s new film invites us to think about human factors in auditing

I had the privilege of being invited to speak at ICAEW’s premiere of False Assurance “An exciting film drama created to provoke discussions on how accountants, auditors and company directors should act when faced with difficult situations.”  Here is a slightly extended version of the speech I gave.

It is a privilege and a pleasure to be given the opportunity to share some personal reflections on False Assurance. It is really excellent – I love it – and I think that Duncan and the Professional Standards team should feel really proud not only of the content but also of the production values.

I first watched the film four weeks ago and I have to say I had an almost visceral reaction to it.  It was very uncomfortable to watch.  The scenarios were so plausible.  Also, I have to confess to being a bit of an aficionado of those classic public information films from the 1970s – you know, the ones that dole out disfigurement and death to drink-drivers, children trespassing on railway lines and women running in the street.  This film is like one of those: it builds up a sensation of mounting dread.  You know something bad is going to happen to these nice people, but what? And to whom?  Here, the answer might as well be: everything that possibly could, and to everyone.

That’s the beauty of it.   The scenario that is developed is one in which there are a number of factors that all contribute to corruption and fraud going undetected for some time.  None of the characters are unbelievably good, and none unbelievably bad – all of them succumb to pressures that we see in real life in one form or another.

I’ve worked in both Professional Practice and Audit Quality for a number of years now, so I’m particularly interested in how the auditors in the film behave and why – and how we should respond to that. In my experience, audit firms tend to take what we might call a “person approach” to dealing with quality issues. Poor decisions are seen as arising primarily from flaws in an individual person’s mental processes such as forgetfulness, inattention, poor motivation, carelessness, negligence, and recklessness.

When we try to eliminate individual weaknesses, the sort of measures we put in place are directed mainly at reducing unwanted variability in human behaviour.  It’s a regulatory compliance approach.  So we ask for more procedures and more checklists; we design disciplinary measures that appeal to fear – if not fear of litigation, fear of sanctions – naming, blaming, shaming and, these days, fining.  There’s an uncomfortable implied moral subtext to this approach in that it seems to inherently assume that bad things happen to bad people.

The film instead highlights the value of what we might call the “system model”.  In this model for understanding failure, human errors are seen as inevitable products of systemic weakness. We can’t change the human condition, so we have to change the conditions in which humans operate.

An audit team is a system of defensive layers – like the “Swiss Cheese” model proposed by James Reason, Professor of Psychology at Manchester University[1]. There are holes continually opening, shutting, and shifting in each slice of cheese. The presence of holes in any one “slice” does not normally cause a bad outcome. Usually, this can happen only when the holes in many layers momentarily line up, as in the film, where there are multiple opportunities for the fraud to be identified, and multiple failures – some individually minor – are required for it to go undetected.

In the film, you see the individual active failures – poor decisions made by each character – but you also observe the latent conditions that increase the possibility of poor decision-making.   Professor Reason uses the analogy of mosquitos for active failures, versus mosquito breeding grounds for latent conditions.  You can swat all the mosquitos you want, but if you don’t drain the swamp, they’ll keep coming – and you’ll have to keep swatting.  In the film, these swampy conditions include overwork, time pressure, a culture of rewarding strong relationships with client executives and the sort of hierarchy where none of the senior people seem to seriously entertain the possibility that the concerns of more junior members of the team might ever amount to much.

I want to make particular mention too of the way the CFO in the film plays on institutional sexism by criticising the female audit partner for “interrogating” him.  Research at Stanford University is ongoing but shows that women receive 2.5 times the amount of feedback that men do about aggressive communication styles. Another study found that negative personality criticism showed up thirty times as frequently in appraisals of women as in appraisals of men, though the population selected for that review had all been considered to be equally strong performers. The women were much more likely to be described as “abrasive”, “coming on strong”, “strident” or “aggressive”.

So one of the latent conditions in our profession is a particular disadvantage to women.  It seems women are much more likely to be criticised for the robust challenge, persistence and scepticism that would be praised in a male colleague.

So why does the idea of personal responsibility for failure persist? Well for one, we tend to prefer it – it resonates with our ideas of responsibility and accountability.  It’s much easier to sanction a person than to change the culture that fostered that person’s mistakes. And sadly, we’re all human and we find blaming individuals emotionally satisfying.

We also like the idea of single causes because we are afraid of risks we can’t control. Sidney Dekker, Professor of Safety Science at Griffith University, Australia says “The failures which emerge from normal everyday systems interactions question what ‘‘normal’’ is. It is this threat to our epistemological and moral accountancy that makes accidents of this kind so problematic. We would much rather see failure as something that can be traced back to a single source, a single individual. When this is not possible in the assignation of blame and responsibility, accuracy or fairness matters less than closing or reducing the anxiety associated with having no cause at all. In the Western scheme of things, being afraid is worse than being wrong, being fair is less important than being appeased. Finding a scapegoat is a small price to pay to maintain the illusion that we actually know how a world full of risk works.”[2]

So what do we do?  We need a reporting culture and we need safe spaces to analyse what is reported.  Without a detailed analysis of mishaps, incidents and near misses we have no way of uncovering recurrent error traps or of knowing where the “edge” is until we fall over it.  Both Reason and Dekker refer in their work to “Just Culture”, in particular restorative Just Culture rather than retributive Just Culture.

A Just Culture is one with a vital “collective understanding of where the line should be drawn between blameless and blameworthy actions” (Reason).  It’s a culture that learns and prevents by asking why it made sense at the time for highly intelligent, highly educated, highly trained and highly regulated professionals to do what they did. How many audit firms are really asking that question about failures?

I am hoping therefore that no-one is going to leave the film thinking that the next step is to warn audit partners about that Bad Things will happen to them if they don’t get written representations about related parties.  Let’s instead look for our swamps and set about draining them.  In the context of the film that might include:

  • Looking at how complaints to “relationship” partners about audit team members are handled
  • An honest look at whether and how individual patronage plays a part in promotion processes and the allocation of valuable work within firms
  • Examining the trends/differences in language used in performance appraisals to describe certain behaviours when shown by women or men.

Those are just a few suggestions – there are many other areas to consider.

As Professor Reason says “Perhaps the most important distinguishing feature of high reliability organisations is their collective preoccupation with the possibility of failure. They expect to make errors and train their workforce to recognise and recover them. They continually rehearse familiar scenarios of failure and strive hard to imagine novel ones. Instead of isolating failures, they generalise them. Instead of making local repairs, they look for system reforms.”

I would like to say that I work in a High Reliability Organisation.  But are we audit firms prepared to turn that unflinching scrutiny on ourselves?

[1] Human error, models and management (Reason) BMJ. 2000 March 18; 320 (7237): 768–770

[2] Cognitive engineering and the moral theology and witchcraft of cause (Dekker, Nyce) 2011

Telling stories: Appraisal

It’s National Storytelling Week so one of the writers on my team has challenged us all to write a 100-word story on a subject loosely related to our workplace.  This is my first attempt.



The manager prepared meticulously. Her introductory monologue well-rehearsed, she was sure of saying the same things to everyone.

“We want our appraisals to be as fair as possible.  They are based on ‘competencies’. That means I evaluate objective evidence of your behaviour, rather than forming subjective opinions.”

She researched in advance. It had been dark when she’d pressed the audit partner for views of the in-charge.

“She has…” he turned away, eyes drawn to light outside, “… a beautiful face.” He tasted the words as they formed.

Now, impassive, behind the desk, she began “You have not met expectations.”

Stark choices for Tony

I have two excuses for this blog post. The first is that any parent of an eight year old is obliged to be able to name a favourite super hero.  The second is that once I’d thought of this title, the only way to get it out of my head was to write the post.  For those who don’t have a favourite super hero, Tony Stark is the billionaire engineer and industrialist who, in the Marvel comic book series created by Stan Lee, and in the recent re-imagined films, becomes Iron Man.

Stark’s wealth comes from his ownership interest in an arms manufacturing company.  However, when captured by terrorists who try to force him to build a weapon of mass destruction, he instead builds a powered suit of armour, which he uses first to escape the terrorists and then, following multiple improvements, in his new guise as Iron Man, to save the world.

While being held hostage, Stark discovers that Stark Industries’ products are making their way into the hands of terrorists, and being used on civilians.  When he arrives home, the first thing he does is announce that Stark Industries will no longer manufacture weaponry, and he diverts its resources to developing clean fusion energy.  But is that his only choice as a responsible businessman or is it possible to be a responsible arms manufacturer?

I believe it’s possible to develop a code for ethical behaviour in any industry.  While an ideal world would be one without armed conflict, there is an ethical role for weaponry in the world we have created – to protect freedom and defend human rights.  But an ethical arms manufacturer will not want to supply those whose aims are the opposite.  It will, therefore, need a code of behaviour which, if adhered to, will prevent its products and intellectual property from ending up in the wrong hands.

In common with other codes of trading behaviour (for example, anti-bribery and corruption), an effective code for ethical supply would need to cover the tone at the top of the organisation, risk assessment and management, policies and procedures, communication to and training of employees, relationships with third parties, internal controls and monitoring, review and discipline.  Within each of those areas, the organisation would identify its objectives, for example setting the limits as to what types of customer behaviour might lead to black-listing, identify the threats to those objectives, and the resulting controls that could be put in place.

Evaluating the effectiveness of the code would require a top to bottom look at how it has been implemented across the organisation including, for example, the effects on corporate governance, recruitment, internal audit, contract due diligence, physical controls over inventory and cyber-security.  Scepticism might demand that the organisation makes an external statement about its principles and how they are applied in practice, and that that statement comes under scrutiny from an independent organisation.

An added complexity for an arms manufacturer is the question of how much of the decision-making around ethical supply can be sub-contracted to the government of the day.  Some would argue that if a particular territory is considered by a democratically elected government to be a suitable customer for weaponry, the manufacturer should not have to think further about this.  I don’t think it’s so clear cut; adding a layer of agency to the decision-making, introducing other agendas, can muddy the decision-making waters.

In fairness to Tony Stark, he wouldn’t have had any luck implementing this approach anyway, as his chief executive officer is a wrong ‘un for whom ethical supply is the bottom of the agenda – without buy-in from the board, any plan to change behaviour would surely be doomed to failure.  And Stark Industries doesn’t seem to have internal or external auditors; or at least, if it has they don’t feature in the story, sadly – auditors could have been Tony’s best ally in implementing and monitoring the effect of a programme to change behaviour.

Becoming Iron Man might be Tony Stark’s only choice, but it isn’t the only choice for an arms manufacturer.  We might wish weapons didn’t exist, but we all rely on their existence. The challenge for a responsible business is how to control their supply in a way that maximises their deterrent power and minimises the devastation they can cause.

On auditors’ independence, self-review and twisted knickers

In July 2009, Rentokil Initial Plc walked into the middle of a storm when they announced:

“The Company invited its existing auditors PricewaterhouseCoopers LLP as well as KPMG Audit Plc to submit proposals for a more integrated financial assurance process extending external audit coverage to some work undertaken by internal audit. The board has decided to proceed with KPMG, who will be appointed to undertake the 2009 audit. Combined internal and external audit costs will reduce by approximately 30%.”

Within a week, under the headline Cut-price Rentokil-KPMG deal raises ethical questions for auditors Accountancy Age reported that:

“The arrangement has led some to re-examine audit’s ethical code which highlights two main risks when external auditors also provide internal audit services.  The first threat, known as the self-review threat, warns against the external auditor relying heavily on its own internal audit work. The second threat, known as the management threat, warns against the internal auditors assuming the role of management.”

Two years later, the Financial Reporting Council’s Audit Inspection Unit (“the AIU”) (now the Audit Quality Review Team) reported:

“In the light of the prior year publicity surrounding the provision of “extended assurance” services, we reviewed the audit of two entities where such services had been provided. We did not identify any relevant requirements of the Ethical Standards applicable at the time that had not been met or any adverse impact on audit quality.”

Confused yet?  Had KPMG engaged to be both external and internal auditors at Rentokil?  If they had, why wasn’t this an ethical problem for them and their regulator?

In 2009, when KPMG proposed for the Rentokil audit, they were working within the boundaries set by the Auditing Practices Board (“the APB”) Ethical Standard 5 (“ES 5”) 2008 Non-audit services provided to audited entities.  This included a prohibition on an  audit firm providing internal audit services to an audit client “where it is reasonably foreseeable that:

a)      for the purposes of the audit of the financial statements, the auditor would place significant reliance on the internal audit work performed by the audit firm; or

b)     for the purposes of the internal audit services, the audit firm would undertake part of the role of management” (APB ES5: 44)

APB ES5 (2008) went on to give an example of a scenario where the self-review threat would be considered “unacceptably high” noting that the auditor of a large bank should conclude that it is unacceptable to provide internal audit services, too, to that bank since the external audit team is likely to place significant reliance on the work performed by the internal audit team in relation to the bank’s internal financial controls (APB ES5 (2008): 44-45).

Wait! Run that by me again please: does it say that the auditor may not test the internal financial controls of its audit client? No – the external auditor is not only permitted but expected to test internal financial controls.  But what if the internal auditors have already tested them? Where that is the case, the external auditor has a decision to make: can the testing carried out by internal audit be relied upon or should the external auditor retest the controls?

APB International Standard on Auditing (United Kingdom and Ireland) 610 reminds the external auditor that he “has sole responsibility for the audit opinion expressed, and that responsibility is not reduced by the external auditor’s use of the work of the internal auditors” (APB ISA (UK&I) 610: 4)  Inevitably, therefore, the external auditor will reperform some testing; the question is how much?

Provided the external auditor can establish to his satisfaction that the work of internal auditors is reliable, he may reduce considerably his own controls testing.  This remains an option if the internal audit services are contracted out to a professional services provider, but, according to APB ES5 (2008), not if that professional services provider is the same firm as the external auditor. Why? Because then the external auditor would be reviewing his own firm’s work, which constitutes self-review, creating exactly the scenario identified in the 2008 standard as constituting an “unacceptably high” threat to the external auditor’s independence.

The flaw in this reasoning can be exposed thus.  Supposing the external audit team does not plan to use the work of internal audit, but rather re-perform all of the controls testing themselves.  This could conceivably result in a scenario where each sampled instance of any one control is tested twice: once to provide comfort to those charged with governance, acting as a proxy for the shareholders, by the internal auditors and once to provide comfort to exactly the same ultimate party, by the external auditors.  This actually seems a dangerous outcome because those charged with governance may wrongly conclude that they are getting twice as much comfort, whereas in fact they are being supplied with the same result twice by different teams. 

Where the effective operation of a control is an objective binary outcome – it either operated effectively or it failed – no matter how many times any one historical instance of that control is tested, assuming no negligence on the part of the testers, the result of the test will be the same.  The same is true for a substantive test with an objective, binary outcome.  Does it matter whether such tests are performed once, by one team, who then report the result directly to those charged with governance (previously the role of the internal auditor) but then go on to use that result as evidence in forming the external audit opinion? 

It affects the cost to the shareholders of the overall level of assurance achieved, with it clearly being preferable to pay for each test only once.  And it affects the effectiveness of assurance activities, because the elimination of duplicate testing provides resource for other controls (generally operational controls) to be brought into the scope of testing for the first time.  Moreover, as noted above, the very existence of separate internal and external audit teams can create a false sense of security for stakeholders, if, in reality, both teams largely duplicate each other’s work.

The APB recognised the rationale behind the extended audit model and, following an open consultation with interested parties, revised APB ES5 accordingly.  This now recognises that some extended audit work is effectively a part of the external audit and, subject to some conditions, presumed not to be a threat to the auditor’s independence.

The conditions imposed by the APB on allowing such extended audit work to be considered part of the audit are that it relates to financial information and/or financial controls, is authorised by those charged with governance, is integrated with the work performed in the audit and is on the same principal terms and conditions as the audit. (APB ES5: 68-69).

So are auditors and those charged with governance of cash-strapped audited entities now living happily ever after?  Not quite, for it is relatively easy for some testing which the audited entity wishes to commission to fall without the four conditions set out above, for example, testing of operational rather than financial controls.

Where this is the case, the additional testing is not prohibited, but the auditor is forced to consider the threats to independence, principally management and self-review!  However, as explained above, where the outcome of any testing is an objective, binary result, it cannot make any difference to the finding whether the test is performed once, twice or many times.  Duplicate testing to avoid self-review is an expensive, redundant safeguard.  So we are left asking why APB ES5, in this specific regard, positively requires the auditor to consider a threat to independence – self-review – that cannot, in fact, have any effect on the auditor’s independence. 

The investigation into the extended audit model significantly failed to justify the historical concern over the external auditor doing too much testing.  The resulting amendment to APB ES5, by leaving in place the requirement to consider the self-review threat in a broad range of circumstances where the four extended audit criteria will not be met, has created a lasting legacy of concern that for the external auditor to go beyond the bare minimum of testing needed to form the audit opinion is, somehow, unethical.  As a result, there has been a relatively low take-up of the opportunity to extend the scope of the external audit, and reduce any duplication of internal audit test work.  This should be a matter of concern to all stakeholders.

Even if we attribute the hysterical response of the financial press on the occasion of the Rentokil announcement to their concern for institutional investors, who are said to regard the independence threats of the external auditor performing internal audit work as some of the most significant of any non-audit service, we get no nearer to understanding why those investors are so vexed. 

An outsider may conclude that one of the greatest threats to the quality of the external audit is for there to be any co-ordination or even combination of internal and external audit in the interests of efficiency and cost savings.  But if that is the case, why haven’t we seen the FRC mandate a minimum level of internal audit testing to be carried out by listed companies?  Indeed, a number of listed companies have no internal audit function at all.

It remains necessary, critical even, that for internal audit services which do not meet the four conditions to be considered an extension of the external audit work, the effect of the management threat to independence be fully assessed by the auditor and either safeguards put in place, or, where no safeguards are adequate, the opportunity declined.  And it is right that the auditor must consider the self-review threat to independence where the non-audit service being offered is one in which the professional forms subjective opinions, such as valuation services.  Here, the auditor’s objectivity must be safeguarded from the inevitable bias that arises when a colleague from the same firm has already formed a view.

But it is high time that the auditing profession and its regulators re-opened the discussion about self-review.  If the auditor can’t be trusted to review anything he or a colleague has done without bias creeping in, where is the value in the many review processes mandated by auditing standards as an essential part of the external audit?  And how can the auditor ever be satisfied with his own work on the previous year’s figures, a genuine instance of self-review that is far more likely to require objective professional judgment than just evaluating the results of internal audit work over financial or non-financial controls? 

In the current environment, the onus is on auditors to deliver to stakeholders the maximum breadth and depth of independent assurance for the fee. This commitment to delivering value is, in itself, an ethical purpose, one that must follow naturally from compliance with applicable ethical standards, if those standards are to achieve their intended result.  It’s vital, therefore, that the auditing profession investigates and addresses the residual concern remaining that extending the scope of the external audit and eliminating duplication from the work performed by internal and external auditors is – somehow, for some reasons, we can’t quite explain why – unethical.

%d bloggers like this: