As ChatGPT rockets AI governance to the forefront of discussion, IAPP and FTI Consulting’s annual privacy management report finds that over half of organizations are building their approaches on top of existing and mature privacy programs. But while the commitment is often there, the tools and skills may not be as the workforce only just begins to develop.
The study finds that the leading risks are harmful biases in AI, followed closely by poor governance and a lack of clarity in a rapidly developing legal landscape. While an assortment of international legal frameworks are available, a standard has yet to emerge and there is a great deal of misunderstanding about exactly what risks organizations are facing.
Privacy management and AI governance report 2023: Steady near-term AI growth to bring new challenges
With growth predicted at 25% per year over the next five years, there is an immediate need to establish AI governance programs. The organizations that are having the greatest amount of early success with this are those that are building on top of strong existing privacy management programs; however, that represents only 20% of respondents.
10% say they do not yet have any plans in place, and the remaining 70% are either just about to begin or are working through implementation of responsible AI governance systems. 40% of those that are working say that they are building algorithmic impact assessments on top of existing privacy management processes.
The main wall that organizations appear to be running into is a lack of available tools and talent. The AI governance workforce is only beginning to shape up, and has some interplay with already existing shortages for qualified privacy and security professionals of all types. The development of necessary technical tools is also in its early stage, with some organizations finding products that meet their needs are simply not available yet or are difficult to track down.
“Harmful bias” leads the risk categories of AI governance that organizations are focused on, with most reporting it as a “high probability.” One of the leading concerns in this area are models that fail to be representative or incorporate some kind of unconscious bias, leading to results that may be lacking validity or even unethical. Organizations are also concerned about making promises based on expected AI capability that they then fail to deliver. Organizations most frequently believe that risk and privacy management in this area require consistent definitions of harm, established risk indicators for determining bias, clear guidelines on fairness requirements, and common tools and standards for bias detection.
“Bad governance” is almost as common of a risk concern as the potential for bias. In addition to opening up a variety of risks, poor AI governance could bloat administrative and legal budgets. Individuals currently involved with privacy management programs say that they have doubts about how principles such as data minimization and purpose specification will translate to algorithmic AI systems. Respondents are looking for clear AI governance strategies tailored to the risks inherent in processing personal data in AI systems, and would like to see AI assessments and assignment of responsibilities embedded in workforce training programs.
Fewer organizations responded that lack of legal clarity was one of their top concerns, but it did make the top three responses overall. Organizations are already grappling with emerging privacy management regulations in numerous nations and localities, with more going into effect every year, and emerging AI governance requirements will only complicate this picture. One main concern is that different regulators will require specific but different statistical assessment tools; another is the present lack of uniform benchmarking practices. In general, quite a few organizations say that they still have a poor overall understanding of AI and what their AI governance responsibilities will ultimately be.
Variety of potential risks, regulations makes ultimate shape of AI governance hard to predict
Compliance programs, privacy management and IT security are all suffering from a skills and hiring gap as there are simply not enough professionals available on the market to meet demand, and AI governance is projected to have the same shortfall even as AI use grows by leaps and bounds over the next five years. Other big questions include liability introduced by third-party partners, nonconsensual use of protected information or material under copyright in training data, and how breached AI systems connected to networks might be leveraged for attack purposes.
AI governance frameworks might provide some answers to all of these questions. However, universal standards have yet to emerge. A few are already in circulation: the US NIST AI Risk Management Framework draft, the European Commission’s Assessment List for Trustworthy AI, the UK ICO’s AI and Data Protection Risk Toolkit, and Singapore’s Model AI Governance Framework. These are all general advice rather than mandates, however, and organizations can pick and choose whether to make use of any of them; these frameworks also have substantial differences in how they define and assess risk.
There also remains wide variation (and confusion) in how organizations determine what triggers an AI assessment. Only 30% report having a formalized process in place, and those processes can vary quite a bit in how they function. Some conduct an assessment every time personally identifiable information is used in an AI application, some perform an AI review every time a data source is added, and for some an AI assessment also triggers a privacy assessment. 80% of respondents said that having high-level declarations for responsible AI governance has not yet led to a breakdown of easy-to-understand requirements for developers.
80% of respondents also said that AI ethics guidelines are still a high-level policy or strategic concept that has not necessarily broken down into specific duties or details as of yet. Only 30% say they have a central model inventory at present. And in terms of existing privacy management responsibilities, many organizations report trouble striking a balance between adequately training data sets and keeping to data minimization promises and requirements.
