Summary: We analyzed the over 1,450 submissions to the National Telecommunications and Information Administration’s request for comment on AI accountability to identify patterns in today’s current AI policy landscape. Our primary observation is that, but for a small number of astute comments, a good deal of today’s AI policy proposals are unlikely to be effective and are rarely matched to the harms they ought to prevent. We present a simple framework that looks at the alignment between market incentives and individual benefit to identify appropriate policy interventions.

We augmented the bulk data download from regulations.gov for our analysis and have made our dataset publicly available here.

Today’s AI policy landscape has a coherence problem. There are plenty of ideas, a surfeit of headline-driven urgency, but not much of a vision for how these ideas might work together or whether they’ll work at all.

The NTIA’s RFC for AI accountability all but confirms this. The more than 1,450¹ comments from companies, industry groups, nonprofits, academics, and unaffiliated individuals paint a picture of concern, thoughtfulness, but insufficient attention to the central question of what makes for a regulatory framework that actually protects individuals while allowing us to realize AI’s promise.

A simple framework for AI regulation

Figure 1.

A simple but powerful way to target where regulation can be most effective is to look at where market incentives and individual benefits diverge. When they’re aligned, we can generally expect self-regulation to work, but when they part the story gets more complicated. If market participants have an incentive to do something harmful (e.g., polluting the local river) we typically need rules and enforcement. If the public would benefit from something that doesn’t make business sense (e.g., broadband in remote areas) we need encouragement and subsidies. And for things that no one wants, it usually falls to governments to prevent them.

While the framework in Figure 1 helps determine the type of regulatory solution that is needed, even a properly targeted policy must still address real harms and use the full range of tools at its disposal to be effective. We mapped the submitted comments to this framework to understand what proposals would be most effective and where there were gaps.

While some of the best ideas came from expected sources, like leading academics and companies with strong public mandates, they also came from non-technical sources – a challenge to the idea that technical AI expertise is a prerequisite for participating in AI policy debates.

Themes from professional commenters

Professional commenters, such as corporations, industry groups, non-profits, and academics had much less diversity across comments than we might have expected. Beyond a general sense of urgency we did observe a number of general themes, which we highlight below.

Government as regulator vs. enabler

A surprising number of submissions,² particularly from corporations, assumed that the government’s primary role would be to support their best practices (audits, transparency reports, etc.). This is reasonable when behaviors fall in the “Self-regulation” quadrant of Figure 1, but certainly not in the “Stick” (or “Enforcement” quadrant) where company behavior needs to be policed. Consider sophisticated uses of private data to make targeted recommendations appear as the correct response to prompts: without privacy laws, there’s little to constrain things like these.

Mismatch between remedies and harms

It was also striking how many corporate proposals simply assumed that their proposals were beneficial without identifying the specific harms they were supposed to address. A few academic commenters highlighted the limits of this approach – both the risks of overreach and of the ineffectiveness of “AI half-measures.”³ The better proposals offered robust frameworks to address actual concerns (like preventing the proliferation of non-consensual intimate imagery), which helps to link technological advances, deployment best practices, and government enforcement to create solutions that are far more likely to succeed.⁴

Risk-based regulation as a near corporate consensus

Corporations were very likely to recommend “risk-based” regulation as the optimal approach for managing AI’s concerns.⁵ This isn’t surprising – risk regulation focuses on impact assessments and internal risk mitigation at the expense of tort liability and individual rights, which certainly favors corporate interests. As we have highlighted before, there are good reasons to reject this approach: making companies responsible for the consequences of the things they build is a much more powerful restraint than, say, an audit requirement.

Emerging confidence in NIST and similar frameworks

Where the National Institute of Standards & Technology (NIST) AI Risk Management Framework (RMF) (or similar standard) was mentioned,⁶ it was uniformly positive. This is a good reason to continue investing in the organization and its efforts and for developers of AI systems to take the NIST AI RMF seriously in their work.

Predictable disagreements

While no group of submitters (corporate, nonprofit, etc.) was entirely homogeneous, there were broad and somewhat expected patterns in the comments.

Corporations and industry groups generally pushed for self-regulation⁷ (without specific detail about the harm that would avoid) while non-profits and academics were more insistent on legal standards and enforceability.⁸ Where corporations were open to government regulation, they typically insisted that it should be as narrow⁹ as possible, whereas non-profits and academics were more disposed to broader regulatory regimes. The obvious exceptions were non-profit and research groups whose mandates are to limit government reach.

Themes from individual commenters

The very human submissions from individuals were a poignant contrast to the abstract policy proposals submitted by the professional commenters.

The RFC process lets anyone submit a comment. So while there were the occasional flip remark or throwaway comment, the bulk were expressions of genuine and often specific worries about what AI will mean in these individuals’ lives. If we want to take addressing real harms seriously, this list can act as a litmus test for proposed policy frameworks. Regulators must also make sense of future harms, identify issues not brought up by commenters, and to provide emphasis and solutions appropriate to the nature of each concern, but this is a powerful starting point for that work.

List of concerns organized by theme

There were over 1,200 individual comments, including many thoughtful policy suggestions and articulated concerns. The majority touched on the following themes:

• Economic disruption: job loss and inequality, loss of livelihood and purpose, financial hardship, copyright infringement¹⁰
• Bias and discrimination: AI bias against minorities, discrimination perpetuated by AI¹¹
• Safety and aligned values: AI harming or manipulating humans, AI lacking human values¹²
• Malicious or misleading uses: spread of misinformation by AI, deepfakes, unauthorized use of likeness¹³
• Loss of privacy and misuse of personal or restricted data¹⁴
• Inappropriate regulation: unaddressed harms, hampering progress¹⁵

Where we go from here

This RFC is a remarkable snapshot of the AI policy landscape in June 2023. Anyone could make a case and it seems like every major player did, giving us a view into the policy intentions of the most significant and influential players in the AI ecosystem. On the other side, concerned individuals were able to express their worries about AI in their own words.

We now have a map of the AI policy landscape as expressed by its constituents – and, by extension, the expected regulatory battle lines. Now is the time to give these perspectives the scrutiny they deserve and draft rules that actually protect individuals while AI pursues its forward march.

If you would like to conduct your own analysis of the RFC, we’ve made our dataset publicly available. And if you’re interested in collaborating on the development of effective AI policy, regulation, and safety best practices, feel free to reach out and say hello.

Appendix: The view from Congress

Senator Richard J. Dubin submitted a short letter on behalf of the Senate Judiciary Committee that gives some visibility into how the branches of government are likely to interact as AI policy develops. It critiques the consequences of Section 230 of the Communications Decency Act (”…[it’s] unquestionably responsible for many of the harms associated with the Internet…”) and then goes on to say:

…[an AI] accountability regime also must include the potential for state and federal civil liability where AI causes harm… we also must review and, where necessary, update our laws to ensure the mere adoption of automated AI systems does not allow users to skirt otherwise applicable laws (e.g., where the law requires “intent”). We also should identify gaps in the law where it may be appropriate to impose strict liability for harms caused by AI, similar to the approach used in traditional products liability. And, perhaps most importantly, we must defend against efforts to exempt those who develop, deploy, and use AI systems from liability for the harms they cause.