We do not see AI taking over from humans in drafting legislation.
- We have tried it for drafting, and it is good for suggesting provisions the drafter might not have thought of, but the drafting of the actual provisions is still nowhere near rigorous enough for workable legislation.
- See our experiments with drafting a Commercial Baking Act/Law with ChatGPT in 2023 and 2024 in our AI & legislation folder on OSF -
- ChatGPT - Commercial Baking Act.pdf
- Drafting legislation with free ChatGPT as at 15 May 2024.pdf
- Drafting legislation with ChatGPT 4o, again as at 15 May 2024.pdf
Nor do we see AI reliably answering questions about legislation on its own.
- Some tech firms are working on AI (generative AI, or LLMs) to make chatbots read legislation and answer questions about it. Examples include -
- Codify AI for Canadian legislation, and
- Max for Jersey Financial Services Commission’s AML/CFT/CPF Handbook
- But LLMs are not able to reason as such, as opposed to developing existing patterns, so they will remain unreliable (“hallucinating”) for tasks that involve working out how legislative provisions apply to given fact scenarios. They often suffer where answering a question involves applying definitions, or following cross-references, as opposed to looking up an answer that is already on the face of the legislation.
But we do see AI helping humans.
- Legal Drafting in the Era of Artificial Intelligence and Digitisation – Palmirani & others – AI could help with the nuts and bolts of drafting and managing draft legislation.
- We are working on using AI to help with systems that reason reliably using marked up legislation. See the article by Dr Guzyal Hill “From pen to algorithm: optimizing legislation for the future with artificial intelligence”, to which we contributed.
- We are collaborating with Digital Jersey, the Government of Jersey’s AI Working Group, and the Working Group on Legislation and AI at the Commonwealth Association of Legislative Counsel.
- We are producing information notes to help tech firms thinking of developing a chatbot for questions about Jersey legislation.
So can AI help human drafters to parse draft legislation (or maybe already enacted legislation) for key elements that a useful computer program would want to be able to read? We currently think a “hybrid” approach looks most likely to be useful –
- statistical/generative AI, and
- rule-based logic systems
helping each other. See Jason Morris “GPT4 & Rules as Code” and a Canadian government team’s bid to develop Rules as Code chatbots.
We reckon AI could also be helpful as part of the tool for legislative drafters that we are calling for, to help us mark up the logical structure of drafts. We are working with Digital Jersey, Go Reply & Google on a test of how Google Cloud’s AIs (beyond LLMs) can learn to identify standard/logical elements in drafts. That would help drafters make their drafts computer-readable.
- Definitions – scope, defined term, definition
- Offences – offence creation, penalty
- Vires to make subsidiary legislation - person given power, type of instrument
- If-then and-or-not structures (next) – conditions, effects, scope of and/or/not
In the longer run we imagine LLMs being useful as the interface with the user, guiding the user to a relevant item of legislation for a logic-based reasoner to ask questions and give fully sourced answers, but perhaps with the LLM then giving a summary and suggesting further questions. We think of that as an AI and reasoner sandwich - see our AI sandwich slides
We are planning to work in 2025 with the Centre for Computational Law (at Singapore Management University) to see how their L4 language could form the basis of the reasoner filling in an AI sandwich of this sort.
This page is being updated. You can find the documents we produce in our AI stream on OSF: https://osf.io/bkjqx/.