Definitions
Weak AI, Strong AI, AGI, and Superintelligence
Weak (or narrow) AI encompasses systems that are focused upon completing specific tasks and which are not designed to generalise. Strong AI refers to systems that can generalise to complete a range of tasks much as human beings do.
AGI is broadly synonymous with Strong AI, as both terms refer to systems that can generalise and perform tasks at a level of competence analogous to human beings. Superintelligence is a term for systems that surpass the capabilities of human beings across a variety of domains.
Generative AI, Agents, and Large Language Models (LLMs)
Generative AI systems are designed to create new content. They receive prompts from the end user and can generate text, images, video or computer code upon request. LLMs are a type of AI system that predicts the next word in a sequence based upon patterns in existing text. They are trained on enormous amounts of data and can be used to generate content in a multitude of contexts. LLMs are currently the most likely form of AI to be used by businesses and are a form of weak AI. Some popular LLMs include:
- Claude (Anthropic)
- GPT (Open AI)
- Copilot (Microsoft)
- Llama (Meta)
- Gemini (Google DeepMind)
- Grok (xAI)
- Qwen (Alibaba)
- DeepSeek (DeepSeek)
LLMs can also be used to build autonomous agents that can independently carry out tasks, often unsupervised by the user, based upon chat prompts. The most popular open-source agent platform is OpenClaw, which can be self-hosted and can make use of a range of LLMs. Various LLMs and agents have been developed with a focus on developers, such as GitHub Copilot, which can be integrated into VS Code.
Data Protection and Ethical Issues
Inaccuracy
LLMs work by associating words together based upon patterns in existing content, including names. They don’t necessarily understand the nature of the associations they find and thus can misrepresent them. One such example is when, in August 2024, a court reporter was erroneously identified by Copilot as having committed the crimes he had reported on. Bias, stereotyping, and discrimination in training data is also a consideration, with models potentially producing content that is inaccurate or discriminatory.
Processing personal data
Many popular generative AI models are based outside of the UK. Any use of such models should be considered within this context. Prompts and their resulting outputs may be stored by the system being used, potentially in countries with which the UK does not have adequacy agreements. Where such processing involves personal data, appropriate safeguards should be in place, and a lawful basis should be identified for carrying out processing.
Security concerns
There have been various examples of successfully persuading an LLM to generate harmful or inappropriate content by exploiting vulnerabilities in the system. This involves using techniques such as jailbreaking, prompt injection, and prompt leakage. These techniques can be used to persuade an LLM to ignore its own safeguards or trick it into releasing potentially confidential information. This adds an additional dimension to the usual security concerns pertaining to the use of cloud software.
Furthermore, foundation models running as autonomous agents have been shown to be capable of carrying out sophisticated cyber-attacks. An OpenAI model with its guardrails disabled recently exploited a zero-day vulnerability in a proxy for hosting software packages so that it could access the open web, then it hacked Hugging Face seeking answers to the benchmarks it was being tested against. A key component of resolving the attack was the use of coding agents by Hugging Face. Thus, AI coding agents may become a required security measure for protecting against future cyber threats.
Using Generative AI Safely
Lawful processing
Personal or sensitive information should not normally be included when prompting generative AI to produce content. This includes images containing individuals. Where it may be necessary, a lawful basis for processing should be identified. An appropriate data processing agreement should also be in place, one that addresses data transfers outside of the UK. Most such agreements require a commercial contract with the service provider, which will also need to be in place.
Disclosure
The outputs of generative AI should be clearly identified when used as a basis for content, or communication. Should such a system be used for automated decision making, further safeguards must exist to comply with UK GDPR. These include seeking consent from data subjects, providing the option of human intervention in the decision-making process, and giving them the right to contest any decision made. An up-to-date data protection impact assessment (DPIA) should be in place to cover such decision-making processes.
Vibe coding
Vibe coding is the process of generating code by prompting an LLM. It should be avoided in certain circumstances, such as when building critical production systems from scratch, as it can result in poorly optimised and insecure code that the LLM user may not be able to debug. There are some circumstances where generative AI can be useful for coding, however. The simplest example is producing commit messages, a feature that is already available in VS code. Other examples include generating dummy data, manipulating data structures, creating simple functions or classes, generating or explaining documentation, and creating unit tests. As discussed previously, lawful bases for processing, confidentiality, and security are considerations where personal or sensitive data is involved.
Approved suppliers
The only supplier with which we have a commercial relationship is Microsoft, and thus Copilot should currently be the model of choice if cloud hosted generative AI is required, with GitHub Copilot used for autocompletion or chat in VS Code. Gemini is integrated into Google search and thus may also be used.
Open-weight models that we can self-host, such as Llama and DeepSeek, are acceptable. Ollama software should be used when hosting models locally. The Continue plugin should be used to integrate open-weight models into VS Code. Training data and the pretraining process must be considered, with reference to this policy, however.
Usage of autonomous AI agents should be agreed with the Technical Director and the Data Protection Officer in advance. Where AI is used for research or code generation purposes, secondary sources should be used to verify information, and code review should be carried out prior to deploying to production.