Policy Review AI Pipeline — Operations Runbook

System Overview

The Policy Review AI Pipeline processes PDF and Word documents from a monitored SharePoint folder. It uses a RAG (Retrieval-Augmented Generation) architecture with Azure OpenAI GPT-4 to produce structured summaries, risk flags, and cross-reference reports.

System components:

• SharePoint document monitoring (Power Automate)
• Azure Document Intelligence (PDF extraction)
• Azure AI Search (vector index)
• Azure OpenAI GPT-4 (summarisation and analysis)
• SharePoint output library (results delivery)

Daily Operations

Starting the Pipeline

The pipeline runs automatically on document upload. No manual start required.

Monitoring

1. Check the Power Automate run history dashboard each morning
2. Verify the Azure AI Search index was refreshed overnight
3. Confirm output documents appeared in the SharePoint results library
4. Review the error log (Teams channel: #policy-ai-alerts)

Expected Processing Times

• Documents < 50 pages: 2–4 minutes
• Documents 50–200 pages: 8–15 minutes
• Documents > 200 pages: 15–25 minutes

Troubleshooting

Issue: Document not processed after 30 minutes

1. Check Power Automate run history for failure
2. Verify document format (PDF or DOCX only; scanned PDFs without OCR will fail)
3. Check Azure OpenAI quota limits in the Azure portal
4. Re-upload the document to trigger a new run

Issue: Output quality is low / hallucinations detected

1. Check document quality — poor OCR produces poor outputs
2. Review the system prompt version (should be v3.2 or later)
3. For critical documents, flag for human review before distribution
4. Log the issue in the quality tracker SharePoint list

Escalation Path