How Microsoft 365 Copilot really works: the architecture in plain language

"Where does our data actually go when someone asks Copilot a question?" It is the question that comes up in every boardroom and on every IT desk — and rightly so. The good news: the answer is well documented and more reassuring than many people expect. In this article we explain the architecture of Microsoft 365 Copilot in plain language.

Four building blocks

Copilot is not a standalone chatbot but an orchestration layer that connects four components. One: large language models that understand and generate text. Two: the Microsoft 365 apps you work in — Word, Excel, PowerPoint, Outlook, Teams. Three: Microsoft Graph, the layer that unlocks all your work data: emails, documents, chats, meetings. Four: the semantic index, which understands meaning rather than just keywords. Search for "satisfied customers" and the index will also surface documents that talk about "enthusiastic responses".

The journey of a prompt

What happens when you ask Copilot something? Step one: you type a question into an app. Step two — and this is the crucial step — Copilot first enriches your question with relevant context from your own environment. This is called grounding: through Microsoft Graph and the semantic index, the emails, documents and conversations that are relevant to your question are retrieved. Step three: the enriched question is sent to the language model. Step four: the answer comes back into your app, with source references. Everything is encrypted, both in transit and at rest.

The entire process takes place within the Microsoft 365 service boundary — the same secure perimeter within which your mail and documents already live. Your data does not leave that boundary while you use Copilot. In doing so, Microsoft uses the Azure OpenAI services, not the public services of OpenAI.

The most important reassurance: permissions stay permissions

Copilot operates within your tenant, but that does not mean it sees everything. Every question is answered within the permissions of the signed-in user. If you have no access to the HR folder, neither does your Copilot. The semantic index never changes anything about permissions; access is re-checked with every single question. Sensitivity labels and encryption from Microsoft Purview also remain in force: anyone without extraction rights on a protected document will not get its contents back through Copilot either.

There is, however, an honest caveat to add: Copilot makes existing problems visible. If a document with salary data has been shared far too widely on SharePoint for years, everyone could always reach it — only nobody ever found it. Copilot does find it. That is why every responsible rollout begins with an oversharing scan and a clean-up of permissions.

Is our data used to train AI?

No. Prompts, responses and the data retrieved via Microsoft Graph are not used to train the underlying language models. Your Copilot conversations are stored alongside your other Microsoft 365 content — manageable with retention policies and eDiscovery, and viewable and deletable by users themselves. Relevant for European organisations: Copilot falls under the existing commitments around the GDPR and the EU Data Boundary, under which traffic from EU users is processed within Europe. Microsoft has also committed to the EU AI Act and is certified for, among others, ISO 27001 and ISO 42001, the standard for AI management systems.

Built-in guardrails

Between your question and the model there is also a security layer: filters against harmful content, detection of copyright-protected material, and classifiers that block jailbreaks and prompt injection before the model executes anything. There is even a filter that prevents Copilot from making statements about the performance or emotional state of employees.

What does this mean for your organisation?

The architecture is solid — the risks almost always sit in your own environment: permissions that are too broad, missing labels, no audit logging. That is exactly what we map out with the Copilot Readiness Scan, including a concrete step-by-step plan towards a safe go-live. Want to know where your environment stands? Schedule a no-obligation introductory call.