# Azure DevOps Continuous Improvement Workshop Charter

> [!IMPORTANT]
> **Disclaimer:** This is not an official Microsoft repository or workshop. These materials are provided "as is" and "where is," without warranty of any kind. Use them entirely at your own risk.

## Table of contents

1. [Purpose and outcomes](#purpose-and-outcomes)
2. [Engagement definition](#engagement-definition)
3. [Assessment method](#assessment-method)
4. [Current-state assessment](#current-state-assessment)
5. [Data collection runbook](#data-collection-runbook)
6. [Key findings](#key-findings)
7. [Finding interpretation and response](#finding-interpretation-and-response)
8. [Desired state](#desired-state)
9. [Best-practice reference implementation](#best-practice-reference-implementation)
10. [Ideal state and gap analysis](#ideal-state-and-gap-analysis)
11. [Change catalog and prioritization](#change-catalog-and-prioritization)
12. [30/60/90-day action plan](#306090-day-action-plan)
13. [Governance and measures](#governance-and-measures)
14. [Workshop completion criteria](#workshop-completion-criteria)
15. [Appendices](#appendices)

## Purpose and outcomes

This workshop creates an evidence-based view of an Azure DevOps organization (the **current state**), records what its owners need it to become (the **desired state**), and combines those inputs with Microsoft guidance and organizational constraints to define a practical **ideal state**. It concludes with an owned, prioritized, and measurable improvement plan.

The workshop will produce:

- A validated inventory of the organization, projects, identities, repositories, pipelines, artifacts, test assets, integrations, policies, and usage patterns.
- Qualitative findings that capture stakeholder goals, pain points, risk tolerance, delivery practices, and constraints.
- Quantitative findings that are reproducible, timestamped, and linked to source evidence.
- A maturity and risk assessment across the agreed assessment domains.
- A desired-state statement written and approved by the workspace owners.
- An ideal-state design that reconciles current capabilities, desired outcomes, Microsoft guidance, compliance obligations, cost, and delivery capacity.
- A catalog of proposed changes with rationale, owners, dependencies, effort, risk, acceptance criteria, and measures of success.
- A 30/60/90-day and beyond roadmap with review gates and accountable owners.

### Assessment workspace

Open [`index.html`](index.html) to use the local-first assessment workspace. It supports:

- Engagement identity, scope, sponsor, assessment lead, evidence cutoff, and restricted evidence location.
- Evidence metadata with stable IDs, source, scope, collection time, completeness, confidence, method, limitations, and restricted links.
- Findings with evidence references, likelihood, impact, owner response, and an explicit approved-limitation path when evidence is unavailable.
- Domain maturity scores that are considered supported only when they include evidence, confidence, and rationale.
- Desired outcomes, traceable gaps, owned changes, a derived 30/60/90/Beyond roadmap, approvals, governed exceptions, and accountable sign-off.
- Timestamped JSON, detailed CSV and Markdown, executive Markdown, and dedicated evidence-to-roadmap traceability exports. Browser imports receive structural, unique-ID, and evidence-to-change reference checks; API writes receive full [`assessment.schema.json`](assessment.schema.json) schema-v4 validation. Schema-v3 browser assessments migrate on import.
- Sequential module readiness gates and a live decision dashboard. Overall completion is derived from approved evidence-to-action artifacts, material-risk disposition, and distinct accountable sign-off roles, not facilitator checklist claims.

The workspace is intentionally single-assessor and browser-local. It keeps the previous parseable browser state as a recovery copy, but browser storage is not an authoritative collaboration or records system. Local approval and sign-off records are assessment artifacts, not identity-verified signatures or immutable audit records. Export schema-v4 JSON at review gates and retain it in an approved location. Raw evidence, secrets, personal data, audit payloads, and sensitive findings must remain outside the workspace; catalog only approved metadata and restricted links.

Workspace completion requires a complete engagement, evidence, traceable findings, all nine supported maturity domains, approved desired outcomes, gaps that link findings to outcomes, accepted changes linked through those gaps, approved changes and active exceptions, disposition of every Critical or High finding, and all four distinct non-declined accountable sign-off roles. The narrower assessment-readiness score remains in exports for comparison. Checklist completion never constitutes assessment completion.

### Collaboration API

Do not extend `localStorage` into a multi-user records system. The [`api`](api) directory contains a local-testable Azure Functions Node.js v4 collaboration API for deployment behind Azure Static Web Apps. It implements:

- SWA `x-ms-client-principal` parsing plus server-side authorization. A custom SWA role (`viewer`, `assessor`, `approver`, or `admin`) establishes global capability; a Cosmos membership document independently scopes non-admin users to an engagement.
- `GET /api/assessments/{assessmentId}`, `PUT /api/assessments/{assessmentId}`, and `GET /api/assessments/{assessmentId}/audit` routes.
- Server-side schema-v4 validation, `If-Match` optimistic concurrency, managed-identity Cosmos access, and explicit 401, 403, 404, 409, 412, 422, and 428 responses.
- One Cosmos transactional batch for each assessment create/update and its actor-attributed audit event. Events retain UTC time and SHA-256 before/after content hashes without duplicating the assessment payload.

The Cosmos container must use `/assessmentId` as its partition key. It stores one `assessment` document, `member:{swaUserId}` documents with `roles` containing `viewer`, `editor`, or `approver`, and append-only `audit:{timestamp}:{uuid}` documents in each engagement partition. The Function App requires `COSMOS_ENDPOINT`, `COSMOS_DATABASE`, and `COSMOS_CONTAINER`; set `AZURE_CLIENT_ID` only for a user-assigned managed identity. Grant the function identity only the required Cosmos data-plane access. Do not use account keys.

[`staticwebapp.config.json`](staticwebapp.config.json) blocks GitHub login and protects every API route with custom roles. Before deployment, assign those roles through an approved Entra/SWA process, create and review engagement membership documents, restrict the accepted Entra tenant, and configure networking, telemetry, retention, backup/restore, and incident procedures. The current browser workspace remains local-first and does not automatically synchronize with this API.

Production acceptance still requires a deployed security review and tests proving concurrent assessor conflict handling, denial and logging of unauthorized access, Entra-derived approval identity, audit retention, backup/restore, and lossless schema-v4 round trips. The code provides enforceable API behavior and local tests; this repository does not claim that an undeployed backend is production verified or that Cosmos records are immutable against privileged infrastructure operators.

### Repository validation

The repository pins its formatter and Playwright version in [`package.json`](package.json), and API runtime dependencies in [`api/package.json`](api/package.json). Run the same static contract, formatting, API unit/audit, offline collector, and five-scenario browser suite used by CI:

```powershell
npm install
npm install --prefix api
npx playwright install chromium
npm run check
```

The 19-test API suite covers SWA principal rejection, dual role/membership authorization, cross-engagement denial, viewer/editor/approver behavior, schema enforcement, the current workshop identity, mandatory finding-to-gap-to-change traceability, ETag preconditions, Cosmos partitioning, transactional audit writes, and HTTP response mapping. The browser suite covers blank-state integrity, module gating, collector-catalog merge, evidence and assessment round trips, full evidence-to-approved-roadmap traceability, executive and traceability exports, dashboard calculations, artifact-derived completion, and 360-pixel mobile overflow. [`.github/workflows/validate.yml`](.github/workflows/validate.yml) runs the same checks for pushes, pull requests, and manual dispatches.

### Delivery modes and 30-minute modules

The workshop supports two delivery modes over the same evidence and outputs:

- **Subject matter expert guided:** An SME facilitates discussion, challenges unsupported claims, calibrates scoring, resolves ambiguity, and records decisions with the accountable participants.
- **Self guided:** An individual follows explicit activities, records assumptions and limitations, retains completion evidence, and escalates decisions that require an accountable owner.

The complete path is eight hours, organized as sixteen 30-minute modules. Breaks and offline evidence collection are scheduled separately. Modules may be completed individually, but their evidence dependencies should be respected.

| Module | 30-minute focus                 | Required completion evidence                                       |
| ------ | ------------------------------- | ------------------------------------------------------------------ |
| M01    | Frame the engagement            | Approved scope, outcomes, exclusions, and sponsor                  |
| M02    | Prepare access and evidence     | Collection identity, evidence location, cutoff, and limitations    |
| M03    | Hear the operating system       | Stakeholder perspectives, disputes, constraints, and gaps          |
| M04    | Collect governance and identity | Timestamped governance and identity evidence                       |
| M05    | Collect boards and repositories | Flow, hygiene, repository, policy, and review evidence             |
| M06    | Collect pipelines and releases  | Pipeline inventory, controls, deployment trace, and exceptions     |
| M07    | Collect quality and operations  | Quality, artifact, integration, cost, and resilience evidence      |
| M08    | Validate the evidence           | Validated observations, confidence, disputes, and follow-up owners |
| M09    | Score domain maturity           | Current and target scores with rationale and confidence            |
| M10    | Form and rate findings          | Evidence-linked, risk-rated findings and owner response            |
| M11    | Define desired outcomes         | Approved desired state, measures, constraints, and tradeoffs       |
| M12    | Design the feasible target      | Target design, decisions, exceptions, owners, and residual risk    |
| M13    | Build and prioritize changes    | Prioritized change catalog with owners and acceptance criteria     |
| M14    | Sequence the roadmap            | 30/60/90-day roadmap, dependencies, measures, and review gates     |
| M15    | Commit ownership                | Owner commitments, accepted risks, escalations, and review dates   |
| M16    | Close and sustain               | Completion record, exported assessment, and governance cadence     |

### Guiding principles

- **Outcomes before tools:** Recommendations must support a stated business, engineering, security, or operational outcome.
- **Evidence before opinion:** Every finding identifies its source, collection date, scope, and confidence.
- **Risk-based depth:** Investigate high-impact and high-exposure areas more deeply than low-risk areas.
- **Least privilege by default:** Access is granted through groups, scoped to need, reviewed periodically, and removed promptly.
- **Standardize the common path:** Prefer reusable processes, templates, policies, and controls while documenting justified exceptions.
- **Automate repeatable controls:** Use policy and automation where reliable enforcement is more effective than documentation alone.
- **Incremental change:** Sequence work to reduce immediate risk, establish foundations, and then improve flow and maturity.
- **No change without ownership:** Every accepted change has an accountable owner and verifiable completion criteria.

### Out of scope unless explicitly added

- Implementing changes during the assessment.
- Reviewing application source code beyond repository and pipeline configuration signals.
- Penetration testing or formal compliance certification.
- Redesigning product operating models that do not involve Azure DevOps.
- Assessing deployed Azure resources except where needed to understand identity, authorization, or deployment controls.

Microsoft Entra, Azure Resource Graph, Defender Vulnerability Management, and other security sources are supporting intelligence, not separate tenant-wide assessments. Collect from them only when the evidence answers an approved Azure DevOps scope question. Record the queried tenant, subscriptions, device groups, repositories, time window, licensing, permissions, and known coverage gaps so a partial result is never presented as complete.

## Engagement definition

### Workload profile

| Field                                                | Value                                               |
| ---------------------------------------------------- | --------------------------------------------------- |
| Organization name / URL                              |                                                     |
| Executive sponsor                                    |                                                     |
| Workspace and technical owners                       |                                                     |
| Security/compliance contact                          |                                                     |
| Assessment lead                                      |                                                     |
| Assessment start / evidence cutoff                   |                                                     |
| Business units and geographies                       |                                                     |
| Regulatory or contractual obligations                |                                                     |
| Total / active / inactive projects                   |                                                     |
| Total processes / custom processes / default process |                                                     |
| Total / active repositories                          |                                                     |
| Total users / guests / service identities            |                                                     |
| Primary delivery models                              | Scrum / Kanban / other                              |
| Azure DevOps services in use                         | Boards / Repos / Pipelines / Test Plans / Artifacts |
| Integrations and extensions                          |                                                     |
| Data classification present                          |                                                     |
| Known incidents or audit findings                    |                                                     |

### Scope

| Scope item                                   | Included? | Notes or exclusions | Approver |
| -------------------------------------------- | --------: | ------------------- | -------- |
| Organization settings and policies           |           |                     |          |
| Identity, groups, permissions, and licensing |           |                     |          |
| Projects and process configuration           |           |                     |          |
| Azure Boards                                 |           |                     |          |
| Azure Repos                                  |           |                     |          |
| Azure Pipelines and deployment environments  |           |                     |          |
| Azure Artifacts                              |           |                     |          |
| Azure Test Plans                             |           |                     |          |
| Extensions, hooks, and external integrations |           |                     |          |
| Audit, monitoring, retention, and cost       |           |                     |          |

### Participants and responsibilities

| Role                             | Named participant(s) | Responsibility                                                               |
| -------------------------------- | -------------------- | ---------------------------------------------------------------------------- |
| Executive sponsor                |                      | Approves outcomes, risk tolerance, and major investments                     |
| Workspace owner                  |                      | Confirms scope, grants read access, validates findings, accepts target state |
| Assessment lead                  |                      | Facilitates workshops, manages evidence, scores findings, drafts roadmap     |
| Project administrators           |                      | Explain local configuration, exceptions, and operating practices             |
| Engineering representatives      |                      | Describe developer and delivery experience                                   |
| Product/delivery representatives |                      | Describe planning, reporting, and traceability needs                         |
| Security/compliance              |                      | Interprets obligations and accepts residual risk                             |
| Platform/operations              |                      | Explains pipelines, environments, integrations, and support practices        |
| Change owners                    |                      | Estimate, implement, and verify accepted changes                             |

### Assumptions, constraints, and dependencies

| ID    | Type       | Statement | Impact | Owner | Resolution or decision date |
| ----- | ---------- | --------- | ------ | ----- | --------------------------- |
| A-001 | Assumption |           |        |       |                             |
| C-001 | Constraint |           |        |       |                             |
| D-001 | Dependency |           |        |       |                             |

## Assessment method

### Phases

1. **Discover:** Confirm scope, stakeholders, desired outcomes, constraints, and evidence access.
2. **Collect:** Conduct interviews and export reproducible quantitative data.
3. **Validate:** Reconcile interviews with evidence; review anomalies and exceptions with owners.
4. **Assess:** Score maturity and risk, and identify strengths, concerns, and root causes.
5. **Design:** Define desired and ideal states, including intentional exceptions.
6. **Plan:** Catalog, score, sequence, assign, and approve changes.
7. **Review:** Establish measures, governance cadence, and evidence required to close each action.

### Assessment domains

| Domain                                | Questions the assessment must answer                                                                                    |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Governance and organization design    | Is ownership clear? Are projects, processes, policies, extensions, and exceptions governed consistently?                |
| Identity, access, and licensing       | Who can access what, why, and for how long? Are privileged and dormant identities controlled? Are licenses appropriate? |
| Boards and delivery management        | Do work item types, states, fields, paths, and reporting support how teams plan and deliver?                            |
| Repositories and source control       | Are repositories owned, active, protected, discoverable, and managed with consistent review controls?                   |
| Pipelines, environments, and releases | Are builds and deployments reproducible, secure, traceable, appropriately approved, and reusable?                       |
| Artifacts and dependencies            | Are feeds, permissions, upstream sources, retention, provenance, and package lifecycle controlled?                      |
| Test strategy and quality             | Can teams trace requirements to tests and releases? Are quality gates proportionate and results retained?               |
| Security, audit, and compliance       | Are secrets, tokens, service connections, audit events, retention, and security alerts governed and reviewed?           |
| Operations, resilience, and economics | Is the service supportable? Are integrations, limits, continuity, consumption, and improvement measures understood?     |

### Evidence standard

Every conclusion must be traceable to an evidence item. Store raw exports in an access-controlled location; do not commit secrets, personal data, tokens, audit payloads, or sensitive security findings to this repository.

| Field                 | Required content                                                         |
| --------------------- | ------------------------------------------------------------------------ |
| Evidence ID           | Stable identifier, for example `E-ORG-001`                               |
| Domain                | Assessment domain supported by the evidence                              |
| Source                | Interview, UI, REST API, Graph API, CLI, policy export, or report        |
| Scope                 | Organization, project, repository, pipeline, feed, or sampled population |
| Collected at          | UTC timestamp and assessment cutoff date                                 |
| Collector             | Person or automation identity                                            |
| Storage location      | Restricted path or immutable report link                                 |
| Method/version        | Command, script, API version, query, filters, and pagination behavior    |
| Completeness          | Complete, sampled, unavailable, or not applicable                        |
| Confidence            | High, medium, or low, with reason                                        |
| Sensitivity/retention | Classification, access restrictions, and disposal date                   |

### Evidence catalog

| Evidence ID | Domain | Description | Source and scope | Collected at | Completeness | Confidence | Restricted location |
| ----------- | ------ | ----------- | ---------------- | ------------ | ------------ | ---------- | ------------------- |
|             |        |             |                  |              |              |            |                     |

### Sampling and limitations

Record all exclusions, failed queries, permission gaps, API limitations, pagination limits, and sampling choices. If full collection is impractical, select a risk-based sample that includes business-critical projects, highly privileged projects, high-change repositories, production deployment pipelines, custom processes, and inactive assets. Findings based on samples must not be represented as organization-wide facts without validation.

| Limitation ID | Affected evidence/domain | Limitation | Potential bias | Mitigation | Owner |
| ------------- | ------------------------ | ---------- | -------------- | ---------- | ----- |
| L-001         |                          |            |                |            |       |

### Maturity scale

Score the observed state, not the team's aspiration.

| Score | Level      | Definition                                                                           |
| ----: | ---------- | ------------------------------------------------------------------------------------ |
|     0 | Unknown    | No reliable evidence is available or the capability is not understood                |
|     1 | Ad hoc     | Practices are informal, reactive, person-dependent, or inconsistent                  |
|     2 | Repeatable | Some teams repeat a practice, but standards, coverage, or ownership are incomplete   |
|     3 | Defined    | A documented standard exists, ownership is clear, and adoption is broadly consistent |
|     4 | Measured   | Compliance and outcomes are measured; exceptions and drift are actively managed      |
|     5 | Optimizing | Feedback and automation continuously improve the standard and user experience        |

### Finding risk rating

Rate likelihood and impact from 1 (low) to 5 (very high).

$$\text{Risk score} = \text{Likelihood} \times \text{Impact}$$

| Score | Rating   | Expected response                                               |
| ----: | -------- | --------------------------------------------------------------- |
| 20-25 | Critical | Escalate immediately; assign an owner and containment action    |
| 12-19 | High     | Include in the first 30 days unless risk is explicitly accepted |
|  6-11 | Medium   | Plan within 60-90 days based on dependencies and capacity       |
|   1-5 | Low      | Add to the improvement backlog or accept with rationale         |

## Current-state assessment

### Qualitative data collection

Interview the workspace owner and a representative cross-section of administrators, developers, product/delivery staff, security, and operations. Separate observed facts from perceptions and preserve meaningful disagreement.

#### Owner and administrator interview

- What business outcomes must Azure DevOps enable over the next 12-24 months?
- Which projects, repositories, pipelines, feeds, and integrations are business-critical?
- Who owns organization-wide policy, project creation, process customization, extensions, identity lifecycle, and exceptions?
- Which controls are mandatory, and which risks can the organization accept?
- Where does configuration intentionally differ between teams, and why?
- What recent incidents, audit findings, failed releases, access issues, or cost surprises should inform the assessment?
- Which manual activities consume the most administrative or engineering time?
- How are inactive users, projects, repositories, pipelines, feeds, tokens, and service connections retired?
- How are platform changes communicated, tested, approved, and rolled back?
- What reporting is trusted today, and which decisions cannot be made from available data?
- What delivery deadlines, budget limits, skills gaps, or organizational changes constrain the roadmap?
- What would make this engagement demonstrably successful after 90 days and after one year?

#### Practitioner interview

- How easily can a new team member find work, code, documentation, build instructions, and deployment history?
- Where do branch, review, build, test, or approval controls slow delivery without reducing risk?
- Which pipeline and repository patterns are reused, copied, or rebuilt by each team?
- How are secrets, dependencies, environments, releases, failed jobs, and production incidents handled?
- Can a production change be traced from request through code review, build, tests, approvals, deployment, and monitoring?
- Where do teams leave Azure DevOps for planning, reporting, approvals, or evidence, and why?
- Which local practices work well and should become an organization standard?

#### Desired outcomes and pain points

| Interview ID | Stakeholder/role | Outcome or pain point | Evidence/examples | Frequency | Impact | Current workaround | Desired measure |
| ------------ | ---------------- | --------------------- | ----------------- | --------- | ------ | ------------------ | --------------- |
| Q-001        |                  |                       |                   |           |        |                    |                 |

### Quantitative data collection

Collect data through read-only REST API, Graph API, Azure CLI/Azure DevOps CLI, and approved reporting tools. Record API versions and follow continuation tokens so counts are complete. Normalize timestamps to UTC. Capture both total counts and ratios; raw counts alone can hide risk.

#### Organization, governance, and identity

| Dataset                | Minimum measures                                                                                                                         |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Organization settings  | Region, owner, tenant connection, security/policy settings, public project and guest posture, authentication posture, retention settings |
| Projects               | ID, name, state, visibility, owner, process, created date, last meaningful activity, service usage, lifecycle status                     |
| Processes              | System/inherited processes, default, custom work item types/fields/states/rules, project adoption                                        |
| Users and licenses     | Entitlement, origin, status, last access where available, access level, memberships, guest/external status                               |
| Groups and permissions | Group type, scope, membership, nesting, direct assignments, privileged roles, explicit allow/deny entries                                |
| Credentials            | PAT governance settings and available inventory metadata; SSH/alternate credential posture without collecting secret values              |
| Extensions and hooks   | Publisher, version, permissions/scopes, owner, installations, service hooks, activity, approval status                                   |
| Audit                  | Availability, retention/export, privileged and permission changes, token events, policy bypasses, review ownership                       |

Key ratios include inactive-user rate, guest-user rate, users with direct permissions, privileged identities without an owner, paid-license utilization, inactive-project rate, and unapproved-extension rate.

#### Boards and delivery

| Dataset            | Minimum measures                                                                                       |
| ------------------ | ------------------------------------------------------------------------------------------------------ |
| Team configuration | Teams, area/iteration paths, backlogs, board columns, WIP limits, delivery cadence                     |
| Work items         | Type, state, age, owner, paths, links, tags, created/changed/closed dates                              |
| Flow               | Lead time, cycle time, throughput, work item age, blocked time where modeled, carryover, reopened rate |
| Hygiene            | Unassigned, stale, orphaned, invalid-state, missing-acceptance-criteria, and unlinked work items       |
| Queries/dashboards | Shared/personal assets, owners, usage, stale assets, critical reporting dependencies                   |

Segment measures by project/team and work item type. Do not compare teams until definitions of start, done, and item size are aligned.

#### Repositories and source control

| Dataset              | Minimum measures                                                                                                     |
| -------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Repository inventory | Project, repository, owner, default branch, size, fork/disabled status, last commit, contributors, criticality       |
| Branches             | Active/stale branches, naming patterns, permissions, force-push/delete controls, long-lived branch age               |
| Policies             | Reviewers, required groups, comment resolution, linked work, build validation, status checks, merge strategy, bypass |
| Pull requests        | Volume, age, completion/abandonment, review time, reviewers, policy bypasses, failed validations                     |
| Repository hygiene   | README, ownership metadata, contribution/build guidance, default branch consistency, archive candidates              |
| Security             | Advanced Security enablement where licensed, active secret/dependency/code alerts by severity and age, owners        |

Key ratios include protected-default-branch coverage, repositories with a named owner, stale-repository rate, pull requests completed with bypass, and high-severity alerts outside their remediation target.

#### Pipelines, environments, and releases

| Dataset                 | Minimum measures                                                                                          |
| ----------------------- | --------------------------------------------------------------------------------------------------------- |
| Pipeline inventory      | YAML/classic type, repository/path, owner, state, last run, schedule, pool, criticality, production reach |
| Runs                    | Success/failure/cancellation, duration percentiles, queue time, retries, last run, retention              |
| Templates               | Shared template usage, pinned references, duplication, versioning, required-template checks               |
| Agents and pools        | Hosted/self-hosted, owner, OS/image, version, capacity, utilization, network reach, patching process      |
| Service connections     | Type, scope, owner, authentication, federation/managed identity use, sharing, last use, checks            |
| Variables and secrets   | Variable groups, Key Vault linkage, access scope, owner, last use; never export secret values             |
| Environments and checks | Resources, owners, approvals, branch controls, locks, checks, production coverage                         |
| Deployments             | Frequency, failure rate, lead time, recovery evidence, traceability, production approval path             |
| Legacy assets           | Classic definitions, task versions, deprecated images/tasks, inactive definitions, migration candidates   |

Key ratios include YAML adoption, pipelines with named owners, protected production deployments, broadly shared service connections, federated identity adoption, stale pipeline rate, and deployment success rate.

#### Artifacts, tests, integrations, and economics

| Dataset      | Minimum measures                                                                                                     |
| ------------ | -------------------------------------------------------------------------------------------------------------------- |
| Artifacts    | Feeds, owners, permissions, views, upstreams, package counts/types, storage, retention, stale/deprecated packages    |
| Test Plans   | Plans/suites/cases, owners, lifecycle, automation/result history, requirement/build linkage, flaky/failing trends    |
| Integrations | Hooks, webhooks, tools, identities, owners, data flows, last use, failure handling, criticality                      |
| Usage/cost   | Paid access, parallel jobs, pipeline minutes, artifact storage, Advanced Security coverage, extension charges, waste |
| Resilience   | Critical dependencies, support path, export/continuity needs, recovery assumptions, vendor/service limits            |

## Data collection runbook

<!-- markdownlint-disable MD010 MD024 -->

This runbook covers Azure DevOps **Services**. For Azure DevOps Server, replace the host and use an API version supported by that server release. Run collection with an approved, read-only identity. Some organization-level inventory requires Organization Owner, Project Collection Administrator, Project Collection Build Service Accounts, or Auditing permissions even though every request below is read-only.

### Collection prerequisites

1. Install [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli-windows) on Windows with `winget install -e --id Microsoft.AzureCLI` and reopen the terminal.
2. Verify the installation with `az version`.
3. Install or update the Azure DevOps extension:

   ```powershell
   az extension add --name azure-devops --upgrade
   az extension show --name azure-devops --output table
   ```

4. Sign in to the Microsoft Entra tenant connected to the Azure DevOps organization. Interactive Entra authentication is preferred to a PAT for this assessment:

   ```powershell
   az login --tenant '<tenant-id>'
   az account show --output table
   ```

5. Define the assessment variables and create a restricted evidence directory outside this repository:

   ```powershell
   $Organization = '<organization-name>'
   $OrganizationUrl = "https://dev.azure.com/$Organization"
   $Project = '<project-name-or-id>'
   $CollectedAt = (Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ')
   $EvidenceRoot = Join-Path $HOME "ado-tune-up-evidence/$Organization/$CollectedAt"
   New-Item -ItemType Directory -Force -Path $EvidenceRoot | Out-Null

   az devops configure --defaults organization=$OrganizationUrl project=$Project
   az devops configure --list
   ```

6. Confirm access before bulk collection:

   ```powershell
   az devops project list --organization $OrganizationUrl --top 1 --output table
   ```

7. Record the CLI versions, identity, UTC collection time, organization, project scope, output path, and any permission failures in the quantitative collection run log.

### Packaged read-only collector

[`scripts/collect-ado-evidence.ps1`](scripts/collect-ado-evidence.ps1) packages continuation handling, raw-page retention, retry behavior, run metadata, alternate Azure DevOps hosts, project-safe URL substitution, and evidence-catalog generation. The starter manifest contains 25 reviewed organization and project inventory datasets. Copy [`scripts/collection-manifest.example.json`](scripts/collection-manifest.example.json) to an engagement-controlled location before adding endpoints that require resource IDs, POST bodies, time windows, licensed features, or engagement-specific sampling.

```powershell
$EvidenceRoot = Join-Path $HOME 'ado-tune-up-evidence/contoso/2026-01-31T120000Z'
& .\scripts\collect-ado-evidence.ps1 `
	-Organization 'contoso' `
	-Project 'Platform Engineering' `
	-OutputPath $EvidenceRoot `
	-ManifestPath .\scripts\collection-manifest.example.json
```

The collector obtains a Microsoft Entra token from the signed-in Azure CLI process and keeps it in memory. It writes one raw JSON file per response page, `collection-run.json` with status, page and record counts, and `evidence-catalog.json` with workspace-compatible metadata. Omit `-Project` for organization-only collection; project datasets are then recorded as `Unavailable` with an explicit limitation instead of being silently omitted. Optional endpoints that deny access are handled the same way, while the required project inventory fails the run. Repeat collection for each in-scope project.

Inspect service-connection and variable-group responses for sensitive fields even when Azure DevOps masks secret values. Move raw output to the approved restricted location and update catalog locations before sharing assessment exports.

Use **Import** in the workspace to merge `evidence-catalog.json`; matching evidence IDs are updated and unrelated assessment artifacts are retained. Use schema-v4 JSON, not the evidence catalog, to restore a complete assessment. Validate collector behavior offline after changes:

```powershell
& .\scripts\test-collector.ps1
```

### Authentication and REST request pattern

Use the Azure DevOps first-party application ID `499b84ac-1321-427f-aa17-267ca6975798` to obtain an Entra access token. Keep the token only in memory and never write headers or tokens to evidence files.

```powershell
$AccessToken = az account get-access-token `
	--resource 499b84ac-1321-427f-aa17-267ca6975798 `
	--query accessToken --output tsv
$Headers = @{ Authorization = "Bearer $AccessToken" }

function Save-AdoResponse {
	param(
		[Parameter(Mandatory)] [string] $Uri,
		[Parameter(Mandatory)] [string] $EvidenceId
	)

	$OutputPath = Join-Path $EvidenceRoot "$EvidenceId.json"
	Invoke-RestMethod -Method Get -Uri $Uri -Headers $Headers |
		ConvertTo-Json -Depth 100 |
		Set-Content -Path $OutputPath -Encoding utf8
}
```

Every request must specify an API version. The examples use `7.2` where released and a precise preview API where the resource remains in preview. Revalidate preview versions against the [Azure DevOps REST API reference](https://learn.microsoft.com/rest/api/azure/devops/) before each engagement.

List APIs are paginated. Do not treat one page as a complete inventory, and do not assume every endpoint uses OData-style parameter names:

1. Request the documented maximum `$top` or `top` value.
2. Inspect the `x-ms-continuationtoken` response header or response body's `continuationToken` property.
3. Repeat the request with the endpoint's documented `continuationToken` parameter until no token is returned.
4. Preserve each raw page as `<evidence-id>-page-001.json`, `<evidence-id>-page-002.json`, and so on.
5. Merge only after preserving raw pages; record page count and merged record count in the run log.

`Save-AdoResponse` is suitable for single-page or deliberately sampled calls. Use `Invoke-WebRequest` when response headers are required for pagination. Treat HTTP `401` as an authentication failure and `403` or a permission-hidden `404` as an evidence limitation; do not silently record an empty dataset.

Some operations use `$top`/`$skip`; others use bare `top` and `continuationToken`. Copy the parameter names from that operation's API reference exactly. For example, Advanced Security alerts and environment deployment records use bare `top`, while Artifact packages use `$top` and `$skip`.

### Evidence naming and collection order

Use these prefixes so the evidence catalog remains traceable:

| Prefix  | Domain                                       |
| ------- | -------------------------------------------- |
| `E-ORG` | Organization settings, projects, processes   |
| `E-ID`  | Users, groups, permissions, credentials      |
| `E-BRD` | Boards, teams, work items, queries           |
| `E-REP` | Repositories, branches, policies, PRs        |
| `E-PIP` | Pipelines, agents, environments, connections |
| `E-ART` | Feeds and packages                           |
| `E-TST` | Test plans, runs, and results                |
| `E-INT` | Extensions, hooks, and integrations          |
| `E-AUD` | Audit and security events                    |
| `E-CST` | Licensing, usage, and cost                   |

Collect in this order: organization and projects, users/groups, project-specific resources, cross-resource relationships, then manual-only settings. This avoids orphaned IDs and makes later joins reproducible.

### Organization, governance, and identity collection

#### Organization settings

##### API and CLI

The organization policy endpoint exposes available organization policies. Organization region, tenant connection, owners, billing, retention, and some security toggles require UI verification.

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/OrganizationPolicy/Policies?api-version=7.2-preview.1" `
	-EvidenceId 'E-ORG-001-organization-policies'
```

##### Manual steps

1. Open `https://dev.azure.com/<organization>/_settings/organizationOverview`.
2. In **Organization settings > Overview**, record organization name, URL, geography, organization owner, and tenant connection.
3. Open **Organization settings > Microsoft Entra** and record the connected tenant and guest-access posture. Do not change the tenant.
4. Open **Organization settings > Policies** and record each policy's displayed state, scope, and exception status.
5. Open **Organization settings > Settings** and record project visibility, SSH/deprecated authentication, pipeline, work item, and retention-related settings shown for the organization.
6. Capture the page URL, UTC time, collector, and redacted screenshots as `E-ORG-002`; note settings not visible to the collector.

#### Projects and teams

##### API and CLI

```powershell
az devops project list --organization $OrganizationUrl --state-filter all --top 1000 --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ORG-010-projects.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/projects?stateFilter=all&`$top=100&api-version=7.2" `
	-EvidenceId 'E-ORG-011-projects-api-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/teams?mine=false&`$top=100&api-version=7.2" `
	-EvidenceId 'E-ORG-012-teams-page-001'

az devops team list --organization $OrganizationUrl --project $Project --top 1000 --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ORG-013-project-teams.json') -Encoding utf8
```

The APIs do not provide a universal "last meaningful project activity" field. Derive it from the latest work-item change, repository commit/PR, pipeline run, package publication, and test run collected later. Document the selected activity types and inactivity threshold.

##### Manual steps

1. Open **Organization settings > Projects**.
2. Include deleted projects if the recycle-bin view is available.
3. For each project, record visibility, process, project administrator group, business owner, criticality, service usage, and intended lifecycle status.
4. Open each sampled project and inspect **Project settings > Teams** to validate team ownership and area/iteration assignments.
5. Reconcile the UI count to the fully paginated API count and record differences.

#### Processes and process customization

##### API and CLI

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/work/processes?`$expand=projects&api-version=7.2-preview.2" `
	-EvidenceId 'E-ORG-020-processes'
```

For every inherited process ID returned, collect work item types and then fields, states, rules, and behaviors for each work item type:

```text
GET https://dev.azure.com/{organization}/_apis/work/processes/{processId}/workitemtypes?api-version=7.2-preview.2
GET https://dev.azure.com/{organization}/_apis/work/processes/{processId}/workitemtypes/{witRefName}/fields?api-version=7.2-preview.2
GET https://dev.azure.com/{organization}/_apis/work/processes/{processId}/workitemtypes/{witRefName}/states?api-version=7.2-preview.1
GET https://dev.azure.com/{organization}/_apis/work/processes/{processId}/workitemtypes/{witRefName}/rules?api-version=7.2-preview.2
GET https://dev.azure.com/{organization}/_apis/work/processes/{processId}/behaviors?api-version=7.2-preview.2
```

##### Manual steps

1. Open **Organization settings > Boards > Process**.
2. Record the default process and every system/inherited process.
3. Open each inherited process and inventory customized work item types, fields, states, rules, backlog levels, and projects using it.
4. Ask the process owner for the business reason and active consumers of each customization.
5. Mark unsupported XML/custom hosted-process details as not applicable for Azure DevOps Services.

#### Users, access levels, and licenses

##### API and CLI

User entitlements are served from `vsaex.dev.azure.com`; Graph identities are served from `vssps.dev.azure.com`. Collect both because entitlement and identity metadata differ.

```powershell
az devops user list --organization $OrganizationUrl --top 10000 --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ID-001-user-entitlements-cli.json') -Encoding utf8

Save-AdoResponse `
	-Uri "https://vsaex.dev.azure.com/$Organization/_apis/userentitlements?top=100&api-version=7.2-preview.3" `
	-EvidenceId 'E-ID-002-user-entitlements-page-001'

Save-AdoResponse `
	-Uri "https://vssps.dev.azure.com/$Organization/_apis/graph/users?api-version=7.2-preview.1" `
	-EvidenceId 'E-ID-003-graph-users-page-001'
```

The available last-access fields vary by API and license. Do not infer activity solely from membership. Correlate user identity to commits, pull requests, work item changes, pipeline requests, and audit events; state the chosen inactivity window.

##### Manual steps

1. Open **Organization settings > Users**.
2. Add or expose columns for access level, license source, status, last access where available, and project entitlement.
3. Export the list if the UI offers export; otherwise validate counts and sampled records against `E-ID-001` and `E-ID-002`.
4. Identify guests, pending invitations, disabled identities, service principals/managed identities, users with paid access, and users whose access is inherited through group rules.
5. Ask identity owners to classify each non-human identity and assign an accountable owner; do not classify from display name alone.

#### Groups, memberships, and permissions

##### API and CLI

```powershell
az devops security group list --organization $OrganizationUrl --scope organization --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ID-010-groups-cli.json') -Encoding utf8

Save-AdoResponse `
	-Uri "https://vssps.dev.azure.com/$Organization/_apis/graph/groups?api-version=7.2-preview.1" `
	-EvidenceId 'E-ID-011-graph-groups-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/securitynamespaces?api-version=7.2" `
	-EvidenceId 'E-ID-012-security-namespaces'
```

For each group descriptor, collect direct members and parent groups. Repeat for nested groups until no unresolved group descriptors remain:

```powershell
$GroupDescriptor = '<graph-group-descriptor>'

az devops security group membership list --organization $OrganizationUrl `
	--id $GroupDescriptor --relationship members --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ID-013-group-members.json') -Encoding utf8

az devops security group membership list --organization $OrganizationUrl `
	--id $GroupDescriptor --relationship member-of --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-ID-014-group-parents.json') -Encoding utf8
```

Permission inventory requires security namespace IDs and resource tokens. Query only approved namespaces and tokens; a collection-wide ACL dump can contain sensitive identity and resource topology.

```text
GET https://dev.azure.com/{organization}/_apis/accesscontrollists/{securityNamespaceId}?token={resourceToken}&includeExtendedInfo=true&recurse=true&api-version=7.2
```

Decode permission bitmasks using the `actions` returned by `securitynamespaces`; do not interpret numeric bits without the namespace definition.

##### Manual steps

1. Open **Organization settings > Permissions** and record members of Project Collection Administrators, Project Collection Service Accounts, Build Administrators, and other privileged collection groups.
2. In each sampled project, open **Project settings > Permissions** and identify direct user grants, explicit denies, custom groups, and nested Microsoft Entra groups.
3. Open **Project settings > Repositories**, **Agent pools**, **Service connections**, **Environments**, and **Library** security views because these use separate security namespaces.
4. For every privileged group, record owner, business purpose, approval source, review date, and whether membership is direct or inherited.
5. Have a second reviewer validate decoded ACLs against at least one UI permission screen before using the decoder for organization-wide metrics.

#### Credential governance

##### Authentication decision and handling rules

Choose the least privileged, shortest-lived mechanism supported by the workload. Use interactive Microsoft Entra authentication for human collection, managed identity for Azure-hosted automation, and workload identity federation or a purpose-specific service principal for other automation. Use a PAT only when the integration cannot use Microsoft Entra authentication. Never use a personal user account as an unattended service identity.

- Scope each non-human identity to one owned workload and environment. Grant only the Azure DevOps permissions, Microsoft Graph permissions, Azure RBAC roles, and resource scope required by the documented operation.
- Keep access tokens in process memory. Do not place tokens or secrets in command arguments, source files, pipeline YAML, logs, exports, tickets, or workshop records. Redact request headers and sensitive response fields before retaining evidence.
- Prefer secretless federation or managed identity. When a secret or certificate is temporarily unavoidable, store it in an approved secret store, set the shortest practical lifetime, name an owner and backup owner, monitor expiration, and test rotation before the previous credential expires.
- Record credential type, owner, purpose, scope, creation and expiry dates, last use, storage system, rotation procedure, and revocation procedure without recording the credential value.
- On suspected exposure, stop collection, revoke or disable the credential, preserve approved audit evidence, notify the security owner, identify affected scope, replace the credential through the normal change path, and verify that the old credential no longer authenticates.
- Review non-human identities and remaining credentials at least quarterly and after an owner, workload, environment, or permission change. Remove unused credentials rather than renewing them.

##### API and CLI

Do not attempt to export secret values. PAT governance is assessed through organization policies and audit events. Individual users can review their own token inventory at `https://dev.azure.com/<organization>/_usersSettings/tokens`; organization administrators do not receive token values.

Use `E-ORG-001` for PAT policies and the audit collection later in this runbook for token creation, modification, revocation, and authentication events available to the collector. Record any use of preview PAT administration APIs as a separate, explicitly approved collection because permissions and availability differ by tenant.

##### Manual steps

1. Open **Organization settings > Policies** and record PAT creation, lifespan, scope, and allow-list policies.
2. Ask sampled users and service owners to open **User settings > Personal access tokens** and attest to owner, purpose, scope, expiration, and last-needed date without sharing token values.
3. Open **User settings > SSH public keys** for sampled identities and record only key metadata shown by the UI, never private keys.
4. Record obsolete alternate/basic credential settings from organization policies.
5. Treat unknown owner, broad scope, excessive lifetime, or use by automation as a finding candidate, subject to owner validation.
6. For every credential retained by exception, record why Entra authentication, managed identity, or workload identity federation is not feasible, the migration owner and target date, and the tested revocation path.

#### Extensions, service hooks, and audit

##### API and CLI

```powershell
Save-AdoResponse `
	-Uri "https://extmgmt.dev.azure.com/$Organization/_apis/extensionmanagement/installedextensions?includeDisabledExtensions=true&includeErrors=true&assetTypes=Microsoft.VisualStudio.Services.Manifest&api-version=7.2-preview.1" `
	-EvidenceId 'E-INT-001-installed-extensions'

Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/hooks/subscriptions?api-version=7.2" `
	-EvidenceId 'E-INT-002-service-hook-subscriptions-page-001'

$AuditStart = '<yyyy-MM-ddTHH:mm:ssZ>'
$AuditEnd = '<yyyy-MM-ddTHH:mm:ssZ>'
Save-AdoResponse `
	-Uri "https://auditservice.dev.azure.com/$Organization/_apis/audit/auditlog?startTime=$AuditStart&endTime=$AuditEnd&batchSize=200&api-version=7.2-preview.1" `
	-EvidenceId 'E-AUD-001-audit-page-001'
```

Follow the audit response continuation token until exhausted. Split long periods into bounded windows, preserve raw pages, and record the organization's available retention period. Audit access requires an authorized organization role or explicit audit permission.

##### Manual steps

1. Open **Organization settings > Extensions** and record publisher, extension, version, state, requested scopes, owner, business justification, and paid status.
2. Open **Project settings > Service hooks** for every project and record subscription type, publisher/consumer, owner, enabled state, target classification, and failure history where displayed.
3. Open **Organization settings > Auditing**; set the exact UTC start/end period and export available events.
4. Review filters for permission changes, group membership, project/repository/pipeline policy changes, PAT lifecycle, service connection changes, extension changes, and bypass-related actions.
5. Record audit retention/export configuration and the team responsible for recurring review.

### Boards and delivery collection

#### Team configuration, area paths, and iterations

##### API and CLI

Run these calls for every team returned by the project team inventory:

```powershell
$Team = '<team-name-or-id>'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/$Team/_apis/work/teamsettings?api-version=7.2-preview.1" `
	-EvidenceId 'E-BRD-001-team-settings'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/$Team/_apis/work/teamsettings/teamfieldvalues?api-version=7.2" `
	-EvidenceId 'E-BRD-002-team-area-paths'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/$Team/_apis/work/teamsettings/iterations?api-version=7.2" `
	-EvidenceId 'E-BRD-003-team-iterations'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/$Team/_apis/work/boards?api-version=7.2" `
	-EvidenceId 'E-BRD-004-boards'
```

For each board ID, collect its columns, rows/swimlanes, and card settings:

```text
GET https://dev.azure.com/{organization}/{project}/{team}/_apis/work/boards/{boardId}/columns?api-version=7.2
GET https://dev.azure.com/{organization}/{project}/{team}/_apis/work/boards/{boardId}/rows?api-version=7.2
GET https://dev.azure.com/{organization}/{project}/{team}/_apis/work/boards/{boardId}/cardsettings?api-version=7.2
```

Classification node APIs provide the complete project hierarchy:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/wit/classificationnodes/areas?$depth=10&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/wit/classificationnodes/iterations?$depth=10&api-version=7.2
```

##### Manual steps

1. Open **Project settings > Teams**, select each team, and verify its administrators and default area/iteration.
2. Open **Project settings > Project configuration > Areas** and **Iterations**; record node owners, active/inactive status, dates, and teams sharing each node.
3. Open **Boards > Boards**, select the team and backlog level, then record columns, state mappings, WIP limits, split columns, swimlanes, card rules, and definition-of-done text.
4. Open **Boards > Backlogs** and record enabled backlog levels, team backlog ownership, and portfolio relationships.
5. Flag shared area paths, missing iteration dates, overlapping cadences, unmapped states, or WIP limits that exist but are routinely bypassed for owner review.

#### Work items, flow, and hygiene

##### API and CLI

Use WIQL to select IDs, then retrieve fields and relations in batches of at most 200 IDs. The query below is a baseline; add organization-specific work item types and fields before collection.

```powershell
$Wiql = @'
SELECT [System.Id]
FROM WorkItems
WHERE [System.TeamProject] = @project
	AND [System.ChangedDate] >= @Today - 365
ORDER BY [System.ChangedDate] DESC
'@

az boards query --organization $OrganizationUrl --project $Project --wiql $Wiql --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-BRD-010-work-item-query.json') -Encoding utf8
```

Equivalent API sequence:

```text
POST https://dev.azure.com/{organization}/{project}/_apis/wit/wiql?timePrecision=true&api-version=7.2
Body: {"query":"SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = @project AND [System.ChangedDate] >= @Today - 365"}

POST https://dev.azure.com/{organization}/{project}/_apis/wit/workitemsbatch?api-version=7.2-preview.1
```

`workitemsbatch` is a POST with a JSON body containing `ids`, `fields`, and `$expand: "Relations"`; it is read-only despite using POST. Collect at least these fields:

```text
System.Id, System.WorkItemType, System.Title, System.State, System.Reason,
System.AssignedTo, System.CreatedDate, System.ChangedDate, System.ClosedDate,
System.AreaPath, System.IterationPath, System.Parent, System.Tags,
Microsoft.VSTS.Common.ActivatedDate, Microsoft.VSTS.Common.ResolvedDate,
Microsoft.VSTS.Common.ClosedDate, Microsoft.VSTS.Common.StateChangeDate
```

Collect revision history only for sampled items needed to calculate blocked time, state-transition time, or reopened rate:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/wit/workItems/{id}/revisions?$top=200&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/wit/workItems/{id}/updates?$top=200&api-version=7.2
```

##### Manual steps

1. Agree with each team which state starts and stops lead time and cycle time; record the definitions before calculating metrics.
2. In **Boards > Queries**, create a temporary private query for the same period and compare its count with WIQL output.
3. Validate at least ten work items across types and states, including parent/child, repository, build, deployment, and test links.
4. Define hygiene rules explicitly, such as stale after 90 unchanged days, unassigned when active without an owner, and orphaned when required hierarchy links are absent.
5. Calculate throughput by completion period, age for open items, carryover across iterations, reopen transitions from revisions, and lead/cycle time using the agreed state dates.
6. Record missing fields, inconsistent state use, deleted identities, and custom blocked-state modeling as data quality limitations.

#### Queries and dashboards

##### API and CLI

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/wit/queries?`$depth=2&`$expand=all&api-version=7.2" `
	-EvidenceId 'E-BRD-020-shared-queries'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/dashboard/dashboards?api-version=7.2-preview.3" `
	-EvidenceId 'E-BRD-021-dashboards'
```

For each dashboard ID, call `GET .../_apis/dashboard/dashboards/{dashboardId}?api-version=7.2-preview.3` to capture widgets and settings. Personal queries are private to their owners and are not a reliable organization-wide dataset; inventory shared queries and document this limitation.

##### Manual steps

1. Open **Boards > Queries > Shared Queries** and identify folder/query owners, consumers, last-known use, and reporting dependencies.
2. Open **Overview > Dashboards**, select each team, and record dashboard owner, intended audience, widgets, broken widgets, and duplicate dashboards.
3. Ask report owners which queries or dashboards drive governance, compliance, release, or executive decisions.
4. Mark assets with no owner or confirmed consumer as retirement candidates, not automatically unused.

### Repositories and source-control collection

#### Repository inventory, branches, and hygiene

##### API and CLI

```powershell
az repos list --organization $OrganizationUrl --project $Project --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-REP-001-repositories.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/git/repositories?includeLinks=true&includeAllUrls=true&includeHidden=true&api-version=7.2" `
	-EvidenceId 'E-REP-002-repositories-api'

$Repository = '<repository-name-or-id>'
az repos ref list --organization $OrganizationUrl --project $Project `
	--repository $Repository --filter heads/ --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-REP-003-branches.json') -Encoding utf8
```

For each repository, collect refs and recent commits:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/refs?filter=heads/&peelTags=true&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits?searchCriteria.fromDate={UTC}&searchCriteria.toDate={UTC}&$top=1000&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/items?scopePath=/&recursionLevel=OneLevel&includeContentMetadata=true&api-version=7.2
```

Use the latest commit across branches and pull requests as activity evidence. Repository owner and criticality are governance metadata, not API fields; obtain owner attestation.

##### Manual steps

1. Open **Repos > Files**, select each repository, and verify default branch, README, contribution/build guidance, ownership metadata, and repository purpose.
2. Open **Project settings > Repositories**, select the repository, and inspect security for direct grants, force push, delete repository, manage permissions, and bypass policies.
3. Ask the business/technical owner to classify criticality, data sensitivity, lifecycle state, and archive eligibility.
4. Define stale repository and stale branch thresholds before calculating rates; exclude mirrors, templates, and intentionally dormant recovery repositories where approved.
5. Inspect repository size and large/binary content only through metadata or approved tools; do not download sensitive repository contents solely for this assessment.

#### Branch policies and pull requests

##### API and CLI

```powershell
az repos policy list --organization $OrganizationUrl --project $Project --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-REP-010-policy-configurations.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/policy/configurations?api-version=7.2" `
	-EvidenceId 'E-REP-011-policy-configurations-api'

az repos pr list --organization $OrganizationUrl --project $Project `
	--repository $Repository --status all --top 1000 --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-REP-012-pull-requests.json') -Encoding utf8
```

For complete PR details and review timing, page the API and collect threads/reviewers for sampled or high-risk PRs:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/git/pullrequests?searchCriteria.status=all&searchCriteria.repositoryId={repositoryId}&$top=1000&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/reviewers?api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/threads?api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/statuses?api-version=7.2
```

##### Manual steps

1. Open **Project settings > Repositories > Policies**, select each default and release branch, and record minimum reviewers, required reviewers, comment resolution, linked work items, build validation, status checks, merge strategies, and path filters.
2. Open **Repos > Pull requests**, filter by repository and status, and validate API counts for the assessment period.
3. Sample completed, abandoned, bypassed, long-running, and failed-validation PRs; record created/closed times, votes, comments, checks, completion options, and bypass reason.
4. Inspect repository and branch security for **Bypass policies when completing pull requests** and **Bypass policies when pushing** permissions.
5. Calculate protected-default-branch coverage against the approved baseline, not merely the presence of any policy.

#### Advanced Security alerts

##### API and CLI

Where GitHub Advanced Security for Azure DevOps is licensed and enabled, use the Advanced Security host and page alerts for each repository:

```text
GET https://advsec.dev.azure.com/{organization}/{project}/_apis/alert/repositories/{repositoryId}/alerts?top=100&orderBy=severity&api-version=7.2-preview.1
```

Record alert type, severity, state, first/last seen, rule, validity/confidence where applicable, repository/default branch, assignee/owner, and dismissal justification. Do not store secret values or vulnerable source excerpts in general workshop evidence.

##### Manual steps

1. Open **Project settings > Repositories > Advanced Security** and record enablement and billed committer coverage for each repository.
2. Open **Repos > Advanced Security** and filter secret, dependency, and code-scanning alerts by active state and severity.
3. Export metadata where available or reconcile API counts by repository, alert type, severity, and state.
4. Validate ownership, remediation target, dismissal reason, and age for critical/high alerts.
5. If the feature is unlicensed or disabled, record `Not enabled` rather than zero alerts.

### Pipelines, environments, and release collection

#### Pipeline definitions, templates, runs, and deployments

##### API and CLI

```powershell
az pipelines list --organization $OrganizationUrl --project $Project --top 10000 --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-PIP-001-pipelines.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/pipelines?`$top=100&api-version=7.2" `
	-EvidenceId 'E-PIP-002-yaml-pipelines-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/build/definitions?includeAllProperties=true&includeLatestBuilds=true&`$top=100&api-version=7.2" `
	-EvidenceId 'E-PIP-003-build-definitions-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/build/builds?minTime=<UTC-start>&maxTime=<UTC-end>&queryOrder=finishTimeDescending&`$top=100&api-version=7.2" `
	-EvidenceId 'E-PIP-004-build-runs-page-001'
```

For classic release definitions and deployments, use the release-management host:

```text
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?$expand=environments&api-version=7.2
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/releases?minCreatedTime={UTC}&maxCreatedTime={UTC}&$top=100&api-version=7.2
GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/deployments?minStartedTime={UTC}&maxStartedTime={UTC}&$top=100&api-version=7.2
```

To identify YAML templates and pinning, inspect pipeline configuration returned by `GET .../_apis/pipelines/{pipelineId}?api-version=7.2`, then read only the referenced YAML files through the Git Items API. Search for `template:`, `extends:`, repository resources, `@ref`, task major versions, `vmImage`, and deprecated tasks/images. Preserve commit IDs so results are reproducible.

##### Manual steps

1. Open **Pipelines > Pipelines > All** and assign owner, criticality, production reach, lifecycle state, and expected schedule to every active definition.
2. Open each sampled pipeline's **Edit** view without saving; record YAML repository/path or classic type, default branch, permissions, schedules, retention, pool, and linked resources.
3. Open **Runs**, set the exact date range, and validate success, failure, cancellation, duration, queue time, retry/rerun, and last-run calculations.
4. Open **Pipelines > Releases** and inventory every classic release definition, environment, approval, schedule, and production deployment path.
5. Define deployment, failure, recovery, retry, and rollback consistently before calculating DORA-style measures.

#### Agent pools, queues, and agents

##### API and CLI

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/_apis/distributedtask/pools?poolType=automation&api-version=7.2" `
	-EvidenceId 'E-PIP-010-agent-pools'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/distributedtask/queues?api-version=7.2" `
	-EvidenceId 'E-PIP-011-agent-queues'
```

For each pool ID, collect agents and recent job requests. Agent capabilities can disclose host details, so collect them only when approved:

```text
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/agents?includeCapabilities=false&includeAssignedRequest=true&api-version=7.2
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/jobrequests?completedRequestCount=1000&api-version=7.2
```

##### Manual steps

1. Open **Organization settings > Agent pools** and record pool type, owner, projects authorized, auto-provision setting, pipeline permissions, and security.
2. For self-hosted pools, open **Agents** and record enabled/online status, agent version, OS, last activity, assigned request, and documented patch owner.
3. Ask infrastructure owners for network reach, image build process, credential persistence, scaling, capacity, disaster recovery, and retirement procedures; these are not fully observable through Azure DevOps.
4. Use job requests to calculate utilization and queue trends, while separating Microsoft-hosted and self-hosted capacity.

#### Service connections and variable groups

##### API and CLI

```powershell
az devops service-endpoint list --organization $OrganizationUrl --project $Project --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-PIP-020-service-connections-cli.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/serviceendpoint/endpoints?includeDetails=true&includeFailed=true&api-version=7.2" `
	-EvidenceId 'E-PIP-021-service-connections-api'

az pipelines variable-group list --organization $OrganizationUrl --project $Project --output json |
	Set-Content -Path (Join-Path $EvidenceRoot 'E-PIP-022-variable-groups-cli.json') -Encoding utf8

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/distributedtask/variablegroups?api-version=7.2" `
	-EvidenceId 'E-PIP-023-variable-groups-api'
```

Before storing responses, inspect and remove any returned authorization parameters or variable values not already masked by Azure DevOps. Retain only metadata needed for authentication type, scope, ownership, authorization, Key Vault linkage, and last-use analysis.

##### Manual steps

1. Open **Project settings > Service connections** and record type, owner, description, scope, authentication scheme, workload identity federation/managed identity status, sharing, pipeline authorization, checks, and last-known use.
2. Open each connection's **Security** and **Pipeline permissions**; record broad **Grant access permission to all pipelines** settings and direct user grants.
3. Open **Pipelines > Library > Variable groups** and record owner, linked Key Vault, authorized pipelines, security, and whether secret fields are present. Never reveal or transcribe secret values.
4. Trace each connection and variable group to consuming pipelines using pipeline YAML/classic definitions and owner attestation.
5. Mark unused only after the owner confirms no external, disabled, scheduled, or emergency consumer exists.

#### Environments, checks, and approvals

##### API and CLI

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/distributedtask/environments?`$top=100&api-version=7.2-preview.1" `
	-EvidenceId 'E-PIP-030-environments-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/pipelines/checks/configurations?api-version=7.2-preview.1" `
	-EvidenceId 'E-PIP-031-check-configurations'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/pipelines/approvals?`$top=100&api-version=7.2-preview.1" `
	-EvidenceId 'E-PIP-032-approvals-page-001'
```

For every environment ID, collect environment details and deployment records:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/distributedtask/environments/{environmentId}?expands=resourceReferences&api-version=7.2-preview.1
GET https://dev.azure.com/{organization}/{project}/_apis/distributedtask/environments/{environmentId}/environmentdeploymentrecords?top=100&api-version=7.1
```

##### Manual steps

1. Open **Pipelines > Environments** and classify each environment as development, test, staging, production, or other; assign an owner and criticality.
2. Open each environment's **Approvals and checks** and record approvals, branch control, business hours, exclusive lock, required template, evaluate artifact, Azure Function, REST API, and Azure Monitor checks.
3. Open **Security** and **Pipeline permissions** and record users/groups that can administer, use, or create resources and pipelines authorized to deploy.
4. Trace all production deployment paths, including classic releases and pipelines that deploy directly without an Azure DevOps environment.
5. Calculate production control coverage from deployment paths, not environment count.

### Artifacts, tests, integrations, economics, and resilience collection

#### Feeds, views, packages, and retention

##### API and CLI

Azure Artifacts uses the `feeds.dev.azure.com` host. Organization-scoped feeds omit the project segment; project-scoped feeds include it.

```powershell
Save-AdoResponse `
	-Uri "https://feeds.dev.azure.com/$Organization/_apis/packaging/feeds?includeUrls=true&includeDeletedUpstreams=true&api-version=7.2-preview.1" `
	-EvidenceId 'E-ART-001-feeds'
```

For every feed ID, collect views and paginated package metadata:

```text
GET https://feeds.dev.azure.com/{organization}/_apis/packaging/feeds/{feedId}/views?api-version=7.2-preview.1
GET https://feeds.dev.azure.com/{organization}/_apis/packaging/feeds/{feedId}/packages?includeAllVersions=true&includeDeleted=true&$top=100&api-version=7.2-preview.1
GET https://feeds.dev.azure.com/{organization}/_apis/packaging/feeds/{feedId}/packages/{packageId}/versions?includeUrls=true&api-version=7.2-preview.1
```

Page package inventory with `$skip` until a page returns fewer than `$top` records. Feed responses include upstream-source, capability, and retention-related metadata where available. Package storage consumption and billing may require the UI.

##### Manual steps

1. Open **Artifacts**, select each feed, then **Feed settings**.
2. Record scope, owner, package types, upstream sources, permissions, views and promotion rules, retention, recycle bin, and public/upstream exposure.
3. Record package/version counts by type, latest publish date, deprecated/unlisted status, and known production consumers.
4. Open **Organization settings > Storage** or **Billing/Usage** where available and record artifact storage consumption and included/charged amounts.
5. Define stale and deletable packages with product owners; do not infer deletion eligibility from age alone.

#### Test plans, suites, cases, runs, and results

##### API and CLI

Run per project and page every list:

```powershell
Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/testplan/plans?includePlanDetails=true&filterActivePlans=false&api-version=7.2-preview.1" `
	-EvidenceId 'E-TST-001-test-plans-page-001'

Save-AdoResponse `
	-Uri "$OrganizationUrl/$Project/_apis/test/runs?minLastUpdatedDate=<UTC-start>&maxLastUpdatedDate=<UTC-end>&`$top=100&api-version=7.2" `
	-EvidenceId 'E-TST-002-test-runs-page-001'
```

Follow `x-ms-continuationtoken` for Test Plans. For each plan, suite, and run, collect hierarchy, test cases, results, and attachments metadata as approved:

```text
GET https://dev.azure.com/{organization}/{project}/_apis/testplan/Plans/{planId}/suites?asTreeView=true&api-version=7.2-preview.1
GET https://dev.azure.com/{organization}/{project}/_apis/testplan/Plans/{planId}/Suites/{suiteId}/TestCase?isRecursive=true&api-version=7.2-preview.3
GET https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results?detailsToInclude=workItems&$top=1000&api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/test/results/resultdetailsbybuild?buildId={buildId}&api-version=7.2-preview.1
```

Test attachments can contain logs, credentials, personal data, or application data. Collect metadata by default and download contents only with explicit approval.

##### Manual steps

1. Open **Test Plans > Test Plans** and record plan/suite owner, area/iteration, active/obsolete status, configurations, and requirement-based linkage.
2. Sample test cases to validate steps, expected results, automation status, assigned tester, requirement links, and recent execution.
3. Open **Runs** and compare run/result counts, pass/fail/blocked/not-executed trends, duration, flaky tests, and failure age with API data.
4. Open sampled pipeline runs and verify published test results and retention.
5. Confirm how obsolete tests, flaky tests, shared steps, attachments, and evidence retention are managed.

#### Integration inventory

##### API and CLI

Start with the extension and service-hook evidence already collected. Add service endpoint metadata, pipeline repository resources, dashboards/widgets, and registered consumers when applicable:

```text
GET https://dev.azure.com/{organization}/_apis/hooks/consumers?api-version=7.2
GET https://dev.azure.com/{organization}/_apis/hooks/publishers?api-version=7.2
GET https://dev.azure.com/{organization}/{project}/_apis/serviceendpoint/endpoints?includeDetails=true&api-version=7.2
```

Join integrations by endpoint URL classification, identity descriptor, service connection, hook subscription, extension publisher, or pipeline resource. Redact credentials and sensitive target parameters.

##### Manual steps

1. Review **Organization settings > Extensions**, every project's **Service hooks**, **Service connections**, dashboards, pipeline resources, and repository webhooks/status integrations.
2. For each integration, record system, direction of data flow, data classification, identity/authentication type, owner, criticality, last-known use, failure handling, and recovery contact.
3. Ask owners about integrations initiated from outside Azure DevOps because inbound API clients are not fully enumerable from Azure DevOps configuration.
4. Reconcile audit caller/user-agent evidence and network/API gateway logs where approved to identify undocumented consumers.

#### Licensing, usage, parallel jobs, and cost

##### API and CLI

Use user-entitlement data to summarize access levels and license sources. Use the feeds, pipelines, builds, agents, and Advanced Security datasets for consumption counts. There is no single Azure DevOps REST response that represents the authoritative invoice or every usage charge; billing must be reconciled manually.

Optional organization-specific API surfaces may expose metering or storage information, but do not rely on undocumented endpoints. Record UI exports or invoices as `E-CST` evidence and preserve their billing period.

##### Manual steps

1. Open **Organization settings > Billing** and record billing subscription, paid user quantities by access level, paid parallel jobs, Artifacts storage, Advanced Security billed committers, and paid extensions.
2. Open **Organization settings > Usage** and export or record usage for the exact billing and assessment periods.
3. Open **Organization settings > Parallel jobs** and record Microsoft-hosted/self-hosted capacity, purchased jobs, current consumption, and queue constraints.
4. Reconcile paid users with entitlement status and agreed inactivity; distinguish group-assigned, Visual Studio subscription, stakeholder, and direct paid access.
5. Reconcile artifact storage with feeds/packages and Advanced Security committers with enabled repositories.
6. Record currency, tax treatment, included allowances, billing period, and whether amounts are list, negotiated, allocated, or invoiced costs.

#### Resilience, support, and service limits

##### API and CLI

Use pipeline runs, agent job requests, hook status, audit events, and integration inventories as operational evidence. Azure DevOps Services continuity, vendor SLA, organization ownership, business recovery requirements, and external dependency recovery cannot be derived from one API.

##### Manual steps

1. Identify critical delivery workflows and map their dependencies: Microsoft Entra, Azure DevOps region/service, agents, source mirrors, package upstreams, service connections, secret stores, target environments, DNS/network, and external integrations.
2. Record Microsoft service-health notification recipients, support plan, escalation contacts, organization owners, vendor SLA, and internal recovery objectives.
3. Review [Azure DevOps service status](https://status.dev.azure.com/) and the organization's incident/problem records for the assessment period.
4. Verify documented procedures for organization-owner loss, agent-pool outage, package-source outage, credential compromise, accidental deletion, unavailable production approvals, and critical extension failure.
5. Confirm which source, work item, pipeline, package, test, configuration, and audit data must be exported or recoverable and test at least one approved recovery procedure.
6. Record service limits or throttling observed during collection, including API rate-limit headers and retry behavior.

### Derived metrics and recurring measure collection

Use preserved raw evidence and version-controlled transformation logic. Every metric must record numerator, denominator, exclusions, observation period, and data-quality caveats.

| Measure                              | Source evidence and calculation                                                                                  |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| Inactive-user rate                   | `E-ID` entitlements joined to approved activity signals; inactive in-scope users / in-scope users                |
| Guest-user rate                      | Guest/external identities / enabled identities; validate identity origin in Graph data                           |
| Direct-permission rate               | Direct user ACL entries / in-scope ACL entries after decoding namespace actions                                  |
| Paid-license utilization             | Paid active users meeting agreed activity threshold / paid enabled users; reconcile billing source               |
| Inactive-project rate                | Projects with no agreed meaningful activity after threshold / projects in scope                                  |
| Protected default branches (`M-001`) | Active repositories whose default branch meets every required baseline policy / active repositories              |
| Privileged access reviews (`M-002`)  | Privileged groups with dated owner attestation / privileged groups in scope                                      |
| Federated connections (`M-003`)      | Eligible active Azure service connections using workload identity federation / eligible active Azure connections |
| Production controls (`M-004`)        | Production deployment paths with approved checks / all identified production deployment paths                    |
| Stale asset reduction (`M-005`)      | Baseline stale assets minus current stale assets, reported by asset type and approved threshold                  |
| Change lead time (`M-006`)           | Median and percentile elapsed time between agreed request/commit start and successful production deployment      |
| Deployment failure rate (`M-007`)    | Production deployments requiring rollback, hotfix, or incident response / production deployments                 |
| High-risk closure (`M-008`)          | Critical/high findings independently verified closed / accepted critical/high findings due by measurement date   |

### Collection closeout

1. Confirm every paginated endpoint reached its terminal page and record page/record counts.
2. Reconcile API and CLI counts with the UI; investigate material differences instead of averaging them.
3. Hash raw evidence files with `Get-FileHash -Algorithm SHA256` and store the manifest with the evidence catalog.
4. Remove access tokens, authorization headers, secret fields, sensitive source excerpts, and unnecessary personal data.
5. Assign completeness and confidence to every evidence item and create a limitation for every failed or partial dataset.
6. Have the workspace owner validate ownership/intent fields and a second assessor spot-check API-to-UI accuracy.
7. Copy only redacted summaries into workshop materials; retain and dispose of raw evidence according to the approved schedule.

<!-- markdownlint-enable MD010 MD024 -->

### Domain assessment worksheet

| Domain                                | Current maturity (0-5) | Target maturity (0-5) | Strengths | Primary concerns | Evidence IDs | Confidence |
| ------------------------------------- | ---------------------: | --------------------: | --------- | ---------------- | ------------ | ---------- |
| Governance and organization design    |                        |                       |           |                  |              |            |
| Identity, access, and licensing       |                        |                       |           |                  |              |            |
| Boards and delivery management        |                        |                       |           |                  |              |            |
| Repositories and source control       |                        |                       |           |                  |              |            |
| Pipelines, environments, and releases |                        |                       |           |                  |              |            |
| Artifacts and dependencies            |                        |                       |           |                  |              |            |
| Test strategy and quality             |                        |                       |           |                  |              |            |
| Security, audit, and compliance       |                        |                       |           |                  |              |            |
| Operations, resilience, and economics |                        |                       |           |                  |              |            |

## Key findings

### What is going well

Record effective controls and practices worth preserving or scaling. A strength requires evidence just as a concern does.

| Finding ID | Domain | Strength | Outcome enabled | Evidence IDs | Recommendation to preserve or scale |
| ---------- | ------ | -------- | --------------- | ------------ | ----------------------------------- |
| S-001      |        |          |                 |              |                                     |

### What needs attention

Describe the condition and root cause, not only the symptom. Do not prescribe a solution until the desired and ideal states are understood.

| Finding ID | Domain | Observed condition | Expected condition | Consequence | Root cause/hypothesis | Likelihood | Impact | Risk score | Evidence IDs | Owner response |
| ---------- | ------ | ------------------ | ------------------ | ----------- | --------------------- | ---------: | -----: | ---------: | ------------ | -------------- |
| F-001      |        |                    |                    |             |                       |            |        |            |              |                |

A finding is ready for review when it states scope precisely, links to recent evidence, labels uncertainty, explains consequences, distinguishes cause from symptom, uses consistent ratings, avoids assuming a remediation, and records owner agreement or disagreement.

## Finding interpretation and response

Use this playbook after evidence validation and before creating a change. Tool severity is an input, not the final risk decision.

1. **Validate:** Confirm the asset, owner, configuration, identity, timestamp, query scope, and detection logic. Reproduce the observation through a second source or owner review where practical.
2. **Contextualize:** Combine likelihood and impact with exploitability, privilege, data sensitivity, production reach, asset criticality, existing controls, exposure duration, and legal or contractual obligations. Do not lower a rating solely because no incident is known.
3. **Select a disposition:** Remediate, contain while remediation is prepared, record an approved exception, report a demonstrated inaccuracy to the source product, or close as no longer applicable. Disagreement is not evidence of a false positive.
4. **Plan and own:** Create a traceable gap and change with one accountable owner, due date, dependencies, rollback needs, acceptance evidence, target measure, and residual-risk approver. Critical and high findings need an immediate containment decision even when permanent remediation takes longer.
5. **Verify and sustain:** A person independent of implementation checks the configured state and refreshes the originating evidence. Observe the control for the agreed period, confirm that the alert or exposure is resolved, capture residual risk, and schedule recurrence monitoring. A work item marked complete is not closure evidence.

### Response expectations

Set response and remediation targets from the organization's obligations and risk appetite. The following are workshop defaults for triage, not a substitute for an approved policy.

| Rating   | Initial action                                                                  | Closure expectation                                                                                 |
| -------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Critical | Notify the security and service owners immediately; contain active exposure     | Track to verified remediation or explicit executive risk acceptance with frequent status review     |
| High     | Assign an owner and containment decision in the first roadmap wave              | Verify by the approved high-risk target; escalate missed dates and document residual risk           |
| Medium   | Validate scope and schedule by dependency, exposure, and available capacity     | Verify in the 60-90 day plan or approve a time-bound exception                                      |
| Low      | Add to the governed backlog, bundle with standards work, or accept deliberately | Review before backlog expiry; do not allow repeated low findings to hide a systemic control failure |

### Entra applications, OAuth consent, and credentials

Use Microsoft Graph or the Entra admin center to connect Azure DevOps extensions, service hooks, service connections, reporting clients, and automation identities to their tenant objects. Inventory application and service-principal owners, publisher verification, sign-in activity, delegated and application permissions, tenant-wide grants, assignments, credentials, and affected Azure DevOps assets.

1. Confirm a named business and technical owner, current purpose, approved data flow, publisher, user population, and last use. Treat unknown ownership or purpose as a finding until resolved.
2. Compare every delegated and application permission with the documented feature. Challenge tenant-wide consent, write permissions, directory-wide access, offline access, unverified publishers, broad user assignment, and grants that exceed the application's current behavior.
3. Prefer verified publishers and the narrowest permissions, assignment, and resource scope. Route elevated requests through the Entra admin consent workflow with security review and recorded justification; do not solve blocked consent with a personal account or unsanctioned application.
4. Remove unused grants and credentials, narrow excessive permissions, restrict assignment, or disable the service principal while impact is investigated. Revoke consent and refresh tokens during confirmed compromise, then follow the incident process.
5. Replace client secrets with managed identity or workload identity federation where supported. For retained credentials, apply the lifecycle and handling rules in [Credential governance](#credential-governance).
6. Verify by reacquiring a token through the intended identity, exercising only the approved operation, confirming denied out-of-scope operations, reviewing sign-in and audit logs, and refreshing the application inventory.

Review privileged and newly consented applications monthly, all in-scope applications at least quarterly, and any application after an owner, publisher, credential, permission, assignment, or usage change. Record the cadence owner, evidence location, escalation path, and next review date.

### Software and dependency vulnerabilities

Use GitHub Advanced Security for Azure DevOps for repository code, dependency, and secret alerts where enabled. Use Defender Vulnerability Management only for in-scope build agents, deployment hosts, or other devices that support an Azure DevOps delivery path.

1. Validate the repository, branch, dependency path, secret validity, device, software version, detection logic, and affected production path. Check whether coverage excludes repositories, branches, devices, architectures, or stale assets.
2. Prioritize active or verified exploitation, valid exposed secrets, internet or production exposure, high-value assets, reachable vulnerable code, privilege, blast radius, and the absence of compensating controls ahead of severity alone.
3. Revoke exposed credentials before removing them from source or history. Upgrade, patch, reconfigure, remove, isolate, or block vulnerable components; use a supported workaround only as temporary containment with an owner and expiry.
4. When no fix exists, reduce exposure, monitor for exploitation and vendor updates, and create a time-bound exception. Do not dismiss an alert merely because a fix is unavailable.
5. Report an inaccuracy only after documenting detection logic, version or architecture evidence, and reproduction results. Keep the local finding open until the source correction or an approved local disposition is verified.
6. Re-run the relevant scan or allow Defender inventory to refresh, confirm the alert state and affected population, independently test the control, and monitor recurrence. Closing an alert without refreshed evidence does not close the workshop finding.

### Exceptions and accepted risk

An exception is appropriate only when the finding is valid and immediate remediation is infeasible or disproportionate. A false positive means the observed condition is factually inaccurate; it is not an exception.

Every exception must identify the finding, exact asset and population, business rationale, threat and consequence, compensating control, residual likelihood and impact, accountable risk owner, approver, creation date, expiry date, review cadence, and exit condition. Validate that the compensating control operates before approval. Expired exceptions return to active review automatically; repeated renewal requires escalation and a new risk decision.

### Intelligence-source prerequisites and limitations

| Source                                      | Minimum access and prerequisites                                                                                           | Interpretation limits                                                                                                                |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Azure DevOps REST, CLI, and audit           | Approved read-only Entra identity; endpoint-specific Azure DevOps permissions; audit permission for audit data             | Results are permission-trimmed; APIs paginate differently; retention, preview APIs, licensing, and UI-only fields can limit coverage |
| Microsoft Graph applications and principals | `Application.Read.All` for tenant-wide read inventory plus a supported Entra role in delegated scenarios                   | Directory data does not prove application use or business approval; relationship objects can return limited properties               |
| Microsoft Entra recommendations             | `DirectoryRecommendations.Read.All` or a documented read-only Entra role; verify recommendation licensing and availability | `/beta` APIs can change and are not supported for production applications; recommendations require owner validation                  |
| Azure Resource Graph                        | At least Azure RBAC read access to every queried resource scope                                                            | Missing access silently produces partial results for inaccessible subscriptions or resources; query and result limits still apply    |
| Defender Vulnerability Management           | Supported/onboarded devices, required license, and Vulnerability management read; manage permissions for actions           | Device activity, platform support, device-group scope, inventory refresh, preview experience, and detection logic affect results     |
| Advanced Security for Azure DevOps          | Feature enabled on in-scope Azure Repos and permission to view alerts; separate permission to dismiss                      | Coverage and alert counts can depend on default branch, successful scanning, repository enablement, tool, validity, and license      |

If minimum access is unavailable, record the source as unavailable or sampled, describe likely bias, and assign a follow-up owner. Never convert a permission-trimmed empty result into a statement that no risk exists.

## Desired state

The desired state expresses the owners' intent without assuming that every aspiration is feasible or a best practice. Capture outcomes, constraints, tradeoffs, and measures before choosing controls or products.

### Desired-state statement

> By **[date]**, the Azure DevOps organization will enable **[teams/personas]** to achieve **[business and engineering outcomes]** while meeting **[security/compliance/reliability constraints]**. Success will be demonstrated by **[measures and targets]**. The organization intentionally accepts **[tradeoffs or exceptions]**.

### Outcome catalog

| Outcome ID | Desired outcome | Persona/beneficiary | Baseline | Target and date | Priority | Non-negotiable? | Sponsor |
| ---------- | --------------- | ------------------- | -------- | --------------- | -------- | --------------: | ------- |
| O-001      |                 |                     |          |                 |          |                 |         |

### Target-state decisions

| Decision ID | Topic | Options considered | Decision | Rationale | Constraint/tradeoff | Decider | Date/review date |
| ----------- | ----- | ------------------ | -------- | --------- | ------------------- | ------- | ---------------- |
| T-001       |       |                    |          |           |                     |         |                  |

### Design constraints and intentional exceptions

| Exception ID | Standard or outcome affected | Scope | Business rationale | Compensating control | Risk owner | Expiration/review date |
| ------------ | ---------------------------- | ----- | ------------------ | -------------------- | ---------- | ---------------------- |
| EX-001       |                              |       |                    |                      |            |                        |

## Best-practice reference implementation

The reference implementation is a decision baseline, not a universal prescription. It applies Microsoft's current Azure DevOps guidance to an enterprise operating model in which global platform owners establish guardrails, business units and product domains own delivery within those guardrails, and teams retain autonomy inside explicit security and lifecycle boundaries. Validate every recommendation against current Microsoft documentation, licensing, tenant capabilities, data-residency and regulatory requirements, service limits, and the organization's operating model at implementation time.

### Enterprise topology and geography

- Start with the simplest viable structure. Microsoft recommends starting with one organization and expanding only for specific security, compliance, identity-tenant, billing, administrative, geographic, or scale requirements. Record the decision and reassess it as the enterprise changes.
- Prefer one project with multiple teams when products share processes, assets, and reporting needs. Use multiple projects when business units or domains require stronger access boundaries, independent processes, administrators, or extension testing. Use multiple organizations only for hard boundaries or extreme scale because cross-organization sharing, reporting, and administration are materially more complex.
- Map teams to durable product or platform ownership rather than temporary initiatives. Use area paths, iteration paths, portfolio backlogs, Delivery Plans, and Analytics to coordinate across teams while retaining team-level backlogs and boards.
- Select each organization's Azure DevOps geography deliberately for data residency, sovereignty, latency, and regulatory needs. Document Microsoft-published exceptions for profile data, token data, disaster recovery, previews, support, and Microsoft-hosted macOS agents before approval.
- For a multi-organization enterprise, define a federation contract covering naming, identity groups, baseline policies, reusable templates, extension approval, audit export, licensing, billing, reporting, exception handling, and lifecycle management. Treat organization boundaries as hard isolation boundaries, not as a substitute for a designed enterprise control plane.
- Keep an authoritative organization/project/domain catalog with business owner, technical owner, geography, tenant, classification, criticality, cost center, approved services, and lifecycle status.

See [Plan your organizational structure](https://learn.microsoft.com/azure/devops/user-guide/plan-your-azure-devops-org-structure), [About projects and scaling your organization](https://learn.microsoft.com/azure/devops/organizations/projects/about-projects), [Data locations for Azure DevOps](https://learn.microsoft.com/azure/devops/organizations/security/data-location), and [Scale Agile practices across teams](https://learn.microsoft.com/azure/devops/boards/plans/).

### Governance and identity

- Connect each organization to the intended Microsoft Entra tenant, restrict unauthorized organization creation, and maintain at least two accountable organization owners with tested emergency-access and ownership-transfer procedures.
- Model enterprise personas and duties with Microsoft Entra and Azure DevOps groups. Assign permissions to groups at the broadest matching scope, use inheritance by default, avoid direct user grants, and reserve explicit `Deny` and object-level exceptions for justified cases.
- Separate routine administration from privileged administration. Use Microsoft Entra Privileged Identity Management where licensed for just-in-time administrative group membership, require multifactor authentication and appropriate device/location controls through Conditional Access, and periodically attest privileged access.
- Keep groups aligned to stable business unit, domain, product, environment, and job-function boundaries. Microsoft identifies groups over 10,000 effective members and security namespaces approaching 1,000,000 access-control entries as permission-performance risks; monitor growth and split access cohorts where requirements differ.
- Restrict guest access, project creation, extension installation, service-hook creation, authentication mechanisms, and policy bypass to approved roles and use cases. Apply time-bound sponsorship, recertification, and removal to external identities.
- Establish automated joiner/mover/leaver processes for employees, guests, contractors, service principals, managed identities, and remaining service accounts. Reconcile Microsoft Entra status, Azure DevOps membership, group rules, access level, paid license, credentials, and owned assets.
- Prefer Microsoft Entra tokens, managed identities, service principals, and workload identity federation over personal access tokens and stored secrets. Where PATs remain necessary, enforce organization policies, minimal scope and duration, inventory, rotation, leak response, and revocation.
- Stream Azure DevOps audit events to an approved monitoring destination, define alert and investigation ownership, and review privileged, identity, permission, policy, credential, extension, and service-connection events.

See [Make your Azure DevOps secure](https://learn.microsoft.com/azure/devops/organizations/security/security-overview), [About permissions and groups](https://learn.microsoft.com/azure/devops/organizations/security/about-permissions), [Optimize permissions for performance](https://learn.microsoft.com/azure/devops/organizations/security/permission-performance-recommendations), [Set up Conditional Access policies on Azure DevOps](https://learn.microsoft.com/azure/devops/organizations/accounts/conditional-access-policies), and [Azure DevOps auditing](https://learn.microsoft.com/azure/devops/organizations/audit/azure-devops-auditing).

### Projects, Boards, and reporting

- Use a documented project, team, repository, and process strategy. Standardize a small set of inherited processes and common taxonomy; approve customization only when it supports a measured domain need that team-level configuration cannot meet.
- Define ownership, naming, lifecycle, classification, creation, review, archive, and deletion criteria for projects, teams, area/iteration paths, queries, dashboards, and reporting integrations.
- Organize portfolio hierarchies around durable outcomes: management teams own epics, product or feature teams own features and backlog items, and dependencies are visible through Delivery Plans or approved portfolio reporting.
- Align state categories, start/done definitions, iteration semantics, and metric calculations before comparing flow across teams, business units, or geographies. Preserve local practices where they do not invalidate enterprise measures.
- Use Analytics and Power BI for governed cross-project reporting and constrain queries to the required project scope. For multiple organizations, implement an external reporting layer with common identifiers and definitions because Azure DevOps doesn't provide native cross-organization portfolio rollup.
- Protect analytics according to source-project, area-path, repository, and pipeline permissions; publish only the minimum data needed by each audience and apply retention and privacy controls to exported data.
- Keep traceability proportionate to organizational risk: objective or request to work item, code change, review, build, test, artifact, approval, deployment, and operational outcome.

See [Scale Agile practices across teams](https://learn.microsoft.com/azure/devops/boards/plans/), [Manage priorities and gain visibility across teams](https://learn.microsoft.com/azure/devops/boards/plans/visibility-across-teams), [What is Analytics?](https://learn.microsoft.com/azure/devops/report/powerbi/what-is-analytics), and [Secure your analytics and reporting](https://learn.microsoft.com/azure/devops/report/secure-your-analytics-reporting).

### Repositories

- Give every active repository a named owner, classification, default branch, and lifecycle status.
- Choose monorepository or multirepository structures from product architecture, deployment independence, code ownership, security, and scale rather than organizational fashion; document repository-to-product and repository-to-service mappings.
- Protect default and release branches with proportionate reviewer, comment-resolution, linked-work, merge-strategy, status-check, and build-validation policies. Protect pipeline and template repositories at least as strongly as production source.
- Limit policy bypass, force-push, branch deletion, repository creation, and destructive administration; monitor their use and require a documented emergency path.
- Standardize repository guidance, ownership metadata, contribution conventions, dependency management, secret prevention, and code scanning. Use GitHub Advanced Security for Azure DevOps where licensed and appropriate to the risk.
- Archive or disable stale repositories through an approved, reversible lifecycle process.

See [Azure Repos Git branch policies and settings](https://learn.microsoft.com/azure/devops/repos/git/branch-policies), [Git repository limits](https://learn.microsoft.com/azure/devops/repos/git/limits), and [GitHub Advanced Security for Azure DevOps](https://learn.microsoft.com/azure/devops/repos/security/github-advanced-security-security-overview).

### Pipelines and releases

- Prefer version-controlled YAML and disable creation of Classic build and release pipelines after approved migration. Establish centrally owned, versioned `extends` templates for required build, test, security, artifact, provenance, and deployment controls, with documented extension points for domains.
- Protect template repositories and pin template, task, tool, package, container-image, and infrastructure-module references to approved versions. Test upgrades centrally, publish migration windows, and remove deprecated versions on a governed schedule.
- Set job authorization scope to the project and use project-scoped build identities. Explicitly authorize protected repositories, variable groups, secure files, agent pools, environments, and service connections to only the pipelines that require them.
- Separate build from authorization to deploy. Model deployment targets as environments and apply resource-owner-controlled approvals and checks, including branch control, required templates, business hours, exclusive locks, and external policy checks where justified.
- Scope service connections to the smallest practical Azure resource boundary and pipeline set. Prefer workload identity federation or managed identity; prohibit broad subscription-level contributor access unless approved with compensating controls.
- Keep secrets out of YAML, command-line arguments, logs, artifacts, and assessment exports. Use secretless authentication where possible and otherwise use approved secret stores, narrow permissions, rotation, and leak-response procedures.
- Prefer Microsoft-hosted agents for isolation. Where network or tooling requirements require self-hosted agents, separate pools by project, trust zone, environment, and workload sensitivity; use low-privilege identities, ephemeral agents where feasible, restrictive egress, hardened images, prompt patching, capacity monitoring, and no untrusted fork code.
- Treat production deployment pipelines as critical services: define owner and support rotation, concurrency and regional capacity, rollback or roll-forward procedures, recovery tests, deployment traceability, and failure/recovery measures.

See [Secure Azure Pipelines](https://learn.microsoft.com/azure/devops/pipelines/security/overview), [Templates for security](https://learn.microsoft.com/azure/devops/pipelines/security/templates), [Approvals and checks](https://learn.microsoft.com/azure/devops/pipelines/process/approvals), [Secure agents, projects, and containers](https://learn.microsoft.com/azure/devops/pipelines/security/misc), and [Workload identity federation for Azure Resource Manager service connections](https://learn.microsoft.com/azure/devops/pipelines/release/configure-workload-identity).

### Artifacts, quality, audit, and operations

- Give feeds, packages, views, test plans, suites, cases, and quality policies named owners, classification, lifecycle rules, least-privilege permissions, and retention appropriate to product and regulatory risk.
- Control upstream package sources, isolate trusted release views, preserve package and build provenance, scan dependencies, and prevent released packages from being overwritten. Monitor storage, package-version, and upstream-source limits.
- Apply branch, pipeline, test, security, and deployment quality gates according to service criticality. Use pass rate, flaky-test, escaped-defect, deployment-failure, and recovery trends to improve quality rather than maximizing test counts.
- Maintain an approved catalog of extensions, hooks, webhooks, applications, reporting clients, and non-human identities. Require owner, purpose, scopes, data flow, support model, failure behavior, review date, and retirement criteria.
- Design REST and integration workloads for Azure DevOps service protection: prefer service hooks over polling, batch operations, paginate completely, cache appropriately, use reporting APIs for bulk reporting, honor retry headers with exponential backoff and jitter, and avoid high-frequency permission reconciliation.
- Monitor rate and usage limits, object limits, repository size, permission scale, parallel-job capacity, agent queues, artifact storage, license consumption, extension charges, audit-stream health, integration failures, and Microsoft service health. Assign thresholds, response playbooks, and escalation owners.
- Establish global and delegated operating forums. Global platform owners manage tenant/org guardrails, common templates, federation, vendor roadmap, and enterprise measures; business-unit/domain owners manage adoption, local exceptions, product lifecycle, and support within those guardrails.
- Maintain continuity procedures for critical delivery paths, including owner and administrator recovery, configuration and metadata exports where supported, dependency maps, alternate communication and deployment procedures, and periodic recovery exercises. Do not describe Azure DevOps exports as a full customer-managed backup unless Microsoft explicitly supports that recovery scenario.
- Review the architecture against current Microsoft documentation and release notes at least quarterly and before major geography, organization, process, identity, agent, or pipeline-platform changes.

See [Best practices for integration](https://learn.microsoft.com/azure/devops/integrate/concepts/integration-bestpractices), [Rate and usage limits](https://learn.microsoft.com/azure/devops/integrate/concepts/rate-limits), [Work tracking, process, and project limits](https://learn.microsoft.com/azure/devops/organizations/settings/work/object-limits), [Azure Artifacts limits](https://learn.microsoft.com/azure/devops/artifacts/reference/limits), and [Azure DevOps service status](https://status.dev.azure.com/).

### Microsoft reference set

Confirm links, documented limits, feature availability, licensing, and recommendations during each engagement because Azure DevOps capabilities evolve.

| Concern                                   | Official Microsoft guidance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Enterprise structure and scale            | [Plan your organizational structure](https://learn.microsoft.com/azure/devops/user-guide/plan-your-azure-devops-org-structure), [About projects and scaling your organization](https://learn.microsoft.com/azure/devops/organizations/projects/about-projects), [Optimize permissions for performance](https://learn.microsoft.com/azure/devops/organizations/security/permission-performance-recommendations)                                                                                                                                                                                                                                 |
| Geography and data protection             | [Data locations for Azure DevOps](https://learn.microsoft.com/azure/devops/organizations/security/data-location), [Data protection overview](https://learn.microsoft.com/azure/devops/organizations/security/data-protection)                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Security and identity                     | [Make your Azure DevOps secure](https://learn.microsoft.com/azure/devops/organizations/security/security-overview), [About permissions and groups](https://learn.microsoft.com/azure/devops/organizations/security/about-permissions), [Conditional Access for Azure DevOps](https://learn.microsoft.com/azure/devops/organizations/accounts/conditional-access-policies), [Manage PATs using policies](https://learn.microsoft.com/azure/devops/organizations/accounts/manage-pats-with-policies-for-your-organization)                                                                                                                       |
| Authentication and application governance | [Azure DevOps authentication guidance](https://learn.microsoft.com/azure/devops/integrate/get-started/authentication/authentication-guidance), [Review enterprise application permissions](https://learn.microsoft.com/entra/identity/enterprise-apps/manage-application-permissions), [Application consent management](https://learn.microsoft.com/entra/identity/enterprise-apps/manage-consent-requests), [Govern Entra service accounts](https://learn.microsoft.com/entra/architecture/govern-service-accounts)                                                                                                                           |
| Boards and enterprise reporting           | [Scale Agile practices across teams](https://learn.microsoft.com/azure/devops/boards/plans/), [Visibility across teams](https://learn.microsoft.com/azure/devops/boards/plans/visibility-across-teams), [Analytics](https://learn.microsoft.com/azure/devops/report/powerbi/what-is-analytics), [Analytics security](https://learn.microsoft.com/azure/devops/report/secure-your-analytics-reporting)                                                                                                                                                                                                                                          |
| Repositories and code security            | [Branch policies and settings](https://learn.microsoft.com/azure/devops/repos/git/branch-policies), [Git limits](https://learn.microsoft.com/azure/devops/repos/git/limits), [GitHub Advanced Security for Azure DevOps](https://learn.microsoft.com/azure/devops/repos/security/github-advanced-security-security-overview)                                                                                                                                                                                                                                                                                                                   |
| Vulnerability remediation                 | [Advanced Security dependency scanning](https://learn.microsoft.com/azure/devops/repos/security/github-advanced-security-dependency-scanning), [Advanced Security secret scanning](https://learn.microsoft.com/azure/devops/repos/security/github-advanced-security-secret-scanning), [Defender Vulnerability Management](https://learn.microsoft.com/defender-vulnerability-management/defender-vulnerability-management), [Defender remediation](https://learn.microsoft.com/defender-vulnerability-management/tvm-remediation), [Defender exceptions](https://learn.microsoft.com/defender-vulnerability-management/tvm-exception-overview) |
| Secure CI/CD and deployment               | [Secure Azure Pipelines](https://learn.microsoft.com/azure/devops/pipelines/security/overview), [Templates for security](https://learn.microsoft.com/azure/devops/pipelines/security/templates), [Approvals and checks](https://learn.microsoft.com/azure/devops/pipelines/process/approvals), [Secure agents, projects, and containers](https://learn.microsoft.com/azure/devops/pipelines/security/misc), [Workload identity federation](https://learn.microsoft.com/azure/devops/pipelines/release/configure-workload-identity)                                                                                                             |
| Artifacts, audit, and integrations        | [Azure Artifacts permissions](https://learn.microsoft.com/azure/devops/artifacts/feeds/feed-permissions), [Azure Artifacts limits](https://learn.microsoft.com/azure/devops/artifacts/reference/limits), [Azure DevOps auditing](https://learn.microsoft.com/azure/devops/organizations/audit/azure-devops-auditing), [Integration best practices](https://learn.microsoft.com/azure/devops/integrate/concepts/integration-bestpractices)                                                                                                                                                                                                      |
| Limits, automation, and operations        | [Rate and usage limits](https://learn.microsoft.com/azure/devops/integrate/concepts/rate-limits), [Object limits](https://learn.microsoft.com/azure/devops/organizations/settings/work/object-limits), [REST API reference](https://learn.microsoft.com/rest/api/azure/devops/), [Azure DevOps CLI reference](https://learn.microsoft.com/cli/azure/service-page/azure%20devops), [Service status](https://status.dev.azure.com/)                                                                                                                                                                                                              |

Additional retained reference: [Azure DevOps security best practices](https://learn.microsoft.com/azure/devops/organizations/security/security-best-practices).

## Ideal state and gap analysis

The ideal state is the best feasible state for this organization. It is derived from the validated current state, owner-approved desired outcomes, the reference implementation, and explicit constraints. It is not automatically the maximum maturity score in every domain.

For each domain, identify the outcomes it supports, select relevant reference practices, tailor controls to criticality, set a realistic target maturity, record exceptions and residual risk, and obtain the appropriate approval.

| Domain                                | Current state summary | Desired outcomes | Ideal capabilities/controls | Target maturity | Target measure | Constraints/exceptions | Approver |
| ------------------------------------- | --------------------- | ---------------- | --------------------------- | --------------: | -------------- | ---------------------- | -------- |
| Governance and organization design    |                       |                  |                             |                 |                |                        |          |
| Identity, access, and licensing       |                       |                  |                             |                 |                |                        |          |
| Boards and delivery management        |                       |                  |                             |                 |                |                        |          |
| Repositories and source control       |                       |                  |                             |                 |                |                        |          |
| Pipelines, environments, and releases |                       |                  |                             |                 |                |                        |          |
| Artifacts and dependencies            |                       |                  |                             |                 |                |                        |          |
| Test strategy and quality             |                       |                  |                             |                 |                |                        |          |
| Security, audit, and compliance       |                       |                  |                             |                 |                |                        |          |
| Operations, resilience, and economics |                       |                  |                             |                 |                |                        |          |

### Gap register

| Gap ID | Domain | Current capability | Ideal capability | Finding IDs | Outcome IDs | Gap size (S/M/L) | Candidate change IDs |
| ------ | ------ | ------------------ | ---------------- | ----------- | ----------- | ---------------- | -------------------- |
| G-001  |        |                    |                  |             |             |                  |                      |

## Change catalog and prioritization

Create a change only after it can be traced through a gap to both a finding and a desired outcome. Split large changes until each has one accountable owner, a verifiable outcome, and an implementable scope.

### Proposed change catalog

| Change ID | Proposed change | Finding/gap/outcome IDs | Scope | Accountable owner | Dependencies | Risk if deferred | Effort | Acceptance criteria | Success measure | Status   |
| --------- | --------------- | ----------------------- | ----- | ----------------- | ------------ | ---------------- | ------ | ------------------- | --------------- | -------- |
| CHG-001   |                 |                         |       |                   |              |                  |        |                     |                 | Proposed |

Allowed statuses: `Proposed`, `Accepted`, `In progress`, `Blocked`, `Complete`, and `Rejected`. Record deferred work as `Proposed` or `Blocked` with rationale; record risk acceptance as a decision or exception rather than a change status.

### Prioritization model

Score each accepted change from 1 (low) to 5 (very high):

- **Risk reduction:** Reduction in security, compliance, operational, or delivery exposure.
- **Outcome value:** Contribution to approved desired outcomes.
- **Urgency:** Time sensitivity such as audit deadlines, expiring technology, or active incidents.
- **Enablement:** Degree to which the change unlocks or simplifies other changes.
- **Effort:** Relative delivery effort, coordination, and complexity; 5 is greatest.

$$\text{Priority score} = 2(\text{Risk reduction}) + \text{Outcome value} + \text{Urgency} + \text{Enablement} - \text{Effort}$$

| Change ID | Risk reduction | Outcome value | Urgency | Enablement | Effort | Priority score | Mandatory/deadline | Proposed wave |
| --------- | -------------: | ------------: | ------: | ---------: | -----: | -------------: | ------------------ | ------------- |
| CHG-001   |                |               |         |            |        |                |                    |               |

The score supports discussion; it does not replace judgment. Mandatory compliance work, active critical exposure, hard dependencies, change freezes, and available capacity may override rank. Record every override in the decision log.

## 30/60/90-day action plan

Plan by outcomes and dependencies rather than filling each period with an equal number of tasks.

| Wave           | Primary intent                | Typical work                                                                                                                            | Exit evidence                                                                              |
| -------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| 0-30 days      | Contain and establish control | Critical/high-risk containment, ownership, evidence gaps, privileged access review, credential response, governance cadence, quick wins | Immediate risks have owners and containment; baseline and decision forums are approved     |
| 31-60 days     | Standardize foundations       | Policies, group model, lifecycle standards, asset ownership, reusable templates, service connection remediation pilots                  | Standards are approved, piloted, measured, and supported by migration guidance             |
| 61-90 days     | Scale and enforce             | Roll out proven controls, migrate priority assets, automate drift detection, train teams, retire superseded paths                       | Target coverage is reached for priority scope; exceptions and residual risk are documented |
| Beyond 90 days | Optimize and sustain          | Complete complex migrations, improve developer experience, tune measures, expand automation, retire legacy assets                       | Governance operates routinely and outcome trends demonstrate sustained improvement         |

### Roadmap

| Wave   | Change ID | Outcome/deliverable | Owner | Start | Due | Dependencies | Capacity/effort | Measure and target | Review gate | Status |
| ------ | --------- | ------------------- | ----- | ----- | --- | ------------ | --------------- | ------------------ | ----------- | ------ |
| 0-30   |           |                     |       |       |     |              |                 |                    |             |        |
| 31-60  |           |                     |       |       |     |              |                 |                    |             |        |
| 61-90  |           |                     |       |       |     |              |                 |                    |             |        |
| Beyond |           |                     |       |       |     |              |                 |                    |             |        |

Before moving a change to `In progress`, confirm its scope, owner, affected users, dependencies, licensing, permissions, rollback needs, testable acceptance criteria, baseline and target, communication and support impacts, rollout approach, and residual-risk approval.

## Governance and measures

### Review cadence

| Forum                      | Cadence                        | Participants                                        | Decisions/outputs                                           |
| -------------------------- | ------------------------------ | --------------------------------------------------- | ----------------------------------------------------------- |
| Remediation stand-up       | Weekly through day 90          | Change owners, assessment lead, platform owner      | Progress, blockers, dependency actions                      |
| Risk and exception review  | Biweekly or monthly            | Workspace owner, security/compliance, risk owners   | Risk acceptance, exception expiry, escalation               |
| Platform governance review | Monthly                        | Owners, admins, engineering/product representatives | Standards, adoption, drift, extensions, lifecycle decisions |
| Sponsor checkpoint         | Day 30, 60, 90, then quarterly | Sponsor, workspace owner, assessment lead           | Outcome progress, capacity, roadmap changes                 |

### Measures and benefits tracking

Choose a small balanced set. Establish baselines with the same definitions and collection method used for targets.

| Measure ID | Measure                       | Definition/query                                                                    | Baseline/date | Target/date | Frequency | Owner | Data quality notes |
| ---------- | ----------------------------- | ----------------------------------------------------------------------------------- | ------------- | ----------- | --------- | ----- | ------------------ |
| M-001      | Protected default branches    | Active repositories meeting approved policy / active repositories                   |               |             | Monthly   |       |                    |
| M-002      | Privileged access reviews     | Privileged groups attested / privileged groups in scope                             |               |             | Quarterly |       |                    |
| M-003      | Federated service connections | Eligible active Azure connections using federation / eligible active connections    |               |             | Monthly   |       |                    |
| M-004      | Production control coverage   | Production paths with approved checks / production paths                            |               |             | Monthly   |       |                    |
| M-005      | Stale asset reduction         | Assets meeting approved retirement criteria                                         |               |             | Monthly   |       |                    |
| M-006      | Change lead time              | Agreed start-to-production elapsed time by team/service                             |               |             | Monthly   |       |                    |
| M-007      | Deployment failure rate       | Failed production deployments / production deployments                              |               |             | Monthly   |       |                    |
| M-008      | High-risk finding closure     | Verified closed critical/high findings / accepted critical/high findings            |               |             | Weekly    |       |                    |
| M-009      | Stale credential exposure     | Active credentials past the approved last-use or age threshold / active credentials |               |             | Monthly   |       |                    |
| M-010      | Excess application permission | In-scope applications with an unapproved or excessive grant / applications reviewed |               |             | Monthly   |       |                    |
| M-011      | Consent review coverage       | Applications with current owner and permission attestation / applications in scope  |               |             | Monthly   |       |                    |
| M-012      | Vulnerability age             | Median and 90th-percentile age of open findings by severity and criticality         |               |             | Weekly    |       |                    |
| M-013      | Remediation target compliance | Findings verified by the approved target / findings due in the measurement period   |               |             | Weekly    |       |                    |
| M-014      | Finding recurrence rate       | Findings reopened or rediscovered in the agreed window / verified closed findings   |               |             | Monthly   |       |                    |

Avoid making measures targets until teams agree on definitions and expected behavior. Pair speed measures with quality and reliability measures to discourage gaming.

The browser records `Complete` as a local status; it does not model or verify closure evidence. Before treating a change as complete, independently confirm its acceptance criteria, cataloged evidence, documentation, ownership, affected-user communication, and stable measure for the agreed observation period. Closing a change does not automatically close its finding; the finding owner verifies residual risk.

## Workshop completion criteria

- Scope, participants, assumptions, limitations, and evidence cutoff are documented.
- Qualitative input represents owners and affected practitioner groups.
- Quantitative inventories are complete or limitations and sampling are explicit.
- Owners have reviewed current-state facts and all disputed conclusions are recorded.
- Every finding links to evidence and has a consistent risk rating.
- Desired outcomes, measures, constraints, and tradeoffs are sponsor-approved.
- The ideal state and target maturity are defined for every in-scope domain.
- Every proposed change traces through a gap to both a finding and a desired outcome.
- Accepted changes have owners, dependencies, acceptance criteria, measures, and roadmap waves.
- Critical and high risks are remediated, contained, or formally accepted.
- Day 30, 60, and 90 reviews and the ongoing governance cadence are scheduled.
- Evidence retention and secure disposal responsibilities are assigned.

### Sign-off

| Role                | Name | Decision                                   | Date | Conditions/notes |
| ------------------- | ---- | ------------------------------------------ | ---- | ---------------- |
| Executive sponsor   |      | Approve / Approve with conditions / Reject |      |                  |
| Workspace owner     |      | Approve / Approve with conditions / Reject |      |                  |
| Security/compliance |      | Approve / Approve with conditions / Reject |      |                  |
| Assessment lead     |      | Complete / Incomplete                      |      |                  |

## Appendices

### A. Quantitative collection run log

Use this log to make collection reproducible and prevent silent partial results.

| Run ID | Dataset | Tool/API and version | Scope/filter | Pagination complete? | Started/completed UTC | Record count | Evidence ID | Errors/notes |
| ------ | ------- | -------------------- | ------------ | -------------------- | --------------------- | -----------: | ----------- | ------------ |
| R-001  |         |                      |              |                      |                       |              |             |              |

### B. Data handling checklist

- Use a dedicated read-only identity with the minimum permissions required for collection.
- Obtain approval before collecting user, audit, security-alert, or credential metadata.
- Never collect secret values, token values, private keys, variable-group secrets, or service connection credentials.
- Redact personal and sensitive information from workshop materials and screenshots.
- Encrypt evidence in transit and at rest; restrict it to named participants.
- Record retention and disposal dates and delete temporary local exports after verification.
- Log collection failures and permission denials rather than silently omitting data.

### C. Terms

- **Current state:** Validated facts and stakeholder experience at the evidence cutoff date.
- **Desired state:** Outcomes and constraints explicitly stated by workspace owners and sponsors.
- **Reference implementation:** A curated baseline of Microsoft guidance and generally applicable control patterns.
- **Ideal state:** The best feasible target after reconciling current state, desired state, reference practices, risk, and constraints.
- **Finding:** An evidence-backed strength or difference between observed and expected conditions.
- **Gap:** A difference between current and approved ideal-state capabilities.
- **Change:** An owned unit of work intended to close a gap or achieve an outcome.
- **Exception:** A time-bound, approved departure from a standard with a named risk owner and compensating control where possible.