Airflow Custom Connector for Copilot Studio

Different Actions You can Configure

Look around (read-only)

  • Get all DAG runs: GET /dags/~/dagRuns. The ~ means “all DAGs”. Filter with state, start_date_gte, end_date_lte, and order_by. Results are paged (default 100), so use limit and offset.

  • Get the last run of a DAG: GET /dags/{dag_id}/dagRuns?order_by=-start_date&limit=1

  • Get failed runs: GET /dags/~/dagRuns?state=failed&start_date_gte=<yesterday>

  • Get failed tasks: GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances?state=failed. Don’t include upstream_failed. Those tasks are victims, not the cause, and they will mislead your RCA.

  • Get long-running jobs: POST /dags/~/dagRuns/~/taskInstances/list with state=["running"] and start_date_lte=<now minus threshold>. Duration may be empty while a task is still running, so filtering on start time is more reliable than duration_gte.

  • Compare with normal: GET /dags/{dag_id}/dagRuns?state=success&limit=10. The agent averages the last ten durations and tells you whether today’s run is slow or just normal.

  • See the DAG structure: GET /dags/{dag_id}/tasks returns the tasks and their downstream tasks. This is useful for impact analysis (“what else gets delayed?”).

Investigate

  • Get task logs: GET .../taskInstances/{task_id}/logs/{try_number}. Ask for text/plain. Logs come back in chunks. Use the token parameter to page, and avoid full_content=true unless you really need it.

  • See all retries: GET .../taskInstances/{task_id}/tries shows what happened on each attempt.

  • Who touched it? GET /eventLogs?dag_id=... is the audit trail for pauses, clears, and manual triggers.

  • Check parse errors: GET /importErrors is the first place to look when a DAG disappears from the UI.

  • Check capacity: GET /pools shows open, running, and queued slots. A “stuck” DAG is often just a full pool.

  • Check scheduler health: GET /health shows the metadatabase and scheduler heartbeat.

Take action (the careful ones)

  • Retry a failed task: POST /dags/{dag_id}/clearTaskInstances with dag_run_id, task_ids, and only_failed: true. Clearing resets the task so the scheduler picks it up again.

  • Mark a task success or failed: PATCH /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id} with new_state. For several tasks at once, use POST /dags/{dag_id}/updateTaskInstancesState.

  • Rerun the whole run: POST /dags/{dag_id}/dagRuns/{dag_run_id}/clear

  • Trigger a fresh run: POST /dags/{dag_id}/dagRuns with an optional conf JSON, so the agent can pass parameters like a date or a source name.

  • Set a run’s state: PATCH /dags/{dag_id}/dagRuns/{dag_run_id} with state set to success, failed, or queued.

  • Pause / Unpause: PATCH /dags/{dag_id}?is_paused=true with {"is_paused": true}. Newer 2.x versions can also pause many DAGs by name pattern, which is handy during maintenance windows.

Setting up the connector: the technical bits

Authentication.

  • Airflow 2: enable the basic auth backend (or use an API gateway) and use a dedicated service account.

  • Airflow 3: the API uses short-lived JWT tokens, so plan for token fetch and refresh. A small flow or gateway in front of Airflow usually handles it cleanly.

  • Managed Airflow (MWAA, Composer, Astronomer) has its own auth wrapper. Check that first.

Split by risk. Create two connections, or even two connectors:

  • A read-only one with a Viewer-level role

  • An operations one for clear, mark success, trigger, and pause

If someone shares the agent, they get read access without write power by default.

Network. If Airflow sits on a private network, you’ll need the on-premises data gateway or VNet-enabled connectivity for the connector. Sort this out before debugging anything else.

Keep it fast and cheap: instructions, prompts, and credits

1. Fewer calls, not smarter calls

  • Bundle the chain. Build one “Investigate DAG failure” operation (via a flow, a function, or connector code) that does last run, then failed tasks, then the trimmed log in one go. Three calls become one. Keep the small individual tools for ad-hoc questions.

  • Use summary endpoints. dagStats answers “how are we doing?” in one call instead of listing runs for every DAG.

  • Never let it loop. Tell the agent not to poll or “keep checking”. Offer to re-check when the user asks.

2. Trim what comes back (this is the big one for tokens)

  • Logs are the token hog. A failed task log can run to thousands of lines. Return only the tail, or only the lines around the first ERROR or Traceback plus about 20 lines of context. Connector custom code can do this, though it has time and size limits, so keep it light.

  • Cut the response schema. Keep only the fields the agent needs, such as dag_id, dag_run_id, state, start_date, end_date, task_id, and try_number.

  • Always set limit. limit=5 beats 100 rows of history the agent will never read.

3. Don’t make the model decide what you already know

  • Hardcode inputs the model shouldn’t choose: order_by, limit, the Accept header, and only_failed. Set them as fixed values in the tool, not as AI-filled inputs. Every parameter the model has to reason about costs tokens and adds a chance of error.

  • Write inputs with examples. For example, a dag_run_id described as scheduled__2026-10-17T00:00:00+00:00 saves a lot of retries.

4. Write short, sharp instructions

  • Role, order, rules. Three short blocks. Don’t paste API docs into the instructions.

  • Give it a playbook. For example: “For ‘why did X fail’: 1) get the last run. 2) If failed, get failed tasks only (ignore upstream_failed). 3) Get logs for the last try of the first failed task. 4) Answer with What, Why, Pattern, Next.”

  • Say what NOT to do. “Never list all runs without a dag_id or state filter. Reuse results already in this conversation instead of calling again.”

  • Cap the output. “RCA in under 120 words. No raw logs unless asked.” Output tokens cost money, and nobody reads a wall of text at 6 AM.

  • Gate the risky stuff in one line. “Before clear, mark success, trigger, or pause: show what will change and wait for a yes.”

5. Keep the toolbox tidy

  • Every tool’s name and description is read on every turn. Twenty tools you never use are a tax on every message. Turn off what you don’t need.

  • One job per tool, and say when NOT to use it. “Use for failed tasks only. For running tasks, use Get long-running jobs.” This cuts wrong-tool picks.

  • Consider splitting. A read-only agent for the team and a separate operations agent for the on-call engineer. Each stays smaller, cheaper, and safer.

6. Match the model to the job

  • Lookups and status checks don’t need a reasoning model. Save that for the RCA step if your setup lets you choose models, because the premium adds up quickly.

7. Measure it

  • Run the same five questions before and after each change, and watch credits per session in the analytics and the activity map. If a change doesn’t move the number, it wasn’t an optimization.

BACK

CONNECT WITH ME