Securing AI skill repositories with Nvidia SkillSpector and GitHub Actions

by

A SKILL.md, AGENTS.md, custom instruction file, or MCP configuration can affect how an agent reads a repository, calls tools, writes code, updates infrastructure, or explains a design. These files may look like documentation, but once an agent uses them to make decisions, they sit inside the trust boundary.

With all this, you could potentially be introducing a new supply-chain surface area. The fastest way to reduce that risk is to scan in CI, on every change, with consistent outputs your team can review.

To assist with the above, i’ve been looking at NVIDIA SkillSpector – It gives you a way to scan skill and instruction content for risky behaviour, then produce output that fits into normal engineering workflows.

In this blog post, I will be showing how to run SkillSpector as part of your GitHub Actions workflow. The pattern is straightforward: scan the files that influence agent behaviour, run the scan in CI, and make the results visible to reviewers.

Contents hide
1 What to scan with SkillSpector?
2 GitHub Actions Setup for SkillSpector
2.1 GitHub Action Baseline repository scan
2.2 Focused pull request scan
3 Wrapping Up

What to scan with SkillSpector?

SkillSpector can scan a directory, Git repository or even a single markdown file. In practice, I would usually point it at either the whole repository for scheduled baseline coverage, or the specific paths where skills and instruction files live.

Agents, skills and instruction files can hide:

  • Prompt-injection patterns
  • Exfiltration behavior
  • Unsafe tool-usage chains
  • Policy-bypass or self-modification tactics
  • Risky script snippets

SkillSpector detects these patterns and produces structured findings with severity, recommendation, and output formats like Markdown and SARIF.

Manual scans are useful with SkillSpector but I do recommend adding it to your CI. CI scans are a control.

  1. Every PR/push is checked automatically.
  2. Scanning stays consistent over time.
  3. Findings are captured as artifacts and/or SARIF.
  4. Reviewers get fast feedback where they already work.

GitHub Actions Setup for SkillSpector

I’ve created two variations:

  • Baseline repository scan
  • Pull Requests scan that adds comments to the PR

For each, i’ve used Azure AI foundry to assist and created the secrets (Go to Settings → Secrets and variables → Actions → New repository secret):

Also: Enable GitHub Code Scanning (Settings → Security → Code scanning) to view SARIF results in the Security tab.

GitHub Action Baseline repository scan

This workflow scans the whole repository and writes SARIF.

It also handles SkillSpector exit codes deliberately. Exit code 1 means the scan completed and returned a DO_NOT_INSTALL recommendation. I would treat that as a finding, not a broken workflow. Exit code 2 is the execution/input failure case.

name: SkillSpector Security Scan

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]
  schedule:
    - cron: "0 6 * * 1" # Every Monday at 06:00 UTC
  workflow_dispatch:
    inputs:
      target:
        description: "Target to scan (git URL, file path, or leave blank to scan this repo)"
        required: false
        default: ""
      use_llm:
        description: "Enable LLM semantic analysis"
        required: false
        default: "true"
        type: choice
        options:
          - "true"
          - "false"

permissions:
  actions: read
  contents: read
  security-events: write # Required to upload SARIF to GitHub Code Scanning; note this has no effect on pull_request runs from forks (GITHUB_TOKEN is read-only there), so SARIF upload is skipped for fork PRs.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  skillspector:
    name: SkillSpector Scan
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

      - name: Set up Python
        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
        with:
          python-version: "3.12"

      - name: Install uv
        uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4

      - name: Install SkillSpector
        # Pinned to a commit SHA: upstream has no tagged releases yet, only branches.
        # Re-pin periodically to pick up fixes/updates.
        run: uv tool install "git+https://github.com/NVIDIA/skillspector.git@326a2b489411a20ed742ff13701be39ba00063c8"

      - name: Determine scan target
        id: target
        run: |
          INPUT="${{ github.event.inputs.target }}"
          if [ -n "$INPUT" ]; then
            echo "path=$INPUT" >> "$GITHUB_OUTPUT"
          else
            echo "path=." >> "$GITHUB_OUTPUT"
          fi

      - name: Run SkillSpector — SARIF (with LLM)
        id: scan_sarif
        env:
          SKILLSPECTOR_PROVIDER: ${{ secrets.SKILLSPECTOR_PROVIDER != '' && secrets.SKILLSPECTOR_PROVIDER || 'openai' }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
          SKILLSPECTOR_MODEL: ${{ secrets.SKILLSPECTOR_MODEL }}
        run: |
          TARGET="${{ steps.target.outputs.path }}"
          LLM_FLAG=""

          if [ "${{ github.event.inputs.use_llm }}" = "false" ] || [ -z "$OPENAI_API_KEY" ]; then
            LLM_FLAG="--no-llm"
            echo "LLM disabled (no key or explicitly disabled)"
          fi

          set +e
          skillspector scan "$TARGET" $LLM_FLAG \
            --format sarif --output skillspector-results.sarif
          SCAN_EXIT=$?
          set -e

          if [ "$SCAN_EXIT" -eq 2 ]; then
            echo "SkillSpector failed to scan due to an execution/input error."
            exit 2
          fi

          if [ "$SCAN_EXIT" -eq 1 ]; then
            echo "SkillSpector reported DO_NOT_INSTALL findings; continuing to upload reports."
          fi

      - name: Upload SARIF to GitHub Code Scanning
        if: always()
        uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
        with:
          sarif_file: skillspector-results.sarif
          category: skillspector

      - name: Upload reports as artifacts
        if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
        with:
          name: skillspector-reports
          path: skillspector-results.sarif
          retention-days: 30

      - name: Print summary to job log
        if: always()
        run: |
          {
            echo "## SkillSpector Report"
            echo ""
            echo "Full SARIF results uploaded to the Security tab (category: \`skillspector\`, if permissions allow) and attached as the \`skillspector-reports\` workflow artifact."
          } >> "$GITHUB_STEP_SUMMARY"

Example of code scanning alert from the above on GitHub:

Example of code scanning alert from the nvidia skillspector GitHub Action
Example of code scanning alert from the nvidia skillspector GitHub Action

Focused pull request scan

A repository-wide scan is useful for coverage, but it is not always the best PR experience. If someone changes one skill, reviewers want the result for that skill.

This workflow finds changed skill directories and instruction files, scans those targets, uploads the reports as an artifact, and updates one PR comment for same-repository pull requests.

name: SkillSpector PR Skills Scan

on:
  pull_request:
    types: [opened, synchronize, reopened]
    paths:
      - ".github/workflows/skills/**"
      # Add one entry per root listed in TARGET_ROOTS below so the workflow
      # only triggers when files under those paths actually change.
      # - "skills/**"
      # - "agents/skills/**"

permissions:
  contents: read
  issues: write
  pull-requests: write

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  skillspector-pr-skills:
    name: Scan changed skill folders
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
        with:
          python-version: "3.12"

      - name: Install uv
        uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4

      - name: Install SkillSpector
        # Pinned to a commit SHA: upstream has no tagged releases yet, only branches.
        # Re-pin periodically to pick up fixes/updates.
        run: uv tool install "git+https://github.com/NVIDIA/skillspector.git@326a2b489411a20ed742ff13701be39ba00063c8"

      - name: Scan changed skill directories and build PR report
        id: scan
        env:
          SKILLSPECTOR_PROVIDER: ${{ secrets.SKILLSPECTOR_PROVIDER != '' && secrets.SKILLSPECTOR_PROVIDER || 'openai' }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
          SKILLSPECTOR_MODEL: ${{ secrets.SKILLSPECTOR_MODEL }}
        run: |
          BASE_SHA="${{ github.event.pull_request.base.sha }}"
          HEAD_SHA="${{ github.event.pull_request.head.sha }}"
          # One or more roots to scan for changed skill folders. Keep this in
          # sync with the `paths:` filter above.
          TARGET_ROOTS=(".github/workflows/skills")

          mkdir -p skillspector-pr-reports

          mapfile -t CHANGED_FILES < <(git diff --name-only --diff-filter=ACMR "$BASE_SHA" "$HEAD_SHA" -- "${TARGET_ROOTS[@]}")

          MARKER="<!-- skillspector-pr-skills -->"
          ROOTS_LIST="$(IFS=', '; echo "${TARGET_ROOTS[*]}")"

          if [ "${#CHANGED_FILES[@]}" -eq 0 ]; then
            echo "No added/updated files found under: ${ROOTS_LIST}."
            {
              echo "$MARKER"
              echo "## SkillSpector PR Skills Scan"
              echo ""
              echo "No added/updated files found under: \`${ROOTS_LIST}\`."
            } > skillspector-pr-reports/summary.md
            exit 0
          fi

          declare -A SEEN_DIRS=()
          SKILL_DIRS=()

          for file in "${CHANGED_FILES[@]}"; do
            for root in "${TARGET_ROOTS[@]}"; do
              case "$file" in
                "$root"/*)
                  rel="${file#${root}/}"
                  skill_name="${rel%%/*}"

                  if [ -z "$skill_name" ] || [ "$skill_name" = "$rel" ]; then
                    break
                  fi

                  dir="${root}/${skill_name}"
                  if [ -d "$dir" ] && [ -z "${SEEN_DIRS[$dir]+x}" ]; then
                    SEEN_DIRS[$dir]=1
                    SKILL_DIRS+=("$dir")
                  fi
                  break
                  ;;
              esac
            done
          done

          if [ "${#SKILL_DIRS[@]}" -eq 0 ]; then
            {
              echo "$MARKER"
              echo "## SkillSpector PR Skills Scan"
              echo ""
              echo "Changes were detected under \`${ROOTS_LIST}\`, but no skill directories were available to scan."
            } > skillspector-pr-reports/summary.md
            exit 0
          fi

          LLM_FLAG=""
          LLM_MODE="enabled"
          if [ -z "$OPENAI_API_KEY" ]; then
            LLM_FLAG="--no-llm"
            LLM_MODE="disabled (OPENAI_API_KEY not set)"
            echo "LLM disabled (OPENAI_API_KEY not set)."
          fi

          HIGH_RISK_FOUND=0

          {
            echo "$MARKER"
            echo "## SkillSpector PR Skills Scan"
            echo ""
            echo "- Base SHA: \`$BASE_SHA\`"
            echo "- Head SHA: \`$HEAD_SHA\`"
            echo "- LLM: $LLM_MODE"
            echo ""
            echo "Scanned directories:"
            for dir in "${SKILL_DIRS[@]}"; do
              echo "- \`${dir}\`"
            done
            echo ""
            echo "---"
            echo ""
          } > skillspector-pr-reports/summary.md

          for dir in "${SKILL_DIRS[@]}"; do
            slug="$(echo "$dir" | tr '/.' '__')"

            set +e
            skillspector scan "$dir" $LLM_FLAG --format markdown --output "skillspector-pr-reports/${slug}.md"
            MD_EXIT=$?
            set -e

            if [ "$MD_EXIT" -eq 2 ]; then
              echo "Failed generating markdown report for ${dir}."
              exit 2
            fi

            if [ "$MD_EXIT" -eq 1 ]; then
              HIGH_RISK_FOUND=1
            fi

            {
              echo "### \`${dir}\`"
              echo ""
              cat "skillspector-pr-reports/${slug}.md"
              echo ""
              echo "---"
              echo ""
            } >> skillspector-pr-reports/summary.md
          done

          if [ "$HIGH_RISK_FOUND" -eq 1 ]; then
            {
              echo "High-risk findings were detected in one or more changed skill directories."
              echo "Review the report sections above before merging."
            } >> skillspector-pr-reports/summary.md
          else
            echo "No DO_NOT_INSTALL outcome detected in changed skill directories." >> skillspector-pr-reports/summary.md
          fi

          COMMENT_MAX=65000
          CURRENT_SIZE=$(wc -c < skillspector-pr-reports/summary.md | tr -d ' ')
          if [ "$CURRENT_SIZE" -gt "$COMMENT_MAX" ]; then
            {
              echo ""
              echo "> ⚠️ Report was truncated to fit GitHub PR comment limits."
            } > skillspector-pr-reports/truncation-note.md
            head -c "$COMMENT_MAX" skillspector-pr-reports/summary.md > skillspector-pr-reports/pr-comment.md
            cat skillspector-pr-reports/truncation-note.md >> skillspector-pr-reports/pr-comment.md
          else
            cp skillspector-pr-reports/summary.md skillspector-pr-reports/pr-comment.md
          fi

      - name: Create or update PR comment
        if: always()
        uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
        with:
          script: |
            const fs = require("fs");
            const marker = "<!-- skillspector-pr-skills -->";
            const fallback = `${marker}\n## SkillSpector PR Skills Scan\n\nThe scan step failed before a full report could be generated. Check the workflow logs.`;
            const body = fs.existsSync("skillspector-pr-reports/pr-comment.md")
              ? fs.readFileSync("skillspector-pr-reports/pr-comment.md", "utf8")
              : fallback;
            const issue_number = context.payload.pull_request.number;
            const { owner, repo } = context.repo;

            const comments = await github.paginate(github.rest.issues.listComments, {
              owner,
              repo,
              issue_number,
              per_page: 100,
            });

            const existing = comments.find(c => c.user?.type === "Bot" && c.body?.includes(marker));
            if (existing) {
              await github.rest.issues.updateComment({
                owner,
                repo,
                comment_id: existing.id,
                body,
              });
            } else {
              await github.rest.issues.createComment({
                owner,
                repo,
                issue_number,
                body,
              });
            }

      - name: Print summary to job log
        if: always()
        run: |
          if [ -f skillspector-pr-reports/pr-comment.md ]; then
            cat skillspector-pr-reports/pr-comment.md >> "$GITHUB_STEP_SUMMARY"
          else
            echo "## SkillSpector PR Skills Scan" >> "$GITHUB_STEP_SUMMARY"
            echo "" >> "$GITHUB_STEP_SUMMARY"
            echo "Report was not generated. Check earlier step logs." >> "$GITHUB_STEP_SUMMARY"
          fi
Screenshot of NVIDIA SkillSpector adding PR comment of report from GitHub Action

See full output here

Wrapping Up

This pattern fits shared repositories of GitHub Copilot skills, internal engineering prompts, platform standards, Terraform guidance, API operation patterns, diagramming rules, and MCP configuration examples.

It is especially useful when skills are copied or reused across teams. Once people start trusting a skill, the repository holding it needs consistent checks and reviewable output.

I would be more cautious with blanket full-repository scanning everywhere. Some repositories are too large, too noisy, or contain content that should not be sent to an LLM provider. Start with targeted static scans if that is the safer path.

If a file can influence what an agent does, treat it as part of the engineering system. Scan it, review it, pin the tooling around it, and make the output visible where engineers already work. The control does not need to be clever. It needs to run every time.

GitHub repository containing all setup examples of the above (& includes a local setup as well)

Leave a Reply

Discover more from Thomas Thornton Blog

Subscribe now to keep reading and get access to the full archive.

Continue reading

Discover more from Thomas Thornton Blog

Subscribe now to keep reading and get access to the full archive.

Continue reading