Skip to content
Create free account
Create free account
CI/CD7 min read

Masked CI variables still leak: where masking stops

CI masking replaces exact strings in the job log, nothing more. Encoded values, JSON blobs, artifacts and untrusted pull requests all get past it.

Crunu teamPublished 7 min read

Example. Kiteline is a fictional company.
On this page

Masking hides a secret only when the exact string the CI knows about appears in the job log. GitHub Actions replaces it with ***, GitLab with [MASKED]. If the value is encoded, split, escaped or wrapped in JSON on its way out, the log shows it in full. If it leaves through an artifact, a cache, a file or a network request, the log was never involved. And any code that runs with the secret in its environment can send it anywhere. Masking is a guard against an accidental echo, not a control over who gets the value. The controls that work are giving the secret only to the step that uses it, preferring short-lived credentials, and keeping production secrets out of CI.

This post goes through where masking stops on GitHub Actions and GitLab CI, and what to do instead. The guide to CI/CD secrets with OIDC covers replacing stored keys altogether.

What masking actually is

On GitHub, "redacting of secrets is performed by your workflow runners", and the documentation is direct about the method: "redaction largely relies on finding an exact match for the specific secret value." It adds that "a secret will only be redacted if it was used within a job and is accessible by the runner", and, in so many words, that "automatic redaction is not guaranteed" (GitHub: security hardening for GitHub Actions).

GitLab says the same from the other side: "Masking a CI/CD variable is not a guaranteed way to prevent malicious users from accessing variable values." A value can be masked only if it is "a single line with no spaces", "8 characters or longer", and, when variable expansion is on, made of letters, digits and a short list of symbols (GitLab: CI/CD variables). A private key in PEM format is multiline, so it can't be a masked variable at all.

So the mechanism is a search-and-replace over log output. Everything below follows from that.

Five ways a value gets past the mask

1. Encoding. Base64, URL encoding, hex or a hash produce a different string. GitHub's guidance: if a workflow transforms a secret, "be sure to register the new value as a secret too." The same applies to values your scripts generate, such as a JWT signed with a secret key: "register that JWT as a secret or else it won't be redacted if it ever enters the log output." On GitHub that means masking the derived value yourself:

YAML
- name: Build basic auth header
  run: |
    header=$(printf '%s:%s' "$API_USER" "$API_PASSWORD" | base64)
    echo "::add-mask::$header"
    echo "AUTH_HEADER=$header" >> "$GITHUB_ENV"

2. Pieces. A tool that prints the first eight characters "for debugging", an error message that truncates a long value, or a loop that prints one line of a multiline secret at a time: none of these outputs equals the full string, so none are replaced.

3. Blobs. Storing several values as one JSON or YAML secret defeats exact matching, because the individual values appear in the log without their surrounding structure. GitHub's advice is to "not use a blob of JSON, XML, or YAML (or similar) to encapsulate a secret value" and to "create individual secrets for each sensitive value."

4. Escaping. GitLab's documentation gives the example of a shell that prints My[value] as My\[value\]: "If a process outputs the value in a slightly modified way, the value can't be masked." Quoting by a logger, JSON string escaping and line wrapping do the same.

5. Values that can't be masked. Short values, values with spaces, and multiline values fall outside GitLab's rules, so they are stored unmasked unless you change their form.

None of these needs bad intent. A set -x in a deploy script, a verbose HTTP client, or a test runner that dumps its configuration on failure is enough.

The log is only one exit

Masking edits one stream. Secrets leave a job through several others.

  • Artifacts. Build outputs, test reports and coverage files are uploaded and kept after the job ends. Until version 4.4.0, actions/upload-artifact uploaded hidden files by default. Since then they are excluded, which the release notes explain as reducing the risk of credentials ending up in artifacts, and you opt back in with include-hidden-files (actions/upload-artifact v4.4.0). A .env is a hidden file; a config.production.json is not.
  • Caches. A dependency cache that happens to include a configuration file carries it into the next run, possibly on another branch.
  • Credentials on disk. actions/checkout keeps the job's token for git by default; "the token is removed during post-job cleanup", and persist-credentials: false opts out (actions/checkout). Any later step can read it until then.
  • Command lines. GitHub notes that on a self-hosted runner reused between jobs, "some jobs will use secrets as command-line arguments which can be seen by another job running on the same runner." Process arguments are visible to other processes on the machine.
  • Debug modes. GitLab warns that with CI_DEBUG_SERVICES on, "the variable value might be revealed."
Example. Kiteline is a fictional company.

Code you didn't write runs with your secrets

The most direct leak needs no mistake at all: a step that runs untrusted code with the secret in its environment can read the value and send it over the network. Masking doesn't see network traffic.

On GitHub, the risky pattern is a privileged trigger combined with a checkout of a pull request. The documentation warns that pull_request_target and workflow_run, "when used with the checkout of an untrusted pull request, expose the repository to security compromises", because such workflows "may have repository write access and access to referenced secrets." Its advice: "Avoid using the pull_request_target workflow trigger if it's not necessary", and workflows that use it "must not explicitly check out untrusted code."

The same logic applies to every dependency your build installs and every action you pin by tag instead of commit. They run inside a job that holds whatever secrets the job holds.

Give each secret to one step

The narrower the scope, the less there is for any of the above to reach.

On GitHub Actions, put secrets on the step that uses them, not on the job or the workflow:

YAML
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          persist-credentials: false
      - run: npm ci && npm test
      - name: Upload coverage
        env:
          COVERAGE_TOKEN: ${{ secrets.COVERAGE_TOKEN }}
        run: ./scripts/upload-coverage.sh

Here npm ci and the tests never see COVERAGE_TOKEN, so a compromised dependency that runs during install can't read it from the environment.

On GitLab, mark variables Protected so they reach only "protected branches or protected tags", and use Masked and hidden for values nobody needs to read back in the UI. GitLab also suggests file-type variables so that env or printenv doesn't print the secret.

Better still, replace stored keys with OIDC. A cloud credential obtained through a CI job's identity token expires on its own, and a trust policy limits which repository and branch can get it; Giving CI only the secrets it needs explains the trade-off.

Keep production values out of the pipeline

Masking matters most for values that hurt when leaked, and those are usually production values. The surest way to keep them out of logs, artifacts and pull request builds is not to give them to CI at all: the production server fetches its own secrets, and the pipeline only ships code and restarts the service. Secrets on servers without files on disk describes that setup with the Crunu agent.

When a job does need secrets from Crunu, three things narrow what it holds:

  • A service token per pipeline, granted only the environments that pipeline uses. A test job's token for web-app/staging has no key for production, so it can't decrypt production even if it's copied.
  • A CI binding (OIDC), so the token works only together with a fresh JWT from the right repository and branch. A token pasted from a log fails elsewhere.
  • crunu run around one command, so the variables go into one process's environment instead of the whole job's:
Shell
crunu run --env web-app/staging -- npm test

crunu run starts the program with a clean environment plus the environment's variables, and never passes its own credential variables to the program. That doesn't stop the program from printing a value; it keeps the value away from the steps before and after it.

Example. Kiteline is a fictional company.

If a secret shows up anyway

Treat a value in a log as leaked, masked elsewhere or not. GitHub's instruction is short: "delete the log and rotate the secret." Reissue the credential at the provider first, then clean up logs, artifacts and caches, because copies may already exist. The rotation pages list the steps per provider, and rotating API keys without downtime covers swapping a key while the old one still works. The glossary entry on masked variables has the short definition.

Checklist

Sources

Terms in this post