GitLab Runner Executor Test Matrix

This project provides a comprehensive test suite for validating GitLab Runner configurations across different executor types. It is part of the PHX (Phoenix) Ansible Reference Implementation and serves as a validation framework for testing GitLab Runner functionality in various deployment scenarios.

Refer to Manage GitLab Runners | C2 Platform for more information about GitLab Runner configuration in the PHX environment.

Purpose

This test project validates GitLab Runner operations across four different runner/executor combinations:

1. Docker-in-Docker (DinD) operations - Testing containerized builds within containerized runners
2. Shell executor functionality - Validating direct shell command execution
3. Cross-executor compatibility - Ensuring pipelines work across different infrastructure types
4. Runner infrastructure validation - Confirming proper configuration of PHX GitLab Runners

Runner Test Matrix

The project tests four distinct runner configurations. These configurations are defined in the PHX Ansible inventory at group_vars/gitlab_runner/runners.yml  .

Runner Configuration Details

1. VM Runner + Docker Executor (vm-runner, docker-exe)

• Infrastructure: GitLab Runner binary installed directly on a VM
• Executor: Docker - jobs run in Docker containers
• Docker Access: Uses host Docker daemon
• Best For: Standard containerized builds with full Docker access
• Configuration: See inventory_hostname-docker in PHX ansible config

2. VM Runner + Shell Executor (vm-runner, shell-exe)

• Infrastructure: GitLab Runner binary installed directly on a VM
• Executor: Shell - jobs run directly on the host shell
• Docker Access: May or may not have Docker depending on host setup
• Best For: Native builds, system-level operations, non-containerized workflows
• Configuration: See inventory_hostname-shell in PHX ansible config

3. Container Runner + Docker Executor (container-runner, docker-exe)

• Infrastructure: GitLab Runner running as a Docker container
• Executor: Docker - jobs run in nested Docker containers (DinD)
• Docker Access: Requires Docker-in-Docker (DinD) service
• Special Requirements:
  • Docker-in-Docker service (docker:dind)
  • TLS certificates configuration
  • DOCKER_HOST=tcp://docker:2376
  • Privileged mode may be needed
• Best For: Fully containerized CI/CD environments
• Configuration: See inventory_hostname-dind in PHX ansible config
• Custom Image: Uses gitlab/c2-gitlab-runner:latest with C2 certificates

4. Container Runner + Shell Executor (container-runner, shell-exe)

• Infrastructure: GitLab Runner running as a Docker container
• Executor: Shell - jobs run in the runner container’s shell
• Docker Access: Typically no Docker access (running inside container)
• Best For: Lightweight jobs in containerized runner environments
• Configuration: See inventory_hostname-sind (Shell-in-Docker) in PHX ansible config
• Custom Image: Uses gitlab/c2-gitlab-runner:latest with C2 certificates

Pipeline Structure

The GitLab CI/CD pipeline (.gitlab-ci.yml) is organized into three stages:

Stage 1: Build

Job: build:test-image

• Builds a test Docker image on Docker-capable executors
• Uses matrix execution to test both VM and Container runners with Docker executor
• Tests Docker build operations and validates the built image
• Runs on:
  • vm-runner, docker-exe
  • container-runner, docker-exe

Stage 2: Test

Job: test:docker-operations

• Executes comprehensive Docker functionality tests
• Validates Docker commands: ps, images, build, run
• Tests Docker-in-Docker capability
• Generates detailed test result artifacts
• Runs on:
  • vm-runner, docker-exe
  • container-runner, docker-exe

Job: test:shell-operations

• Tests basic shell executor functionality
• Validates file system operations
• Checks environment variable access
• Tests Docker availability (if present)
• Runs on:
  • vm-runner, shell-exe
  • container-runner, shell-exe

Stage 3: Validate

Job: validate:matrix-complete

• Confirms all four runner types were successfully tested
• Provides summary of validation results
• Single job that runs after all tests complete

Job: diagnostics:inspect-runner (Manual)

• Detailed runner inspection across all four configurations
• System information gathering
• Tool availability checks
• Docker configuration details (if applicable)
• Triggered manually for troubleshooting

Files in This Project

Core Files

• Dockerfile: Test container image definition

  • Based on Alpine Linux
  • Includes Docker CLI, bash, curl, git
  • Used for testing containerized operations

• entrypoint.sh: Container entrypoint script

  • Displays environment information
  • Checks Docker availability
  • Runs validation tests

• test-docker.sh: Docker functionality test script

  • Tests Docker installation and daemon access
  • Validates Docker commands (ps, images, build, run)
  • Detects Docker-in-Docker configuration
  • Generates detailed test reports

• .gitlab-ci.yml: GitLab CI/CD pipeline configuration

  • Matrix execution for 4 runner types
  • Docker-in-Docker configuration
  • Comprehensive test coverage

Usage

Running the Full Test Suite

1. Push changes to trigger the pipeline automatically:

   git push origin master
   

2. Or trigger manually from GitLab UI:

   • Navigate to CI/CD → Pipelines
   • Click “Run Pipeline”

Understanding Test Results

Successful Pipeline

A successful pipeline indicates:

• ✅ All four runner types are properly configured
• ✅ Docker operations work on Docker executors
• ✅ Shell operations work on all executors
• ✅ Docker-in-Docker is properly configured for container runners

Artifacts

Each test job generates artifacts:

• Docker tests: docker-test-results.txt - Detailed Docker operation results
• Shell tests: test-shell.txt - Shell execution validation
• Diagnostics: Complete system information for each runner type

Artifacts are available for download from the pipeline page and expire after 30 days.

Manual Diagnostics

For detailed runner inspection:

1. Navigate to CI/CD → Pipelines → Select a pipeline
2. Find the diagnostics:inspect-runner jobs in the validate stage
3. Click the play button (▶️) on the specific runner type you want to inspect
4. Review the job output for detailed system information

Docker-in-Docker (DinD) Configuration

The most complex scenario tested is Container Runner + Docker Executor, which requires Docker-in-Docker:

Key Configuration Elements

services:
  - docker:24-dind

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_VERIFY: 1
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  DOCKER_TLS_CERTDIR: "/certs"

How DinD Works

1. GitLab Runner container starts
2. Job container is created inside runner
3. Docker-in-Docker (DinD) service container starts
4. DinD service generates TLS certificates
5. Job container connects to DinD daemon via TCP
6. Docker commands in job execute on DinD daemon

Common DinD Issues

• Certificate errors: Ensure DOCKER_TLS_CERTDIR is properly configured
• Connection refused: Check DOCKER_HOST points to tcp://docker:2376
• Permission denied: May require privileged mode in runner configuration

PHX Infrastructure Requirements

This test project requires the PHX environment to have:

1. At least one VM runner with both Docker and Shell executors configured
2. At least one Container runner with both Docker and Shell executors configured
3. Proper runner tags matching the matrix configuration
4. Docker daemon accessible on VM runners
5. Docker-in-Docker capability for container runners
6. C2 Platform certificates trusted in container runners

Verifying Runner Registration

Check that runners are registered with correct tags:

# On the GitLab instance
sudo gitlab-runner list

Expected tags for each runner:

• VM Docker runner: vm-runner, docker-exe, plus host-specific tags
• VM Shell runner: vm-runner, shell-exe, plus host-specific tags
• Container Docker runner: container-runner, docker-exe, plus host-specific tags
• Container Shell runner: container-runner, shell-exe, plus host-specific tags

Troubleshooting

Pipeline Fails on Specific Runner Type

1. Check runner is registered and active in GitLab (Settings → CI/CD → Runners)
2. Verify runner tags match the expected tags
3. Run manual diagnostics job for that specific runner type
4. Check runner logs on the host system

Docker-in-Docker Fails

1. Verify Docker-in-Docker service is starting:

   services:
     - docker:24-dind
   

2. Check TLS certificate generation in job logs

3. Ensure DOCKER_HOST is set to tcp://docker:2376

4. Verify runner has Docker privileged mode enabled (if required)

Shell Executor Can’t Find Commands

1. Check if required tools are installed on the host
2. Verify PATH is properly set for the gitlab-runner user
3. For container shell executors, check if tools are in the runner image

No Runners Available

1. Ensure runners are registered with the project or group
2. Check runner tags match job requirements
3. Verify runners are not paused
4. Check runner token is valid

Integration with PHX Ansible

This project validates the GitLab Runner configuration managed by PHX Ansible:

• Runner Configuration: group_vars/gitlab_runner/runners.yml
• Custom Image: group_vars/gitlab_runner/image.yml
• Runner Role: roles/gitlab_runner/

Changes to runner configuration in PHX Ansible should be validated by running this test pipeline.

Expected Outcomes

All Tests Pass

• GitLab Runners are properly configured
• Both VM and container-based runners are operational
• Docker executors can build and run containers
• Shell executors can execute commands
• Docker-in-Docker is working for containerized runners

Specific Test Fails

• Check the specific runner configuration
• Review job logs for detailed error messages
• Run manual diagnostics for the failing runner type
• Verify PHX Ansible runner configuration

Contributing

When modifying this project:

1. Test changes on all four runner types
2. Update documentation for any new test scenarios
3. Ensure backward compatibility with existing runners
4. Add new test cases as separate jobs when possible

Related Documentation

• PHX GitLab Runner Documentation
• GitLab Runner Executors 
• Docker-in-Docker with GitLab CI 
• PHX Ansible Repository 

License

This project is part of the C2 Platform PHX Reference Implementation.