Create and Test S3 Service

Set up a MinIO-based S3 service in a local development environment and test it using Ansible plays to create buckets, upload, and download files.

Tags:

Projects: c2platform/phx/ansible , c2platform.core , c2platform.mw

Overview

This how-to guide shows how to set up a simple S3-compatible service using MinIO in the PHX development environment. It includes two Ansible plays: one to provision the S3 service on a node named pxd-s3, and another to test it by creating a bucket, uploading a file, and downloading it. This verifies compatibility with the amazon.aws Ansible collection for managing S3 resources.

The setup runs locally using Vagrant and LXD, simulating an S3 service that can serve as a starting point for developing Ansible roles, such as those for automated binary downloads in lifecycle management (LCM) tasks.

Executing vagrant up pxd-s3 creates an LXD node running Ubuntu 22.04 and deploys a MinIO container. The container exposes port 9000 for the S3 API (compatible with tools like AWS CLI or Ansible’s amazon.aws modules) and port 9090 for the MinIO Console web interface.

With the development environment prerequisites met, you can provision the S3 service in about 5 minutes and run tests immediately after.

Prerequisites

Setting Up the PHX Development Environment on Ubuntu 22.04

Setup

To create the S3 node pxd-s3, which provides an S3-compatible service based on MinIO, execute the following command from the root of your local clone of the PHX inventory project:

vagrant up pxd-s3

Show me

Bringing machine 'pxd-s3' up with 'lxd' provider...
==> pxd-s3: Machine has not been created yet, starting...
==> pxd-s3: Importing LXC image...
==> pxd-s3: Mounting shared folders...
    pxd-s3: /vagrant => /home/onknows/git/gitlab/c2/ansible-phx
    pxd-s3: /home/vagrant/.marker => /home/onknows/.marker
    pxd-s3: /home/vagrant/.local/share/marker => /home/onknows/.local/share/marker
    pxd-s3: /root/.marker => /home/onknows/.marker
    pxd-s3: /root/.local/share/marker => /home/onknows/.local/share/marker
    pxd-s3: /software => /software/projects/phx
    pxd-s3: /software-cache => /software/projects/phx/cache
    pxd-s3: /ansible-dev-collections => /home/onknows/git/gitlab/c2/ansible-dev-collections
==> pxd-s3: Waiting for machine to boot. This may take a few minutes...
    pxd-s3: SSH address: 10.190.101.188:22
    pxd-s3: SSH username: vagrant
    pxd-s3: SSH auth method: private key
==> pxd-s3: Machine booted and ready!
==> pxd-s3: Setting hostname...
==> pxd-s3: Running provisioner: shell...
    pxd-s3: Running: inline script
==> pxd-s3: Running provisioner: ansible...
    pxd-s3: Running ansible-playbook...
[DEPRECATION WARNING]: community.general.yaml has been deprecated. The plugin
has been superseded by the the option `result_format=yaml` in callback plugin
ansible.builtin.default from ansible-core 2.13 onwards. This feature will be
removed from community.general in version 13.0.0. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [S3] **********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [pxd-s3]

TASK [Include Linux roles] *****************************************************

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/update_cache.yml for pxd-s3

TASK [c2platform.core.server_update : Apt update cache] ************************
changed: [pxd-s3]

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/update.yml for pxd-s3

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/debian.yml for pxd-s3

TASK [c2platform.core.server_update : Upgrade all packages] ********************
changed: [pxd-s3]

TASK [c2platform.core.server_update : Check reboot] ****************************
ok: [pxd-s3]

TASK [c2platform.core.server_update : Fact server_update_reboot] ***************
ok: [pxd-s3]

TASK [c2platform.core.bootstrap : Include package tasks] ***********************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=['nano', 'wget', 'tree', 'unzip', 'zip', 'jq', 'build-essential', 'python3-dev', 'python3-wheel', 'libsasl2-dev', 'libldap2-dev', 'libssl-dev', 'git', 'git-lfs', 'nfs-common', 'net-tools', 'telnet', 'curl', 'dnsutils', 'python2'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=python3-pip)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/pip.yml for pxd-s3 => (item=['docker', 'requests==2.28.1'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/pip.yml for pxd-s3 => (item=['botocore', 'boto3'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=['realmd', 'sssd', 'sssd-ad', 'sssd-krb5', 'krb5-user', 'adcli', 'policykit-1', 'sssd-tools', 'libnss-sss', 'libpam-sss', 'bind9-utils', 'samba-common-bin'])

TASK [c2platform.core.bootstrap : OS package] **********************************
changed: [pxd-s3] => (item=['nano', 'wget', 'tree', 'unzip', 'zip', 'jq', 'build-essential', 'python3-dev', 'python3-wheel', 'libsasl2-dev', 'libldap2-dev', 'libssl-dev', 'git', 'git-lfs', 'nfs-common', 'net-tools', 'telnet', 'curl', 'dnsutils', 'python2'])

TASK [c2platform.core.bootstrap : OS package] **********************************
ok: [pxd-s3] => (item=python3-pip)

TASK [c2platform.core.bootstrap : PIP package] *********************************
changed: [pxd-s3] => (item=['docker', 'requests==2.28.1'])

TASK [c2platform.core.bootstrap : PIP package] *********************************
changed: [pxd-s3] => (item=['botocore', 'boto3'])

TASK [c2platform.core.bootstrap : OS package] **********************************
changed: [pxd-s3] => (item=['realmd', 'sssd', 'sssd-ad', 'sssd-krb5', 'krb5-user', 'adcli', 'policykit-1', 'sssd-tools', 'libnss-sss', 'libpam-sss', 'bind9-utils', 'samba-common-bin'])

TASK [c2platform.core.os_trusts : CA distribute ( Debian )] ********************
changed: [pxd-s3] => (item=https://letsencrypt.org/certs/isrgrootx1.pem)
changed: [pxd-s3] => (item=file:///vagrant/.ca/c2/c2.crt)

TASK [c2platform.core.os_trusts : Execute update-ca-certificates ( Debian )] ***
changed: [pxd-s3] => (item=https://letsencrypt.org/certs/isrgrootx1.pem)
changed: [pxd-s3] => (item=file:///vagrant/.ca/c2/c2.crt)

TASK [c2platform.core.secrets : Stat secret dir] *******************************
ok: [pxd-s3 -> localhost] => (item=/home/onknows/git/gitlab/c2/ansible-phx/secret_vars/development)
ok: [pxd-s3 -> localhost] => (item=/runner/project/secret_vars/development)

TASK [c2platform.core.secrets : Include secrets] *******************************
ok: [pxd-s3] => (item=/home/onknows/git/gitlab/c2/ansible-phx/secret_vars/development)

TASK [c2platform.core.linux : Include linux_resources] *************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/fail.yml for pxd-s3 => (item=0_bootstrap Environment pxd-s3 → development)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/lineinfile.yml for pxd-s3 => (item=marker Marker)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/file.yml for pxd-s3 => (item=marker Python link for marker)

TASK [c2platform.core.linux : Manage lines in text files] **********************
changed: [pxd-s3] => (item=/home/vagrant/.bashrc)
changed: [pxd-s3] => (item=/root/.bashrc)

TASK [c2platform.core.linux : Manage files and file properties] ****************
changed: [pxd-s3] => (item=/usr/bin/python → link)

TASK [geerlingguy.docker : include_tasks] **************************************
included: /home/onknows/git/gitlab/c2/ansible-phx/roles/external/geerlingguy.docker/tasks/setup-Debian.yml for pxd-s3

TASK [geerlingguy.docker : Ensure old versions of Docker are not installed.] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Ensure dependencies are installed.] *****************
ok: [pxd-s3]

TASK [geerlingguy.docker : Ensure additional dependencies are installed (on Ubuntu >= 20.04).] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Add Docker apt key.] ********************************
changed: [pxd-s3]

TASK [geerlingguy.docker : Add Docker repository.] *****************************
changed: [pxd-s3]

TASK [geerlingguy.docker : Install Docker (Ansible >=2.12).] *******************
changed: [pxd-s3]

TASK [geerlingguy.docker : Ensure Docker is started and enabled at boot.] ******
ok: [pxd-s3]

RUNNING HANDLER [geerlingguy.docker : restart docker] **************************
changed: [pxd-s3]

TASK [geerlingguy.docker : include_tasks] **************************************
included: /home/onknows/git/gitlab/c2/ansible-phx/roles/external/geerlingguy.docker/tasks/docker-compose.yml for pxd-s3

TASK [geerlingguy.docker : Check current docker-compose version.] **************
ok: [pxd-s3]

TASK [geerlingguy.docker : set_fact] *******************************************
ok: [pxd-s3]

TASK [geerlingguy.docker : Delete existing docker-compose version if it's different.] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Install Docker Compose (if configured).] ************
changed: [pxd-s3]

TASK [c2platform.mw.docker : Networks] *****************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.mw.docker : Images] *******************************************
changed: [pxd-s3] => (item=quay.io/minio/minio)

TASK [c2platform.mw.docker : Volumes] ******************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.mw.docker : Container] ****************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.core.linux : Include linux_resources] *************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/user.yml for pxd-s3 => (item=s3 vagrant)

TASK [c2platform.core.linux : Manage users on a system] ************************
changed: [pxd-s3] => (item=vagrant → present)

PLAY RECAP *********************************************************************
pxd-s3                     : ok=47   changed=20   unreachable=0    failed=0    skipped=33   rescued=0    ignored=0

This command uses the Vagrantfile.yml configuration to provision the node. It applies the playbook plays/mgmt/s3.yml, which installs Docker, pulls the MinIO image, and starts the container with default credentials (admin user: admin, password: Supersecret!).

After provisioning, access the services at:

S3 API	MinIO Console
`http://192.168.60.14:9000`	`http://192.168.60.14:9090`

S3 Test Play

To test the S3 service, run the test playbook on the pxd-s3 node. This play creates an S3 bucket named ansible, downloads a Tomcat archive from the internet, uploads it to the bucket, and then downloads it back to verify the process.

From the inventory project root, execute:

PLAY=mgmt/s3_test vagrant provision pxd-s3

Show me

==> pxd-s3: Running provisioner: shell...
    pxd-s3: Running: inline script
==> pxd-s3: Running provisioner: ansible...
    pxd-s3: Running ansible-playbook...
[WARNING]: Collection amazon.aws does not support Ansible version 2.16.0
[DEPRECATION WARNING]: community.general.yaml has been deprecated. The plugin 
has been superseded by the the option `result_format=yaml` in callback plugin 
ansible.builtin.default from ansible-core 2.13 onwards. This feature will be 
removed from community.general in version 13.0.0. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [S3] **********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [pxd-s3]

TASK [Create "ansible" bucket] *************************************************
changed: [pxd-s3]
[WARNING]: packaging.version Python module not installed, unable to check AWS
SDK versions
[WARNING]: Failed to get bucket public access block settings (not supported by
cloud)
[WARNING]: Failed to get bucket ownership settings (not supported by cloud)
[WARNING]: Failed to get bucket inventory settings (not supported by cloud)

TASK [Download Tomcat from internet] *******************************************
changed: [pxd-s3]

TASK [Upload to S3] ************************************************************
changed: [pxd-s3]

TASK [Download from S3] ********************************************************
changed: [pxd-s3]

PLAY RECAP *********************************************************************
pxd-s3                     : ok=5    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

This uses modules from the amazon.aws collection to interact with the S3 endpoint.

Verify

To manually verify the test results, navigate to the MinIO Console web interface http://192.168.60.14:9090/. Log in with username admin and password Supersecret!.

After login, you should see the bucket ansible containing the file apache-tomcat-10.1.48.tar.gz.

Review

This section reviews the key files in the PHX inventory project that enable the S3 setup and testing. They demonstrate integration between Vagrant, Ansible, and the amazon.aws collection.

Vagrantfile.yml

The Vagrantfile.yml defines the pxd-s3 node using the ubuntu22-lxd box, assigns IP 192.168.60.14, and specifies the playbook mgmt/s3 for provisioning. Vagrant uses this to create and configure the node.

Vagrantfile.yml

201  - name: s3
202    short_description: S3
203    description: MinIO S3
204    box: ubuntu22-lxd
205    ip-address: 192.168.60.14
206    plays:
207      - mgmt/s3

hosts.ini

The inventory file hosts.ini defines Ansible groups s3 and s3_download_server. In this test setup, pxd-s3 is in both groups, acting as both the S3 server and the test client. In production-like scenarios, these roles would be separated.

hosts.ini

43[s3]
44pxd-s3
45
46[s3_download_server]
47pxd-s3

Playbooks

Two playbooks are used:

plays/mgmt/s3.yml: Provisions the MinIO service on the s3 group, applying roles for Linux setup and Docker management.
plays/mgmt/s3_test.yml: Tests S3 operations on the s3 group using amazon.aws modules to create a bucket, upload, and download a file.

plays/mgmt/s3.yml

---
- name: S3
  hosts: s3
  become: true

  roles:
    - { role: c2platform.core.linux }
    - { role: geerlingguy.docker, tags: ["docker"] }
    - { role: c2platform.mw.docker, tags: ["docker"] }

plays/mgmt/s3_test.yml

---
- name: S3
  hosts: s3

  vars:
    px_tomcat_download_url: >-
      https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.48/bin/apache-tomcat-10.1.48.tar.gz
    px_upload_src: "/home/vagrant/{{ px_tomcat_download_url | basename }}"
    px_upload_dest: "tomcat/{{ px_tomcat_download_url | basename }}"
    px_download_dest: "/home/vagrant/s3-tomcat.tar.gz"

  tasks:
    - name: Create "ansible" bucket
      amazon.aws.s3_bucket:
        access_key: admin
        secret_key: Supersecret!
        name: ansible
        state: present
        endpoint_url: http://192.168.60.14:9000

    - name: Download Tomcat from internet
      ansible.builtin.get_url:
        url: "{{ px_tomcat_download_url }}"
        dest: "{{ px_upload_src }}"

    - name: Upload to S3
      amazon.aws.s3_object:
        access_key: admin
        secret_key: Supersecret!
        bucket: ansible
        object: "{{ px_upload_dest }}"
        src: "{{ px_upload_src }}"
        mode: put
        endpoint_url: http://192.168.60.14:9000
        encrypt: false

    - name: Download from S3
      amazon.aws.s3_object:
        access_key: admin
        secret_key: Supersecret!
        mode: get
        bucket: ansible
        dest: "{{ px_download_dest }}"
        object: "{{ px_upload_dest }}"
        endpoint_url: http://192.168.60.14:9000

Group Variables

group variables configure the groups:

group_vars/s3/main.yml: Defines Docker resources for MinIO, including the image, container, and environment variables.
group_vars/s3_download_server/main.yml: Installs Python dependencies like botocore and boto3 for S3 interactions.

group_vars/s3/main.yml

---
bootstrap_packages:
  2_s3:
    - name:
        - docker
        - requests==2.28.1

docker_networks:
  - name: minio

docker_images:
  - name: quay.io/minio/minio
    tag: latest

docker_volumes:
  - name: minio

docker_containers:
  - name: minio
    image: "{{ docker_images[0]['name'] }}:{{ docker_images[0]['tag'] }}"
    ports:
      - "9000:9000"
      - "9090:9090"
    volumes:
      - minio:/data
    restart_policy: unless-stopped
    env:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: Supersecret!  # vault
    command: server /data --console-address ":9090"

linux_resource_groups_disabled: [kerberos]

docker_resources:
  s3:
    - name: vagrant
      module: user
      groups: docker
      append: true

group_vars/s3_download_server/main.yml

---
bootstrap_packages:
  2_s3_download_server:
    - name:
        - botocore
        - boto3

collections/requirements.yml

This file specifies required collections, including amazon.aws for S3 modules.

collections/requirements.yml

13  - name: amazon.aws
14    version: 10.1.2

Additional Information

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified November 7, 2025: phx how-to s3 PHX-270 PHX-275 (5fab55b)