gitea/ansible-ceph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

deep_scrub_choked

Browse Source
This commit is contained in:
a.pivkin
2026-04-06 07:08:17 +03:00
commit 9149dc1d7f
840 changed files with 52471 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/testing/development.rst
+4
View File
@@ -0,0 +1,4 @@
.. _development:
ceph-ansible testing for development
====================================
docs/source/testing/glossary.rst
+14
View File
@@ -0,0 +1,14 @@
Glossary
========
.. toctree::
:maxdepth: 1
index
running.rst
development.rst
scenarios.rst
modifying.rst
layout.rst
tests.rst
tox.rst
docs/source/testing/index.rst
+38
View File
@@ -0,0 +1,38 @@
.. _testing:
Testing
=======
``ceph-ansible`` has the ability to test different scenarios (collocated journals
or dmcrypt OSDs for example) in an isolated, repeatable, and easy way.
These tests can run locally with VirtualBox or via libvirt if available, which
removes the need to solely rely on a CI system like Jenkins to verify
a behavior.
* **Getting started:**
* :doc:`Running a Test Scenario <running>`
* :ref:`dependencies`
* **Configuration and structure:**
* :ref:`layout`
* :ref:`test_files`
* :ref:`scenario_files`
* :ref:`scenario_wiring`
* **Adding or modifying tests:**
* :ref:`test_conventions`
* :ref:`testinfra`
* **Adding or modifying a scenario:**
* :ref:`scenario_conventions`
* :ref:`scenario_environment_configuration`
* :ref:`scenario_ansible_configuration`
* **Custom/development repositories and packages:**
* :ref:`tox_environment_variables`
docs/source/testing/layout.rst
+60
View File
@@ -0,0 +1,60 @@
.. _layout:
Layout and conventions
----------------------
Test files and directories follow a few conventions, which makes it easy to
create (or expect) certain interactions between tests and scenarios.
All tests are in the ``tests`` directory. Scenarios are defined in
``tests/functional/`` and use the following convention for directory
structure:
.. code-block:: none
tests/functional/<distro>/<distro version>/<scenario name>/
For example: ``tests/functional/centos/7/journal-collocation``
Within a test scenario there are a few files that define what that specific
scenario needs for the tests, like how many OSD nodes or MON nodes. Tls
At the very least, a scenario will need these files:
* ``Vagrantfile``: must be symlinked from the root directory of the project
* ``hosts``: An Ansible hosts file that defines the machines part of the
cluster
* ``group_vars/all``: if any modifications are needed for deployment, this
would override them. Additionally, further customizations can be done. For
example, for OSDs that would mean adding ``group_vars/osds``
* ``vagrant_variables.yml``: Defines the actual environment for the test, where
machines, networks, disks, linux distro/version, can be defined.
.. _test_conventions:
Conventions
-----------
Python test files (unlike scenarios) rely on paths to *map* where they belong. For
example, a file that should only test monitor nodes would live in
``ceph-ansible/tests/functional/tests/mon/``. Internally, the test runner
(``py.test``) will *mark* these as tests that should run on a monitor only.
Since the configuration of a scenario already defines what node has a given
role, then it is easier for the system to only run tests that belong to
a particular node type.
The current convention is a bit manual, with initial path support for:
* mon
* osd
* mds
* rgw
* journal_collocation
* all/any (if none of the above are matched, then these are run on any host)
.. _testinfra:
``testinfra``
-------------
docs/source/testing/modifying.rst
+4
View File
@@ -0,0 +1,4 @@
.. _modifying:
Modifying (or adding) tests
===========================
docs/source/testing/running.rst
+169
View File
@@ -0,0 +1,169 @@
.. _running_tests:
Running Tests
=============
Although tests run continuously in CI, a lot of effort was put into making it
easy to run in any environment, as long as a couple of requirements are met.
.. _dependencies:
Dependencies
------------
There are some Python dependencies, which are listed in a ``requirements.txt``
file within the ``tests/`` directory. These are meant to be installed using
Python install tools (pip in this case):
.. code-block:: console
pip install -r tests/requirements.txt
For virtualization, either libvirt or VirtualBox is needed (there is native
support from the harness for both). This makes the test harness even more
flexible as most platforms will be covered by either VirtualBox or libvirt.
.. _running_a_scenario:
Running a scenario
------------------
Tests are driven by ``tox``, a command line tool to run a matrix of tests defined in
a configuration file (``tox.ini`` in this case at the root of the project).
For a thorough description of a scenario see :ref:`test_scenarios`.
To run a single scenario, make sure it is available (should be defined from
``tox.ini``) by listing them:
.. code-block:: console
tox -l
In this example, we will use the ``luminous-ansible2.4-xenial_cluster`` one. The
harness defaults to ``VirtualBox`` as the backend, so if you have that
installed in your system then this command should just work:
.. code-block:: console
tox -e luminous-ansible2.4-xenial_cluster
And for libvirt it would be:
.. code-block:: console
tox -e luminous-ansible2.4-xenial_cluster -- --provider=libvirt
.. warning::
Depending on the type of scenario and resources available, running
these tests locally in a personal computer can be very resource intensive.
.. note::
Most test runs take between 20 and 40 minutes depending on system
resources
The command should bring up the machines needed for the test, provision them
with ``ceph-ansible``, run the tests, and tear the whole environment down at the
end.
The output would look something similar to this trimmed version:
.. code-block:: console
luminous-ansible2.4-xenial_cluster create: /Users/alfredo/python/upstream/ceph-ansible/.tox/luminous-ansible2.4-xenial_cluster
luminous-ansible2.4-xenial_cluster installdeps: ansible==2.4.2, -r/Users/alfredo/python/upstream/ceph-ansible/tests/requirements.txt
luminous-ansible2.4-xenial_cluster runtests: commands[0] | vagrant up --no-provision --provider=virtualbox
Bringing machine 'client0' up with 'virtualbox' provider...
Bringing machine 'rgw0' up with 'virtualbox' provider...
Bringing machine 'mds0' up with 'virtualbox' provider...
Bringing machine 'mon0' up with 'virtualbox' provider...
Bringing machine 'mon1' up with 'virtualbox' provider...
Bringing machine 'mon2' up with 'virtualbox' provider...
Bringing machine 'osd0' up with 'virtualbox' provider...
...
After all the nodes are up, ``ceph-ansible`` will provision them, and run the
playbook(s):
.. code-block:: console
...
PLAY RECAP *********************************************************************
client0 : ok=4 changed=0 unreachable=0 failed=0
mds0 : ok=4 changed=0 unreachable=0 failed=0
mon0 : ok=4 changed=0 unreachable=0 failed=0
mon1 : ok=4 changed=0 unreachable=0 failed=0
mon2 : ok=4 changed=0 unreachable=0 failed=0
osd0 : ok=4 changed=0 unreachable=0 failed=0
rgw0 : ok=4 changed=0 unreachable=0 failed=0
...
Once the whole environment is all running the tests will be sent out to the
hosts, with output similar to this:
.. code-block:: console
luminous-ansible2.4-xenial_cluster runtests: commands[4] | testinfra -n 4 --sudo -v --connection=ansible --ansible-inventory=/Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster/hosts /Users/alfredo/python/upstream/ceph-ansible/tests/functional/tests
============================ test session starts ===========================
platform darwin -- Python 2.7.8, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /Users/alfredo/python/upstream/ceph-ansible/.tox/luminous-ansible2.4-xenial_cluster/bin/python
cachedir: ../../../../.cache
rootdir: /Users/alfredo/python/upstream/ceph-ansible/tests, inifile: pytest.ini
plugins: testinfra-1.5.4, xdist-1.15.0
[gw0] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
[gw1] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
[gw2] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
[gw3] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
[gw0] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw3] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35) -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [154] / gw1 [154] / gw2 [154] / gw3 [154]
scheduling tests via LoadScheduling
../../../tests/test_install.py::TestInstall::test_ceph_dir_exists[ansible:/mon0]
../../../tests/test_install.py::TestInstall::test_ceph_dir_is_a_directory[ansible:/mon0]
../../../tests/test_install.py::TestInstall::test_ceph_conf_is_a_file[ansible:/mon0]
../../../tests/test_install.py::TestInstall::test_ceph_dir_is_a_directory[ansible:/mon1]
[gw2] PASSED ../../../tests/test_install.py::TestCephConf::test_ceph_config_has_mon_host_line[ansible:/mon0]
../../../tests/test_install.py::TestInstall::test_ceph_conf_exists[ansible:/mon1]
[gw3] PASSED ../../../tests/test_install.py::TestCephConf::test_mon_host_line_has_correct_value[ansible:/mon0]
../../../tests/test_install.py::TestInstall::test_ceph_conf_is_a_file[ansible:/mon1]
[gw1] PASSED ../../../tests/test_install.py::TestInstall::test_ceph_command_exists[ansible:/mon1]
../../../tests/test_install.py::TestCephConf::test_mon_host_line_has_correct_value[ansible:/mon1]
...
Finally the whole environment gets torn down:
.. code-block:: console
luminous-ansible2.4-xenial_cluster runtests: commands[5] | vagrant destroy --force
==> osd0: Forcing shutdown of VM...
==> osd0: Destroying VM and associated drives...
==> mon2: Forcing shutdown of VM...
==> mon2: Destroying VM and associated drives...
==> mon1: Forcing shutdown of VM...
==> mon1: Destroying VM and associated drives...
==> mon0: Forcing shutdown of VM...
==> mon0: Destroying VM and associated drives...
==> mds0: Forcing shutdown of VM...
==> mds0: Destroying VM and associated drives...
==> rgw0: Forcing shutdown of VM...
==> rgw0: Destroying VM and associated drives...
==> client0: Forcing shutdown of VM...
==> client0: Destroying VM and associated drives...
And a brief summary of the scenario(s) that ran is displayed:
.. code-block:: console
________________________________________________ summary _________________________________________________
luminous-ansible2.4-xenial_cluster: commands succeeded
congratulations :)
docs/source/testing/scenarios.rst
+211
View File
@@ -0,0 +1,211 @@
.. _test_scenarios:
Test Scenarios
==============
Scenarios are distinct environments that describe a Ceph deployment and
configuration. Scenarios are isolated as well, and define what machines are
needed aside from any ``ceph-ansible`` configuration.
.. _scenario_files:
Scenario Files
==============
The scenario is described in a ``vagrant_variables.yml`` file, which is
consumed by ``Vagrant`` when bringing up an environment.
This yaml file is loaded in the ``Vagrantfile`` so that the settings can be
used to bring up the boxes and pass some configuration to ansible when running.
.. note::
The basic layout of a scenario is covered in :ref:`layout`.
There are just a handful of required files, this is the most basic layout.
There are just a handful of required files, these sections will cover the
required (most basic) ones. Alternatively, other ``ceph-ansible`` files can be
added to customize the behavior of a scenario deployment.
.. _vagrant_variables:
``vagrant_variables.yml``
-------------------------
There are a few sections in the ``vagrant_variables.yml`` file which are easy
to follow (most of them are 1 line settings).
* **docker**: (bool) Indicates if the scenario will deploy Docker daemons
* **VMS**: (int) These integer values are just a count of how many machines will be
needed. Each supported type is listed, defaulting to 0:
.. code-block:: yaml
mon_vms: 0
osd_vms: 0
mds_vms: 0
rgw_vms: 0
nfs_vms: 0
rbd_mirror_vms: 0
client_vms: 0
mgr_vms: 0
For a deployment that needs 1 MON and 1 OSD, the list would look like:
.. code-block:: yaml
mon_vms: 1
osd_vms: 1
* **CEPH SOURCE**: (string) indicate whether a ``dev`` or ``stable`` release is
needed. A ``stable`` release will use the latest stable release of Ceph,
a ``dev`` will use ``shaman`` (http://shaman.ceph.com)
* **SUBNETS**: These are used for configuring the network availability of each
server that will be booted as well as being used as configuration for
``ceph-ansible`` (and eventually Ceph). The two values that are **required**:
.. code-block:: yaml
public_subnet: 192.168.13
cluster_subnet: 192.168.14
* **MEMORY**: Memory requirements (in megabytes) for each server, e.g.
``memory: 512``
* **interfaces**: some vagrant boxes (and linux distros) set specific
interfaces. For Ubuntu releases older than Xenial it was common to have
``eth1``, for CentOS and some Xenial boxes ``enp0s8`` is used. **However**
the public Vagrant boxes normalize the interface to ``eth1`` for all boxes,
making it easier to configure them with Ansible later.
.. warning::
Do *not* change the interface from ``eth1`` unless absolutely
certain that is needed for a box. Some tests that depend on that
naming will fail.
* **disks**: The disks that will be created for each machine, for most
environments ``/dev/sd*`` style of disks will work, like: ``[ '/dev/sda', '/dev/sdb' ]``
* **vagrant_box**: We have published our own boxes to normalize what we test
against. These boxes are published in Atlas
(https://atlas.hashicorp.com/ceph/). Currently valid values are:
``ceph/ubuntu-xenial``, and ``ceph/centos7``
The following aren't usually changed/enabled for tests, since they don't have
an impact, however they are documented here for general knowledge in case they
are needed:
* **ssh_private_key_path**: The path to the ``id_rsa`` (or other private SSH
key) that should be used to connect to these boxes.
* **vagrant_sync_dir**: what should be "synced" (made available on the new
servers) from the host.
* **vagrant_disable_synced_folder**: (bool) when disabled, it will make
booting machines faster because no files need to be synced over.
* **os_tuning_params**: These are passed onto ``ceph-ansible`` as part of the
variables for "system tunning". These shouldn't be changed.
.. _vagrant_file:
``Vagrantfile``
---------------
The ``Vagrantfile`` should not need to change, and it is symlinked back to the
``Vagrantfile`` that exists in the root of the project. It is linked in this
way so that a vagrant environment can be isolated to the given scenario.
.. _hosts_file:
``hosts``
---------
The ``hosts`` file should contain the hosts needed for the scenario. This might
seem a bit repetitive since machines are already defined in
:ref:`vagrant_variables` but it allows granular changes to hosts (for example
defining different public_network values between monitors) which can help catch issues in
``ceph-ansible`` configuration. For example:
.. code-block:: ini
[mons]
mon0 public_network=192.168.1.0/24
mon1 public_network=192.168.2.0/24
mon2 public_network=192.168.3.0/24
.. _group_vars:
``group_vars``
--------------
This directory holds any configuration change that will affect ``ceph-ansible``
deployments in the same way as if ansible was executed from the root of the
project.
The file that will need to be defined always is ``all`` where (again) certain
values like ``public_network`` and ``cluster_network`` will need to be defined
along with any customizations that ``ceph-ansible`` supports.
.. _scenario_wiring:
Scenario Wiring
---------------
Scenarios are just meant to provide the Ceph environment for testing, but they
do need to be defined in the ``tox.ini`` so that they are available to the test
framework. To see a list of available scenarios, the following command (ran
from the root of the project) will list them, shortened for brevity:
.. code-block:: console
$ tox -l
...
luminous-ansible2.4-centos7_cluster
...
These scenarios are made from different variables, in the above command there
are 3:
* ``jewel``: the Ceph version to test
* ``ansible2.4``: the Ansible version to install
* ``centos7_cluster``: the name of the scenario
The last one is important in the *wiring up* of the scenario. It is a variable
that will define in what path the scenario lives. For example, the
``changedir`` section for ``centos7_cluster`` that looks like:
.. code-block:: ini
centos7_cluster: {toxinidir}/tests/functional/centos/7/cluster
The actual tests are written for specific daemon types, for all daemon types,
and for specific use cases (e.g. journal collocation), those have their own
conventions as well which are explained in detail in :ref:`test_conventions`
and :ref:`test_files`.
As long as a test scenario defines OSDs and MONs, the OSD tests and MON tests
will run.
.. _scenario_conventions:
Conventions
-----------
.. _scenario_environment_configuration:
Environment configuration
-------------------------
.. _scenario_ansible_configuration:
Ansible configuration
---------------------
docs/source/testing/tests.rst
+99
View File
@@ -0,0 +1,99 @@
.. _tests:
Tests
=====
Actual tests are written in Python methods that accept optional fixtures. These
fixtures come with interesting attributes to help with remote assertions.
As described in :ref:`test_conventions`, tests need to go into
``tests/functional/tests/``. These are collected and *mapped* to a distinct
node type, or *mapped* to run on all nodes.
Simple Python asserts are used (these tests do not need to follow the Python
``unittest.TestCase`` base class) that make it easier to reason about failures
and errors.
The test run is handled by ``py.test`` along with :ref:`testinfra` for handling
remote execution.
.. _test_files:
Test Files
----------
.. _test_fixtures:
Test Fixtures
=============
Test fixtures are a powerful feature of ``py.test`` and most tests depend on
this for making assertions about remote nodes. To request them in a test
method, all that is needed is to require it as an argument.
Fixtures are detected by name, so as long as the argument being used has the
same name, the fixture will be passed in (see `pytest fixtures`_ for more
in-depth examples). The code that follows shows a test method that will use the
``node`` fixture that contains useful information about a node in a ceph
cluster:
.. code-block:: python
def test_ceph_conf(self, node):
assert node['conf_path'] == "/etc/ceph/ceph.conf"
The test is naive (the configuration path might not exist remotely) but
explains how simple it is to "request" a fixture.
For remote execution, we can rely further on other fixtures (tests can have as
many fixtures as needed) like ``File``:
.. code-block:: python
def test_ceph_config_has_inital_members_line(self, node, File):
assert File(node["conf_path"]).contains("^mon initial members = .*$")
.. _node:
``node`` fixture
----------------
The ``node`` fixture contains a few useful pieces of information about the node
where the test is being executed, this is captured once, before tests run:
* ``address``: The IP for the ``eth1`` interface
* ``subnet``: The subnet that ``address`` belongs to
* ``vars``: all the Ansible vars set for the current run
* ``osd_ids``: a list of all the OSD IDs
* ``num_mons``: the total number of monitors for the current environment
* ``num_devices``: the number of devices for the current node
* ``num_osd_hosts``: the total number of OSD hosts
* ``total_osds``: total number of OSDs on the current node
* ``cluster_name``: the name of the Ceph cluster (which defaults to 'ceph')
* ``conf_path``: since the cluster name can change the file path for the Ceph
configuration, this gets sets according to the cluster name.
* ``cluster_address``: the address used for cluster communication. All
environments are set up with 2 interfaces, 1 being used exclusively for the
cluster
* ``docker``: A boolean that identifies a Ceph Docker cluster
* ``osds``: A list of OSD IDs, unless it is a Docker cluster, where it gets the
name of the devices (e.g. ``sda1``)
Other Fixtures
--------------
There are a lot of other fixtures provided by :ref:`testinfra` as well as
``py.test``. The full list of ``testinfra`` fixtures are available in
`testinfra_fixtures`_
``py.test`` builtin fixtures can be listed with ``pytest -q --fixtures`` and
they are described in `pytest builtin fixtures`_
.. _pytest fixtures: https://docs.pytest.org/en/latest/fixture.html
.. _pytest builtin fixtures: https://docs.pytest.org/en/latest/builtin.html#builtin-fixtures-function-arguments
.. _testinfra_fixtures: https://testinfra.readthedocs.io/en/latest/modules.html#modules
docs/source/testing/tox.rst
+75
View File
@@ -0,0 +1,75 @@
.. _tox:
``tox``
=======
``tox`` is an automation project we use to run our testing scenarios. It gives us
the ability to create a dynamic matrix of many testing scenarios, isolated testing environments
and a provides a single entry point to run all tests in an automated and repeatable fashion.
Documentation for tox can be found `here <https://tox.readthedocs.io/en/latest/>`_.
.. _tox_environment_variables:
Environment variables
---------------------
When running ``tox`` we've allowed for the usage of environment variables to tweak certain settings
of the playbook run using Ansible's ``--extra-vars``. It's helpful in Jenkins jobs or for manual test
runs of ``ceph-ansible``.
The following environent variables are available for use:
* ``CEPH_DOCKER_REGISTRY``: (default: ``quay.io``) This would configure the ``ceph-ansible`` variable ``ceph_docker_registry``.
* ``CEPH_DOCKER_IMAGE``: (default: ``ceph/daemon``) This would configure the ``ceph-ansible`` variable ``ceph_docker_image``.
* ``CEPH_DOCKER_IMAGE_TAG``: (default: ``latest``) This would configure the ``ceph-ansible`` variable ``ceph_docker_image_name``.
* ``CEPH_DEV_BRANCH``: (default: ``main``) This would configure the ``ceph-ansible`` variable ``ceph_dev_branch`` which defines which branch we'd
like to install from shaman.ceph.com.
* ``CEPH_DEV_SHA1``: (default: ``latest``) This would configure the ``ceph-ansible`` variable ``ceph_dev_sha1`` which defines which sha1 we'd like
to install from shaman.ceph.com.
* ``UPDATE_CEPH_DEV_BRANCH``: (default: ``main``) This would configure the ``ceph-ansible`` variable ``ceph_dev_branch`` which defines which branch we'd
like to update to from shaman.ceph.com.
* ``UPDATE_CEPH_DEV_SHA1``: (default: ``latest``) This would configure the ``ceph-ansible`` variable ``ceph_dev_sha1`` which defines which sha1 we'd like
to update to from shaman.ceph.com.
.. _tox_sections:
Sections
--------
The ``tox.ini`` file has a number of top level sections defined by ``[ ]`` and subsections within those. For complete documentation
on all subsections inside of a tox section please refer to the tox documentation.
* ``tox`` : This section contains the ``envlist`` which is used to create our dynamic matrix. Refer to the `section here <http://tox.readthedocs.io/en/latest/config.html#generating-environments-conditional-settings>`_ for more information on how the ``envlist`` works.
* ``purge`` : This section contains commands that only run for scenarios that purge the cluster and redeploy. You'll see this section being reused in ``testenv``
with the following syntax: ``{[purge]commands}``
* ``update`` : This section contains commands taht only run for scenarios that deploy a cluster and then upgrade it to another Ceph version.
* ``testenv`` : This is the main section of the ``tox.ini`` file and is run on every scenario. This section contains many *factors* that define conditional
settings depending on the scenarios defined in the ``envlist``. For example, the factor ``centos7_cluster`` in the ``changedir`` subsection of ``testenv`` sets
the directory that tox will change do when that factor is selected. This is an important behavior that allows us to use the same ``tox.ini`` and reuse commands while
tweaking certain sections per testing scenario.
.. _tox_environments:
Modifying or Adding environments
--------------------------------
The tox environments are controlled by the ``envlist`` subsection of the ``[tox]`` section. Anything inside of ``{}`` is considered a *factor* and will be included
in the dynamic matrix that tox creates. Inside of ``{}`` you can include a comma separated list of the *factors*. Do not use a hyphen (``-``) as part
of the *factor* name as those are used by tox as the separator between different factor sets.
For example, if wanted to add a new test *factor* for the next Ceph release of luminious this is how you'd accomplish that. Currently, the first factor set in our ``envlist``
is used to define the Ceph release (``{jewel,kraken}-...``). To add luminous you'd change that to look like ``{luminous,kraken}-...``. In the ``testenv`` section
this is a subsection called ``setenv`` which allows you to provide environment variables to the tox environment and we support an environment variable called ``CEPH_STABLE_RELEASE``. To ensure that all the new tests that are created by adding the luminous *factor* you'd do this in that section: ``luminous: CEPH_STABLE_RELEASE=luminous``.