Use systemd to manage the containerized services

The latest version of RHOSP uses systemd units to manage the lifecycle of service containers. systemd handles common operations such as starting, stopping, and others, managing containers in the same way as other systemd units and services, using the podman command to interact with the containers.

For example:

To verify the status of a containerized service, run the systemctl status command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@controller0 ~]# systemctl status tripleo_cinder_api
● tripleo_cinder_api.service - cinder_api container
   Loaded: loaded (/etc/systemd/system/tripleo_cinder_api.service; enabled; vendor preset: di>
   Active: active (running) since Tue 2025-04-29 02:27:54 UTC; 6h ago
 Main PID: 4963 (conmon)
    Tasks: 0 (limit: 101103)
   Memory: 1.7M
   CGroup: /system.slice/tripleo_cinder_api.service
           ‣ 4963 /usr/bin/conmon --api-version 1 -s -c a89efc04180c85f9e5a2a0f6a7d1835b543b9>

Apr 29 02:27:46 controller0 systemd[1]: Starting cinder_api container...
Apr 29 02:27:54 controller0 podman[3564]: 2025-04-29 02:27:54.043315718 +0000 UTC m=+6.779304>
Apr 29 02:27:54 controller0 podman[3564]: 2025-04-29 02:27:54.375331615 +0000 UTC m=+7.111320>
Apr 29 02:27:54 controller0 podman[3564]: cinder_api
Apr 29 02:27:54 controller0 systemd[1]: Started cinder_api container.

Timer

Systemd monitors container health checks, which are not displayed by podman ps, using systemd timers. To list containers timers, use the systemctl list-timers command. This example filters to view only tripleo unit services.

1
2
3
4
5
6
7
8
9
10
11
[root@controller0 ~]# systemctl list-timers | grep tripleo
Tue 2025-04-29 10:34:36 UTC  34ms left     Tue 2025-04-29 10:33:27 UTC  1min 9s ago  tripleo_swift_rsync_healthcheck.timer                tripleo_swift_rsync_healthcheck.service
Tue 2025-04-29 10:34:36 UTC  365ms left    Tue 2025-04-29 10:33:22 UTC  1min 14s ago tripleo_heat_engine_healthcheck.timer                tripleo_heat_engine_healthcheck.service
Tue 2025-04-29 10:34:39 UTC  2s left       Tue 2025-04-29 10:33:12 UTC  1min 24s ago tripleo_octavia_worker_healthcheck.timer             tripleo_octavia_worker_healthcheck.service
Tue 2025-04-29 10:34:40 UTC  4s left       Tue 2025-04-29 10:32:57 UTC  1min 39s ago tripleo_manila_scheduler_healthcheck.timer           tripleo_manila_scheduler_healthcheck.service
Tue 2025-04-29 10:34:43 UTC  7s left       Tue 2025-04-29 10:33:12 UTC  1min 24s ago tripleo_cinder_scheduler_healthcheck.timer           tripleo_cinder_scheduler_healthcheck.service
Tue 2025-04-29 10:34:44 UTC  8s left       Tue 2025-04-29 10:33:17 UTC  1min 19s ago tripleo_neutron_api_healthcheck.timer                tripleo_neutron_api_healthcheck.service
Tue 2025-04-29 10:34:51 UTC  14s left      Tue 2025-04-29 10:33:12 UTC  1min 24s ago tripleo_clustercheck_healthcheck.timer               tripleo_clustercheck_healthcheck.service
Tue 2025-04-29 10:34:55 UTC  19s left      Tue 2025-04-29 10:33:27 UTC  1min 9s ago  tripleo_logrotate_crond_healthcheck.timer            tripleo_logrotate_crond_healthcheck.service
Tue 2025-04-29 10:35:00 UTC  24s left      Tue 2025-04-29 10:33:27 UTC  1min 9s ago  tripleo_swift_object_replicator_healthcheck.timer    tripleo_swift_object_replicator_healthcheck.service
...output omitted...

To verify a specific container timer, use the systemctl status command.

1
2
3
4
5
6
7
[root@controller0 ~]# systemctl status tripleo_cinder_api_healthcheck.timer
● tripleo_cinder_api_healthcheck.timer - cinder_api container healthcheck
   Loaded: loaded (/etc/systemd/system/tripleo_cinder_api_healthcheck.timer; enabled; vendor >
   Active: active (waiting) since Tue 2025-04-29 02:26:51 UTC; 8h ago
  Trigger: Tue 2025-04-29 10:53:36 UTC; 13s left

Apr 29 02:26:51 controller0 systemd[1]: Started cinder_api container healthcheck.

Let’s check the configuration of tripleo_cinder_api_healthcheck.timer!

1
2
3
4
5
6
7
8
9
10
11
[root@controller0 ~]# systemctl cat tripleo_cinder_api_healthcheck.timer
# /etc/systemd/system/tripleo_cinder_api_healthcheck.timer
[Unit]
Description=cinder_api container healthcheck
PartOf=tripleo_cinder_api.service
[Timer]
OnActiveSec=120
OnUnitActiveSec=60
RandomizedDelaySec=45.0
[Install]
WantedBy=timers.target
  • [Unit] section:

Description: Describes the purpose of the timer.

PartOf=tripleo_cinder_api.service only affects the lifecycle of the timer and does not change the service triggered by the timer.

By default, the timer will trigger the service file with the same name, i.e., tripleo_cinder_api_healthcheck.service. This is not controlled by PartOf, but rather by the naming convention of systemd.

If you want the timer to trigger a different service (such as tripleo_cinder_api.service), you need to explicitly specify the desired service in the Unit section or ExecStart of the timer file.

  • [Timer] section:

OnActiveSec=120: The timer will start 120 seconds after the service is activated.

OnUnitActiveSec=60: The timer will start again 60 seconds after the last active period of the service.

RandomizedDelaySec=45.0: A random delay of up to 45 seconds will be added before each timer start.

  • [Install] section:

WantedBy=timers.target: Specifies that this timer will start when timers.target is triggered.

To verify, you can repeatedly check the status systemctl status tripleo_cinder_api_healthcheck. After about 60 seconds, you will see the service start once again.

Log

In the latest version of RHOSP, the standard output (stdout) and standard errors (stderr) are consolidated in a single file per container, located in the /var/log/containers/stdouts directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@controller0 ~]# ls -1 /var/log/containers/stdouts
aodh_api.log
aodh_db_sync.log
aodh_evaluator.log
aodh_init_log.log
aodh_listener.log
aodh_notifier.log
ceilometer_agent_central.log
ceilometer_agent_notification.log
ceilometer_gnocchi_upgrade.log
ceilometer_init_log.log
cinder_api_cron.log
cinder_api_cron.log.1
cinder_api_cron.log.10.gz
cinder_api_cron.log.11.gz
cinder_api_cron.log.12.gz
cinder_api_cron.log.13.gz
cinder_api_cron.log.14.gz
cinder_api_cron.log.2.gz
cinder_api_cron.log.3.gz
cinder_api_cron.log.4.gz
cinder_api_cron.log.5.gz
cinder_api_cron.log.6.gz
...output omitted...

Configuration

In RHOSP versions with containerized services, container configuration files are located in the /var/lib/config-data/puppet-generated/container_name directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@controller0 ~]# ls -1 /var/lib/config-data/puppet-generated/ | more
aodh
aodh.md5sum
ceilometer
ceilometer.md5sum
cinder
cinder.md5sum
clustercheck
clustercheck.md5sum
crond
crond.md5sum
glance_api
glance_api.md5sum
gnocchi
gnocchi.md5sum
haproxy
haproxy.md5sum
heat
heat_api
heat_api_cfn
heat_api_cfn.md5sum
heat_api.md5sum
heat.md5sum
horizon
horizon.md5sum
...output omitted...

The systemd unit files of containerized services are named with a tripleo_ prefix, because they are installed by TripleO.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@controller0 ~]# ls -1 /etc/systemd/system/tripleo* | more
/etc/systemd/system/tripleo_cinder_api_cron_healthcheck.service
/etc/systemd/system/tripleo_cinder_api_cron_healthcheck.timer
/etc/systemd/system/tripleo_cinder_api_cron.service
/etc/systemd/system/tripleo_cinder_api_healthcheck.service
/etc/systemd/system/tripleo_cinder_api_healthcheck.timer
/etc/systemd/system/tripleo_cinder_api.service
/etc/systemd/system/tripleo_cinder_scheduler_healthcheck.service
/etc/systemd/system/tripleo_cinder_scheduler_healthcheck.timer
/etc/systemd/system/tripleo_cinder_scheduler.service
/etc/systemd/system/tripleo_clustercheck_healthcheck.service
/etc/systemd/system/tripleo_clustercheck_healthcheck.timer
/etc/systemd/system/tripleo_clustercheck.service
...output omitted...