A Patch Train Solution for OCI OS Management with OCI Ansible Modules

 

With the OS Management Service (OSMS) and i-ansible-collection, Oracle Cloud Infrastructure provides a service to fully automate the patching of your Oracle Linux or Windows instances. It allows you to organize your systems into groups and then schedule jobs to apply the latest updates to all these systems. There is a wide range of predefined software sources to choose from, providing the full wealth of the Oracle yum repositories to your Linux systems. In the simplest case, this will keep all your systems up to date with the latest and consistent  patches all the time using Ansible modules.

In many cases, this eliminates the burden of constantly chasing security updates and keeping your fleet patched. However, sometimes your patching goals are more complex than just having the "latest and greatest" all the time. In this article, I'll describe one such scenario and a solution.

Patch trains.

The patch train idea can be easily described: Start with defining consistent patch bundles or in oci term custom software source, i.e. a set of exact packages including their version. Then apply the patch bundle to a testing environment and test it. If test shows good results, promote patch to next environment. And you do this in your testing environments until you are ready for applying the bundle to the production environment.

 

Enabling OS Management

OS Management will not out of the box in your tenancy. You first need to create a dynamic group and set some policies and configure accessing-the-oracle-cloud-infrastructure-api-using-instance-principals. This requires some administration rights on tenancy level.

Example: These policies are created in Root level to allow all Linux instance across the compartment that need a patching . If you think that the policies are more exposed then you can play with some more restricted rules.

 Allow dynamic-group linux-patching to manage osms-managed-instances in tenancy

Allow dynamic-group linux-patching to manage osms-managed-instance-groups in tenancy

Allow dynamic-group linux-patching to manage osms-software-sources in tenancy

Allow dynamic-group linux-patching to manage osms-errata in tenancy

Allow dynamic-group linux-patching to manage osms-scheduled-jobs in tenancy

Allow dynamic-group linux-patching to manage osms-work-requests in tenancy

Allow dynamic-group linux-patching to use instance-family in tenancy

Allow dynamic-group linux-patching to read instance-family in tenancy

Allow service osms to read instances in tenancy

For Automation hosts running Ansible task defined below needed below privilege.

Allow dynamic-group automation_servers to manage instance-family in compartment OS_Images

Allow dynamic-group automation_servers to read app-catalog-listing in tenancy

Allow dynamic-group automation_servers to use volume-family in compartment OS_Images

Allow dynamic-group automation_servers  to manage volume-family in tenancy

Allow dynamic-group automation_servers to use virtual-network-family in compartment OS_Images

Allow dynamic-group automation_servers to use virtual-network-family in compartment Networks

Allow dynamic-group automation_servers to read instance-images in tenancy

Allow dynamic-group automation_servers to read instance in tenancy

Allow dynamic-group automation_servers to manage volume-backups in tenancy

Allow dynamic-group automation_servers to manage osms-managed-instances in tenancy

Allow dynamic-group automation_servers to manage osms-managed-instance-groups in tenancy

Allow dynamic-group automation_servers to manage osms-software-sources in tenancy

Allow dynamic-group automation_servers to read osms-software-sources in tenancy

Allow dynamic-group automation_servers to manage osms-errata in tenancy

Allow dynamic-group automation_servers to manage osms-scheduled-jobs in tenancy

Allow dynamic-group automation_servers to manage osms-work-requests in tenancy

Allow dynamic-group automation_servers to manage instance-family in tenancy

Allow dynamic-group automation_servers to manage buckets in tenancy

Allow dynamic-group automation_servers to manage objects in tenancy

Allow dynamic-group automation_servers to read virtual-network-family in tenancy

Allow dynamic-group automation_servers to inspect compartments in tenancy

 

Note:  Indentation might  misplaces during pasting.

Ansible Collection block must need to call Ansible modules, see how to configure getting start and github

---

- name :Oracle Ansible collection

  collections:

    - oracle.oci

    - oracle.oci.oci_compute_dedicated_vm_host_facts

  connection: local

  hosts: localhost

 

Custom Parent Software Source

First thing we need to prepare is an  empty custom parent software source. This will later have the custom software sources attached that can be applied to specific managed instance groups.

 

- name: Create custom parent software_source

  oci_os_management_software_source:

    compartment_id: "{{ compartment_id }}"

    display_name: "{{ display_name }}"

    region: "{{  region | default('us-phoenix-1') }}"

    description: This shoftware source will use for all patching in OL7 release

    arch_type: X86_64

    state: present

  register: custom_parent_ss

  tags:

    - ss

 

Can to be done via CLI as well

oci os-management software-source create  -c ocid1.tenancy.oc1..bbbbbaatdylvcgv2r6xxplnkemowv123455nl4d2t6kpbp4s6ygq --display-name latest_july_2021 --arch-type X86_64

 

 

 

Creating Managed Instance Groups

I recommend to create a special managed instance group for reference/template instances. These instances must keep on using the standard software sources.

Create a managed instance group for each stage you plan to include in your patching process.

 

 

- name: Create managed_instance_group

  oci_os_management_managed_instance_group:

    display_name: linux_patching

    region: "{{  region | default('us-phoenix-1') }}"

    compartment_id: "{{ compartment_id }}"

  register: mns

 

- name: Dump managed_instance_group name

  debug:

    msg: '{{ mns.managed_instance_group.display_name }}'

 

 

CLI

oci os-management managed-instance-group create -c ocid1.tenancy.oc1..bbbbbaatdylvcgv2r6xxplnkemowv123455nl4d2t6kpbp4s6ygq --display-name test_patch

 

Adding Managed Instances to Managed Instance Groups

All we got so far are empty managed instance groups. Now it is time to add some Oracle Linux managed instances to the managed instance groups using  dynamic inventory file Working with Ansible Inventory

 

Example inventory.oci.yaml, I am searching for all Linux host with their OCID numbers those have batch1 free-form tags.

-------------------------------------------------------------------------------------------------------------------------------------

plugin: oracle.oci.oci

auth_type: instance_principal

regions:

  - us-ashburn-1

  - us-phoenix-1

debug: false

enable_parallel_processing: yes

compartments:

  - compartment_ocid: ocid1.tenancy.oc1..bbbbbaatdylvcgv2r6xxplnkemowv123455nl4d2t6kpbp4s6ygq

    fetch_hosts_from_subcompartments: true

hostname_format_preferences:

  #- "display_name+'.oci.com'"

  - "id"

include_host_filters:

  #- "'batch1' in freeform_tags.os_patching and 'RUNNING' in lifecycle_state"

  #- "'batch1' in freeform_tags.os_patching and 'RUNNING' in lifecycle_state"

  - "'batch1' in freeform_tags.os_patching and 'RUNNING' in lifecycle_state"

--------------------------------------------------------------------------------------------------

 

 

- name: Perform action attach_managed_instance on managed_instance_group

  oci_os_management_managed_instance_group_actions:

    managed_instance_group_id: '{{ mns.managed_instance_group.id }}'

    region: "{{  region | default('us-phoenix-1') }}"

    action: attach_managed_instance

    managed_instance_id: '{{item}}'

  with_items: '{{inventory_hostname}}'

 

 

List and then detach the standard parent software source

- name: List available_software_sources in instances

  oci_os_management_available_software_source_facts:

    managed_instance_id: '{{item}}'

    region: "{{  region | default('us-phoenix-1') }}"

  with_items:

    - '{{inventory_hostname}}'

  register: parent_software_id

 

 

 

- name: Perform action detach_parent_software_source on managed_instance

  oci_os_management_managed_instance_actions:

    software_source_id: "{{parent_software_id.results|community.general.json_query('[*].available_software_sources[*].parent_id') |flatten| min}}"

    action: detach_parent_software_source

    region: "{{  region | default('us-phoenix-1') }}"

    managed_instance_id: '{{item}}'

  with_items:

    - '{{inventory_hostname}}'

  when: "parent_software_id.results|community.general.json_query('[*].available_software_sources[*].parent_id') |flatten| length  != 0"

 

 

Also need to detach any custom software source if attached

- name: Get a specific managed_instance information for custom software source

  oci_os_management_managed_instance_facts:

    managed_instance_id: '{{item}}'

  with_items:

    - '{{inventory_hostname}}'

  register: attached_ss_id

- set_fact:

    ss: "{{attached_ss_id.results|community.general.json_query('[].managed_instances[].parent_software_source.id')}}"

 

- name: Perform action detach attached custom parent software_source on managed_instance

  oci_os_management_managed_instance_actions:

    software_source_id: "{{attached_ss_id.results|community.general.json_query('[].managed_instances[].parent_software_source.id')| first }}"

    action: detach_parent_software_source

    region: "{{  region | default('us-phoenix-1') }}"

    managed_instance_id: '{{item}}'

  with_items:

    - '{{inventory_hostname}}'

  when: ' "ss|length" != 0'

 

 

Attach the empty custom parent software source that was created before. Pass software source ID through vars are can be passed directly.

 

  - name: Perform action attach custom parent software_source on managed_instance

    oci_os_management_managed_instance_actions:

      software_source_id: '{{ software_source_phx_id  }}'

      action: attach_parent_software_source

      region: "{{  region | default('us-phoenix-1') }}"

      managed_instance_id: '{{item}}'

    with_items:

      - '{{inventory_hostname}}'

 

Extracting a packages and their version from default software source.

As all the instances in oci has their own list of packages hence generating a list of packages from sample instance will not practical for other instance and can skip many packages to update or patch. To avoid that situations and make the patching consistence we need a stable and robust solutions that can be applied in both stage and production environment. Hence I am extracting a list of packages and put in a file, later that file will be use to adding a packages in the empty custom software source.

 

- name: List software_sources

  oci_os_management_software_source_facts:

    compartment_id: "{{ compartment_id }}"

    lifecycle_state: ACTIVE

  register: software_list

- set_fact:

    date: "{{ lookup('pipe','date +%Y-%m-%d') }}"

 

Here I am looking for OL7 default latest software source list/

 

- name: Dump latest OL7 software source id list

  set_fact:

    test_software_source_id: '{{ item.parent_id }}'

  when: 'item.parent_name == "Oracle Linux 7Server Latest (x86_64)"'

  loop: "{{ software_list.software_sources }}"

 

- name: version of all available packages from OL7 latest source source

  oci_os_management_software_package_facts:

    software_source_id: '{{ test_software_source_id }}'

  register: software_list

 

Then copy an all packages to a local file that can be used added in custom software source

 

- name: copy packages name to a local file

  copy:

    content: "{{ software_list.software_packages | json_query('[*].name') }}"

    dest: "/u01/ansible/patching/group_vars/package_list"

  delegate_to: localhost

 

 

Adding the Packages to the Custom parent Software Source created before.

Now I am calling packages list of files created above. As the list items are in huge size and could be generate long API calls hence here I am adding that package in chunk of 500 in custom software source. Tag ss is used to just run required functions not everything to add the package in other region of software source.

 

- set_fact:

    list: '{{item}}'

  with_file: "/u01/idmauto/ansible/patching/group_vars/package_list"

  tags:

    - ss

 

- name: add packages to custom client software_source

  oci_os_management_software_source_actions:

    software_source_id: '{{custom_parent_ss.software_source.id}}'

      region: "{{  region | default('us-phoenix-1') }}"

      action: add_packages

      package_names: "{{list[(item|int):(item|int)+500]}}"

  with_sequence: start=0 end="{{list | length}}" stride=500

  tags:

    - ss

 

Installing Packages and Updates

Updating a managed instance group can be done by scheduling a job. This job then is executed once or on a regular schedule, for the current description i will focus on onetime update jobs only.

- set_fact:

    date: "{{ lookup('pipe','date +%Y-%m-%d') }}"

- name: shedule a patch job

  oci_os_management_scheduled_job:

    compartment_id: "{{ compartment_id }}"

    region: "{{  region | default('us-phoenix-1') }}"

    managed_instance_groups:

      - id: '{{ mns.managed_instance_group.id }}'

    display_name: '{{ mns.managed_instance_group.display_name }}'

    display_name: "patch_scheduler-{{ date }}"

    schedule_type: ONETIME

    time_next_execution: "2021-09-09T09:30:00+00:00"

    operation_type: UPDATEALL

    update_type: ALL

  register: schedule_job