Tutorial - Custom Combiner Development¶
In the Map-Reduce model, Parsers are responsible for mapping the data from a datasource and Combiners are responsible for reducing the data from multiple Parsers into a reduced dataset. Combiners help to consolidate information from different data sources, hide differences between the same data source across operating system versions, and make Parser output more rule friendly.
For example, the hostname of a system may be obtained from the
SystemID parsers. A rule could rely on all three parsers
and might need to process each one differently depending upon which was present.
However, a rule could instead simply rely on the
hostname combiner which
does the job of evaluating all of the parser data and determining the best
source of hostname, greatly simplifying logic in the rule.
Another example is the
GrubConf combiner which evaluates multiple versions
(1, 2, non-EFI, EFI) Grub configuration data to provide one consolidated source
of information for Grub configuration on a system.
You can find the complete implementation of the combiner and test code in the
The same development environment will be used that was setup at the beginning of the tutorial using the Preparing Your Development Environment section.
For this tutorial we will create a new hostname combiner that will consolidate
information from the
insights.parsers.hostname.Hostname parsers. There is an existing
insights.combiners.hostname.Hostname combiner so we will call ours
HostnameUH to avoid confusion.
Creating the Initial Combiner Files¶
First we need to create the combiner file. Combiner files are implemented in
modules. The module should be limited to one purpose. In this case we are
hostname data so we will create an
Also there is already a
hostname combiner module so we want to avoid
confusion. Create the module file
mycomponents/combiners/hostname_uh.py in the
(env)[userone@hostone mycomponents]$ touch combiners/hostname_uh.py
Now edit the file and create the combiner skeleton:
1 2 3 4 5 6 7 8 9 10
from insights.core.plugins import combiner from insights.parsers.hostname import Hostname from insights.parsers.uname import Uname @combiner([Hostname, Uname]) class HostnameUH(object): def __init__(self, hostname, uname): pass
We start by importing the
combiner decorator. As discussed above our
combiner will depend upon the
Uname parsers and these
are imported and included as arguments to the
combiner decorator. Notice
that the decorator arguments are in a
list meaning that our combiner
needs at least one of the parsers in the list. See the discussion of
Rule Decorators for more information on required,
at least one, and optional arguments to the combiner decorator.
We also need to pass the parser instance objects as arguments to the
method of our combiner. If either of these objects is not present then its
value with be
Next we’ll create the combiner test file
as a skeleton that will aid in the combiner development process:
1 2 3 4 5
from mycomponents.combiners.hostname_uh import HostnameUH def test_hostname_uh(): pass
Once you have created and saved both of these files, you can the test to make sure everything is setup correctly:
(env)[userone@hostone insights-core-tutorials]$ pytest -k hostname_uh ======================= test session starts ============================== platform linux -- Python 3.6.6, pytest-3.0.6, py-1.7.0, pluggy-0.4.0 rootdir: /home/userone/work/insights-core-tutorials, inifile: collected 16 items / 14 deselected insights_examples/combiners/tests/test_hostname_uh.py . [ 50%] mycomponents/combiners/tests/test_hostname_uh.py . ============ 2 passed, 14 deselected in .27 seconds ====================
When you invoke
pytest with the
-k option it will only run tests
which match the filter, in this case tests that match hostname_uh. So our
test passed as expected.
You may sometimes see a message that
pytest cannot be found,
or see some other related message that doesn’t make sense. The first
think to check is that you have activated your virtual environment by
executing the command
source bin/activate from the root directory
of your insights-core-tutorials project. You can deactivate the virtual
environment by typing
deactivate. You can find more information about
virtual environments here:
Typically parser and combiner development is driven by rules that need facts generated by the parsers and combiners. Regardless of the specific requirements, it is important (1) to implement basic functionality by getting the raw data into a usable format, and (2) to not overdo the implementation because we can’t anticipate every use of the combiner output. In our example the output is simple, but some combiners can be complicated so keep these two criteria in mind when developing new parsers or combiners. You can always add more capability later on if needed by your rules.
We will start by creating a test for the output that we want from our combiner
using the two input sources. You can look at the documentation for
insights.parsers.uname to see
what methods will be available. In our tests we want to ensure that we can
test with the parser object so we’ll use input data to feed the parsers and
then use the parsers as input to our combiner tests.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
from mycomponents.combiners.hostname_uh import HostnameUH from insights.parsers.hostname import Hostname from insights.parsers.uname import Uname from insights.tests import context_wrap HOSTNAME = "hostone_h.example.com" UNAME = "Linux hostone_u.example.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux" def test_hostname_uh(): hostname = Hostname(context_wrap(HOSTNAME)) uname = Uname(context_wrap(UNAME)) hostname_uh = HostnameUH(hostname, None) assert hostname_uh.hostname == HOSTNAME hostname_uh = HostnameUH(None, uname) assert hostname_uh.hostname == "hostone_u.example.com" hostname_uh = HostnameUH(hostname, uname) assert hostname_uh.hostname == HOSTNAME
First we added an import for the combiner object and the parser objects. Next
we import a helper function
context_wrap which we’ll
use to create our parser instance objects:
1 2 3 4
from insights.combiners.hostname_uh import HostnameUH from insights.parsers.hostname import Hostname from insights.parsers.uname import Uname from insights.tests import context_wrap
Next we include the sample data that will be used for the test. We will use
data for input to the parsers so we need both sample outputs of the
command and the
uname -a command:
HOSTNAME = "hostone_h.example.com" UNAME = "Linux hostone_u.example.com 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux"
Next, to the body of the test, we add code to create instances of the necessary parser classes:
10 11 12
def test_hostname_uh(): hostname = Hostname(context_wrap(HOSTNAME)) uname = Uname(context_wrap(UNAME))
Finally we add our tests using the attributes that we want to be able to
access in our rules. For our combiner we trust
hostname more than
uname so we give
hostname priority by checking it first and then
fall back to
uname if hostname is not available. If neither of these is
available the combiner will not be called. It is always guaranteed that our
combiner will get at least one of the parsers when called.
Now here are the tests:
14 15 16 17 18 19 20 21
hostname_uh = HostnameUH(hostname, None) assert hostname_uh.hostname == HOSTNAME hostname_uh = HostnameUH(None, uname) assert hostname_uh.hostname == "hostone_u.example.com" hostname_uh = HostnameUH(hostname, uname) assert hostname_uh.hostname == HOSTNAME
We use a different hostname in each parser so that we can confirm that the correct parser data is chosen.
__init__ method performs all of the work in our combiner. If
your combiner is more complex you may need to add additional methods and utility
functions. Some general recommendations for the combiner class implementation
- Choose attributes that make sense for use by actual rules, or how you
anticipate rules to use the information. If rules need to iterate over
the information then a
listmight be best, or if rules could access via keywords then
dictmight be better.
- Choose attribute types that are not so complex they cannot be easily understood or serialized. Unless you know you need something complex keep it simple.
- Use the
@propertydecorator to create read-only getters and simplify access to information.
Now we need to implement the combiner that will satisfy our tests.
1 2 3 4 5 6 7 8 9 10 11 12 13
from insights.core.plugins import combiner from insights.parsers.hostname import Hostname from insights.parsers.uname import Uname @combiner([Hostname, Uname]) class HostnameUH(object): def __init__(self, hostname, uname): if hostname: self.hostname = hostname.fqdn else: self.hostname = uname.nodename
We’ve replaced our original
__init__ to include the logic for our combiner.
Hostname parser is passed in as the
hostname attribute, and if it
is present then we use it to acquire the hostname data. If
None, meaning that there was no data or there was some error in the data
Hostname parser, we fall back to use the
Uname parser data
passed in the
Now save this file and run the tests again to confirm that we have successfully written our combiner to pass all tests:
(env)[userone@hostone insights-core-tutorials]$ pytest -k hostname_uh ======================= test session starts ============================== platform linux -- Python 3.6.6, pytest-3.0.6, py-1.7.0, pluggy-0.4.0 rootdir: /home/userone/insights-core-tutorials, inifile: setup.cfg plugins: cov-2.6.1 collected 6 items / 5 deselected insights_examples/combiners/tests/test_hostname_uh.py . [ 50%] mycomponents/combiners/tests/test_hostname_uh.py . ============ 2 passed, 14 deselected in 0.35 seconds ====================
Combiner Documentation and Testing¶
The last step to complete implementation of our combiner is to create the documentation. The guidelines and examples for combiner documentation is provided in the section Documentation Guidelines and parallels the information provided in the instructions for Parser Documentation. Combiner testing parallels the information provided in the instructions for the Parser Testing