Day 16 - Inspec gives insight

By: Jamie Andrews (@vorsprungbike)

Edited By: Ben Cotton (@FunnelFiasco)

What is Inspec?

How often to do want to scan your servers, all of them, to check a library has the correct version or that the right disk clone is mounted?

Configuration control systems like Puppet, Chef and Ansible have become de rigueur in running even modest sized groups of servers. Configuration control offers centralised control over a state which is set by policy.

However, sometimes something happens to one or more servers which challenges this happy state of affairs.

There might be a change requirement that doesn't fit in with the way that the configuration system works. You might decide that part of the system should be managed in a different way. Someone in your team might "temporarily" disable the configuration agent and then not turn it back on again - configuration drift happens

When it does happen, inevitably there are hard to trace and puzzling to understand problems.

This might be a job for a quick shell script and some ssh magic. But often I've done this and then realised that it's difficult to get just right. Inspec is a framework for systematically carrying out these types of tests.

Inspec can be very useful in these circumstances. It works like a configuration management tool like Puppet or Chef in that it holds a model of what the correct configuration should be. However, it does not modify the configuration. Instead it tests the targeted systems for compliance to the correct configuration.

I have found this to be useful in the situation where there are many systems, just a few of which have a problem. Inspec can pinpoint the problem systems, given a configuration profile.

And of course Inspec can be used to proactively check servers to detect problems before they occur

Inspec is based on an earlier system called "serverspec" and it uses the ruby classes from the rspec package.

Although it is promoted by Chef as part of their wider product offering, it works just fine standalone and is fully open source

What I'm covering in the article

Below, I'll look at installing Inspec, making a simple project "profile", how it works in action and installing a third party security checking profile

Setting up and installing

The easiest way of installing it is to use a package. Packages for Redhat and varients, Ubuntu, Suse, MacOSX and MS Windows are available here https://downloads.chef.io/inspec

Inspec has a dependency on Ruby. The packages include a bundled version of Ruby that avoids compatiblity problems. If you already have ruby installed and want to use it then "gem install inspec" is available. See the github repo https://github.com/chef/inspec for more details

To check Inspec is installed try this command

inspec version

Which will come back with the current installed version

Inspec does not have to be installed on all the target nodes. I have it installed on one admin host with ssh access to everything else. This allows any profile rule sets to be tested on anything it can ssh to. No agents, install of ruby or anything else is required

Make your own profile

inspec works by using a profile directory with a a set of control files that contain tests The "init profile" cli command is used to make a new profile

To see a generic blank profile do

inspec init profile example

The output from this command is something like

$ inspec init profile example
WARN: Unresolved specs during Gem::Specification.reset:
      rake (>= 0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Create new profile at /home/jan/temp/example
 * Create directory libraries
 * Create file inspec.yml
 * Create directory controls
 * Create file controls/example.rb
 * Create file README.md

It has made a set of files. The most interesting is "example/controls/example.rb"

This is very simple test that checks if /tmp exists, take a look at it

# encoding: utf-8
# copyright: 2017, The Authors

title 'sample section'

# you can also use plain tests
describe file('/tmp') do
  it { should be_directory }
end

# you add controls here
control 'tmp-1.0' do                  # A unique ID for this control
  impact 0.7                          # The criticality, if this control fails.
  title 'Create /tmp directory'       # A human-readable title
  desc 'An optional description...'
  describe file('/tmp') do            # The actual test
    it { should be_directory }
  end
end

The tests can be declared as "plain tests" or "controls". Being a control adds some metadata which makes it easier to track the test within a set_con

The actual test assertions "it { should be_directory }" follow the rspec syntax. The tests operate on a resource type, in this case "file". There are many useful built in test resourse types, including

• apache_conf
• crontab
• docker
• etc_fstab
• http

And a lot more, see https://www.inspec.io/docs/reference/resources/

A more complex example

Here's a real test I wrote a couple of weeks ago to deal with a DNS configuration drift.

The old DNS servers had been retired but we noticed that some servers still mentioned servers on the old network.

# encoding: utf-8

title 'DNS'

# some crude way to build a list of network interfaces
eth_files= ['/etc/sysconfig/network-scripts/ifcfg-eth0']
eth_files << '/etc/sysconfig/network-scripts/ifcfg-eth1' if file('/etc/sysconfig/network-scri
pts/ifcfg-eth1').exist?
eth_files << '/etc/sysconfig/network-scripts/ifcfs-ens32' if file('/etc/sysconfig/network-scr
ipts/ifcfg-ens32').exist?

control 'resolv' do                        # 
  impact 0.7                               # 
  title 'check old dns is not present'     # 
  desc 'old dns'
  describe file('/etc/resolv.conf') do     # The actual test
    its ('content')  { should_not match /193/ }
  end

  eth_files.each do|ef| 
  describe file(ef) do
    its ('content') { should_not match /^DOMAIN=193/ }
    its ('content') { should_not match /^DNS[123]=193/ }
  end
  end

end

I won't explain exactly how it works, you can see that there are regexps in there and that a ruby "each do" construct is used.

To run the tests do

inspec exec tc-config myuser@myhostname

As mentioned above, Inspec does not correct these problems. It is great at one job: checking compliance. Once the problem is found then you will have to devise a good method for fixing it.

I when looking at large numbers of live servers I usually run it in a shell and then redirect the output to a file. Once the time consuming checking is done I look at the file The colourizing makes it easy to spot the non-compliant areas

In the above example, I found 12 servers non-compliant out of 146. Some problems were found where Puppet conflicted with Redhat system policy I devised a simple, non-idempotent bash script and applied it to the affected servers only. This was quicker and a more certain result than running it on all the servers. After the correction, I reran the Inspec profile to see that everything was ok

Check to check the check

Once you start trying to use your own tests there is always scope for typos or syntax error or other sorts of mayhem. Inspec tries to help with a static checker.

inspec check example

Comes back with a report of how many controls there are plus if everything is valid.

This feature is a great idea, especially for those of us that are only using this tool occassionally.

Ready to use sets of tests

As the Inspec system is supported and promoted by Chef there are a set of profiles ready made that perform various types of compliance. These can be downloaded and used, see https://supermarket.chef.io/tools?type=compliance_profile

Installing and using the CIS DIL

One really useful profile is the CIS Distribution Independent Linux Benchmark https://github.com/dev-sec/cis-dil-benchmark

To try it, clone that github repo and then

 inspec check cis-dil-benchmark

It has 88 controls, many of which check multiple resources. On our LAN running it against a host took over 3 minutes.

The report it generates is interesting reading.

We will be using this profile for testing images generated with Packer and Puppet. The issues it reports will act as feedback for security improvements to our configuration scripts in Puppet and Packer

Further features

I am just scratching the surface with the many features that Inspec offers.
Please do look at http://inspec.io for a fuller run down!

Thanks

Thanks to Chef for supporting this excellent software and keeping it open source

Thanks to my employers, Institute of Physics Publishing for enabling me to try out cool stuff like this as part of my job.