Why not to use Puppet::Parser::Functions.autoloader.loadall

Recently (about 5 minutes ago), I was writing a custom puppet-function to offload some puppet magic. In short: I’m writing a wrapper around create_resources so I can keep syntax for the end-users of my module crispy clean. This means I need the create_resources function to be available in my custom function. This can be done by using Puppet::Parser::Functions.autoloader.loadall as suggested on the puppetlabs custom modules guide. Unfortunately, when using #loadall, all functions will be loaded.

Why unfortunately? In my case: A function defined in puppet-foreman depends on the rest-client gem and I do not have this installed. Some people might say: Just install the gem and be done with it! This is hardly a proper solution. The way to go would to be only include the function I really  need, being create_resources.

And here is how:

Puppet::Parser::Functions.autoloader.load(:create_resources) unless Puppet::Parser::Functions.autoloader.loaded?(:create_resources)

This will basically load the create_resources function after checking that it has not been loaded before. This (the function already being loaded) could be the case if you properly depend on puppetlabs-create_resources in your manifests. Side note: I added a small dummy class so my modules can depend on this function being available.

This has resolved my issues with #loadall, but if I ever needed to include another function that DOES use #loadall, I’ll be screwed all over again. So (pretty) pls, don’t use #loadall.

Puppet Module Patterns

INTRODUCTION

I’ve used puppet quite intensively since a couple of months (about 4 I would guess). Before that, I’ve played with it, change something here and there. But quite not as much as now. I’ve used several puppet modules from wherever google leads me, roamed github, inherited a few from colleagues and created several from scratch. While doing so, I saw a lot of stuff I disliked and learned a lot on how we I can (ab)use puppet to do what I want it to do. Over those last months, I have grown my set of ideas on how a puppet module should look. So, before every statement I make, you should probably add ‘IMHO’.

(more…)

Reducing vagrant box size

Here are some tricks I use to make my vagrant boxes as small as possible:

Tips:

Booting in single user mode:

I boot in single user mode since it will prevent running services that could output logs. I do this because I zero out all my logs before packaging the box.

Updating:

After updating any package, run yum clean (or the apt equivalent).

When booted in single user mode, don’t forget to start-up your network before updating.

When updating kernels, install the kernel packages, reboot and remove the old kernel packages that are no longer in use. Remember to re-install the VirtualBox add-ons too after a kernel update.

Cleanup:

After doing whatever you need to do with the box, I do some rather nasty stuff to make sure the box uses as little as possible place. If you are using a RAW hard-disks, these might be a bad idea (stuff gets BIG).

  • Zero out all remaining unused disk space
  • Zero out the swap
  • Clear out all log files (I just make them empty, I do NOT delete them)
Script:

(You can find this script – or an older version in /root/tools/cleanup_diskspace.sh on my newer vagrant boxes.)

cat - << EOWARNING
WARNING: This script will fil up your left over disk space.
 
DO NOT RUN THIS WHEN YOUR VIRTUAL HD IS RAW!!!!!!
 
You should NOT do this on a running system.
This is purely for making vagrant boxes damn small.
 
Press Ctrl+C within the next 10 seconds if you want to abort!!
 
EOWARNING
sleep 10;
 
echo 'Cleanup bash history'
unset HISTFILE
[ -f /root/.bash_history ] && rm /root/.bash_history
[ -f /home/vagrant/.bash_history ] && rm /home/vagrant/.bash_history
 
echo 'Cleanup log files'
find /var/log -type f | while read f; do echo -ne '' > $f; done;
 
echo 'Whiteout root'
count=`df --sync -kP / | tail -n1  | awk -F ' ' '{print $4}'`; 
let count--
dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count;
rm /tmp/whitespace;
 
echo 'Whiteout /boot'
count=`df --sync -kP /boot | tail -n1 | awk -F ' ' '{print $4}'`;
let count--
dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count;
rm /boot/whitespace;
 
swappart=`cat /proc/swaps | tail -n1 | awk -F ' ' '{print $1}'`
swapoff $swappart;
dd if=/dev/zero of=$swappart;
mkswap $swappart;
swapon $swappart;

Furthermore – about this script – USE IT AT YOUR OWN RISK

Puppet modules in Jenkins.

Code style checking

Prerequisites:

  • You will need a recent enough version of puppet-lint that supports the --log-format flag. Install the gem so that the Jenkins can use it.
  • On Jenkins, you will need the Warnings Plugin and the HTML Publisher Plugin.
  • Make sure that when checking the module from your VCS, it ends up in WORKSPACE/modules/module_name.

Configuration:

Jenkins

Go to the Configure System page and find the Compiler Warnings settings. Add a new console log parser and call it puppet-lint. I use following configuration for parsing puppet-lint warnings and errors.

The warnings plugin has been updated and now has puppet-lint support out of the box! So configuring puppet-lint manually is kind of useless now.

Name:

puppet-lint

Regular Expression:

^\s*([^:]+):([0-9]+):([^:]+):([^:]+):\s*(.*)$

Mapping Script:

import hudson.plugins.warnings.parser.Warning
// map regular expression to strings
String fileName = matcher.group(1);
String lineNumber = matcher.group(2);
String kind = matcher.group(3);
String check = matcher.group(4);
String message = matcher.group(5);
// return a Warning.
return new Warning(fileName, Integer.parseInt(lineNumber), check, kind, message);

Example Log Message:

./manifests/params.pp:25:autoloader_layout:error:apache::params not in autoload module layout

Jenkins job configuration

We will add several build steps that will run certain actions on our puppet modules.

  1. Check syntax
  2. Check style
  3. Generate documentation

1. For the syntax check, I use following shell script (add a build step):

for file in $(find . -iname '*.pp'); do
&nbsp; puppet parser validate --color false --render-as s --modulepath=modules $file || exit 1;
done;

2. For the style check, we use puppet-lint (add another build step):

find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {} \;

3. And for generating documentation:

## Cleanup old docs.
[ -d doc/ ] && rm -rf doc/
## Dummy manifests folder.
! [ -d manifests/ ] && mkdir manifests/
## Generate docs
puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc
 
## Fix docs to how I want them, I don't like that the complete workspace is included in all file paths.
if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then
  mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules"
fi;
grep -l -R ${WORKSPACE} * | while read fname; do sed -i "s@${WORKSPACE}/@/@g" $fname; done;

In your post build section:

  • Enable Scan for compiler warnings and select puppet-lint.
  • Enable publish HTML reports (use ‘doc‘, ‘index.html‘ and ‘Puppet Docs‘ as values). This will add a link to the Job page linking your generated puppet docs.

That’s about it! Any suggestions / improvements on this are always welcome!

Notes:

  • I have some examples/tests setup on my Jenkins instance for testing at http://jenkins.vstone.eu. Since I use this for testing, it might be offline / broken / buggy at times.
  • The scripts I use may also require some changes if you are using an older version of puppet. I’m currently using 2.7.x for testing my modules.

Puppet modules and using dot graphs (both are unrelated but related to each other)

Puppet modules…  How I feel about them in a dot file:

digraph PuppetModules {
  node [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
    shape    = "record"
  ]
  edge [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
  ]
  question [label="Do I need to edit a file in your module for changing settings?", shape="oval"]
  ok  [label="Great.", shape="oval"]
  bah [label="You are doing it WRONG!", shape="oval"]

  question -> ok [label="No"]
  question -> bah [label="Yes"]
}

Read on if you want a rendered version.
(more…)

My vagrant base boxes for puppet dev(op)s

You can, from now on, find any vagrant box I package here: http://packages.vstone.org/vagrant-boxes

Note:
Boxes that end with .box-pp are packaged with my modified Vagrantfile which has been preconfigured for use with puppet. In addition there are some optional commands available to do (puppet) development quicker. See vagrant-addons @ github for more information on this subject.

Vagrant quickstart for Puppet dev(op)s

A quick introduction on how I use vagrant for developing my puppet manifests/modules/. You can almost certain also use this for other purposes. In general, this will get you up to speed fast!
We will quickly go over installation and/or updating and maybe even removing an old version using ruby gems.
Furthermore: adding a vagrant box and preparing a project to develop a puppet module (or something of the likes).
(more…)

Simulating hash tables in bash

On my previous blog (which I did not migrate), I posted this snippet to simulate hash tables in bash. Is it useful? Maybe. In any case, this is an slightly improved version which allows ‘values’ to have spaces in them. Apparently (I’ve never tested it before today), this was kind of a problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash
hash_table=(
    key1='value1'
    key2='value2'
)
 
hash_get() {
    local _table="$1";
    local _key="$2";
    local _value="${_table}[@]";
    local "${!_value}";
    eval echo "\$${_key}";
}
 
hash_get "hash_table" "key2";

Hello world!

My new wordpress installation! I have yet to figure out wether or not to import my old blog entries. I don’t feel much like it for now, but I might change my mind later. For now, it’s going to be empty! (unless I feel like blogging).

Some info on the setup? I’m using wordpress svn with some git-svn magic to keep my code in version control. Using git-svn also solved my biggest hastle I used to have that I couldn’t easily mix a wordpress installation in my document root with existing files and/or folders (error pages, some stats folder, various other thingies).

The wordpress plugins I use are also managed in git, but not with git-svn. I’m unsure if its possible to use git-svn with git-submodules. And even if it is, it may be too much of a command mess to update them all. Instead, I’ll just write a little script I can run whenever wordpress complains about updating plugins that will loop over them and run a svn update (with a git commit with a nicely predefined commit message.

Code will follow!