Git

Git is a version control system written by Linus Torvalds, which is used to collaborate development on the Linux kernel source code. Its support for rapid branching and merging makes it the perfect choice for a Puppet implementation. In Git, each source code commit has references to its parent commit; to reconstruct a branch, you only need to follow the commit trail back. We will be exploiting the rapid branch support to have environments defined from Git branches.

Note

It is possible to use Git without a server and to make copies of repositories using only local Git commands.

In your organization, you are likely to have some version control software. The software in question isn't too important, but the methodology used is important.

Tip

Remember that passwords and sensitive information stored in version control will be available to anyone with access to your repository. Also, once stored in version control, it will always be available.

Long running branches or a stable trunk are the terms used in the industry to describe the development cycle. In our implementation, we will assume that development and production are long running branches. By long running, we mean that these branches will persist throughout the lifetime of the repository. Most of the other branches are dead ends—they solve an immediate issue and then get merged into the long running branches and cease to exist, or they fail to solve the issue and are destroyed.

Why Git?

Git is the de facto version control software with Puppet because of its implementation of rapid branching. There are numerous other reasons for using Git in general. Each user of Git is given a complete copy of the revision history whenever they clone a Git repository. Each developer is capable of acting as a backup for the repository, should the need arise. Git allows each developer to work independently from the master repository; thus, allowing developers to work offsite and even without network connectivity.

This section isn't intended to be an exhaustive guide of using Git. We'll cover enough commands to get your job done, but I recommend that you do some reading on the subject to get well acquainted with the tool.

Note

The main page for Git documentation is http://git-scm.com/documentation. Also worth reading is the information on getting started with Git by GitHub at http://try.github.io or the Git for Ages 4 and Up video available at http://mirror.int.linux.conf.au/linux.conf.au/2013/ogv/Git_For_Ages_4_And_Up.ogv.

To get started with Git, we need to create a bare repository. By bare, we mean that only the meta information and checksums will be stored; the files will be in the repository but only in the checksum form. Only the main location for the repository needs to be stored in this fashion.

In the enterprise, you want the Git server to be a separate machine, independent of your Puppet master. Perhaps, your Git server isn't even specific for your Puppet implementation. The great thing about Git is that it doesn't really matter at this point; we can put the repository wherever we wish.

To make things easier to understand, we'll work on our single worker machine for now, and in the final section, we will create a new Git server to hold our Git repository.

Note

GitHub or GitHub Enterprise can also be used to host Git repositories. GitHub is a public service but it also has pay account services. GitHub Enterprise is an appliance solution to host the same services as GitHub internally, within your organization.

A simple Git workflow

On our standalone machine, install Git using yum, as shown here:

[root@stand ~]# yum install -y git
...
Installed: git.x86_64 0:1.8.3.1-5.el7

Now, decide on a directory to hold all your Git repositories. We'll use /var/lib/git in this example.

Note

A directory under /srv may be more appropriate for your organization. Several organizations have adopted the /apps directory for application specific data, as well and using these locations may have SELinux context considerations. The targeted policy on RedHat systems provides for the /var/lib/git and /var/www/git locations for Git repository data.

The /var/lib/git path closely resembles the paths used by other EL packages. Since running everything as root is unnecessary, we will create a Git user and make that user the owner of the Git repositories.

Create the directory to contain our repository first (/var/lib/git) and then create an empty Git repository (using the git init –bare command) in that location, as shown in the following code:

[root@stand ~]# useradd git -c 'Git Repository Owner' -d /var/lib/git
[root@stand ~]# sudo -iu git
[git@stand ~]$ pwd
/var/lib/git
[git@stand ~]$ chmod 755 /var/lib/git
[git@stand ~]$ git init --bare control.git
Initialized empty Git repository in /var/lib/git/control.git/
[git@stand ~]$ cd /tmp
[git@stand tmp]$ git clone /var/lib/git/control.git
Cloning into 'control'...
warning: You appear to have cloned an empty repository.
done.
[git@stand tmp]$ cd control
[git@stand control]$ git status
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)

Tip

Using git --bare will create a special copy of the repository where the code is not checked out; it is known as bare because it is without a working copy of the code. A normal copy of a Git repository will have the code available at the top-level directory and the Git internal files in a .git directory. A bare repository has the contents of the .git directory located in the top level directory.

Now that our repository is created, we should start adding files to the repository. However, we should first configure Git. Git will store our username and e-mail address with each commit. These settings are controlled with the git config command. We will add the --global option to ensure that the config file in ~/.git is modified, as shown in the following example:

[git@stand ~]$ git config --global user.name 'Git Repository Owner'
[git@stand ~]$ git config --global user.email 'git@example.com' 

Now, we'll copy in our production modules and commit them. We'll copy the files from the /etc/puppet/environments/production directory of our worker machines and then add them to the repository using the git add command, as shown here:

[git@stand ~]$ cd /tmp/control/
[git@stand control]$ cp -a /etc/puppetlabs/code/environments/production/* .
[git@stand control]$ ls environment.conf hieradata manifests modules
[git@stand control]$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# environment.conf
# hieradata/
# manifests/

# modules/
nothing added to commit but untracked files present (use "git add" to track) 

We've copied our hieradata, manifests, and modules directories, but Git doesn't know anything about them. We now need to add them to the Git repository and commit to the default branch master. This is done with two Git commands, first using git add and then using git commit, as shown in the following code:

[git@stand control]$ git add hieradata manifests modules environment.conf
[git@stand control]$ git commit -m "initial commit"
[master (root-commit) 316a391] initial commit14 files changed, 98 insertions(+)…
 create mode 100644 modules/web/manifests/init.pp

Note

To see the files that will be committed when you issue git commit, use git status after the git add command. It is possible to commit in a single command using git commit –a. This will commit all staged files (these are the files that have changed since the last commit). I prefer to execute the commands separately to specifically add the files, which I would like to add in the commit. If you are editing a file with vim, you may inadvertently commit a swap file using git commit –a.

At this point, we've committed our changes to our local copy of the repository. To ensure that we understand what is happening, we'll clone the initial location again into another directory (/tmp/control2), using the following commands:

[git@stand control]$ cd /tmp
[git@stand tmp]$ mkdir control2
[git@stand tmp]$ git clone /var/lib/git/control.git .fatal: destination path '.' already exists and is not an empty directory.
[git@stand tmp]$ cd control2
[git@stand control2]$ git clone /var/lib/git/control.git .Cloning into '.'...warning: You appear to have cloned an empty repository.done.
[git@stand control2]$ ls 

Our second copy doesn't have the files we just committed, and they only exist in the first local copy of the repository. One of the most powerful features of Git is that it is a self-contained environment. Going back to our first clone (/tmp/control), examine the contents of the .git/config file. The url setting for remote "origin" points to the remote master that our repository is based on (/var/lib/git/control.git), as shown in the following code:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = /var/lib/git/control.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master 

In Git, origin is where the original remote repository lives. In this example, it is a local location (/var/lib/git/control.git), but it can also be an HTTPS URI or SSH URI.

To push the local changes to the remote repository, we use git push; the default push operation is to push it to the first [remote] repository (named origin by default) to the currently selected branch (the current branch is given in the output from git status). The default branch in Git is named master, as we can see in the [branch "master"] section. To emphasize what we are doing, we'll type in the full arguments to push (although git push will achieve the same result in this case), as you can see here:

[git@stand control]$ cd
[git@stand ~]$ cd /tmp/control
[git@stand control]$ git push origin master
Counting objects: 34, done.
Compressing objects: 100% (15/15), done.
Writing objects: 100% (34/34), 3.11 KiB | 0 bytes/s, done.
Total 34 (delta 0), reused 0 (delta 0)
To /var/lib/git/control.git
 * [new branch] master -> master 

Now, even though our remote repository has the updates, they are still not available in our second copy (/tmp/control2). We must now pull the changes from the origin to our second copy using git pull. Again, we will type in the full argument list (this time, git pull will do the same thing), as shown here:

[git@stand ~]$ cd /tmp/control
[git@stand control]$ git push origin master
Counting objects: 34, done.
Compressing objects: 100% (15/15), done.
Writing objects: 100% (34/34), 3.11 KiB | 0 bytes/s, done.
Total 34 (delta 0), reused 0 (delta 0)
To /var/lib/git/control.git
 * [new branch] master -> master
[git@stand control]$ cd
[git@stand ~]$ cd /tmp/control2
[git@stand control2]$ git status
# On branch master
#
# Initial commit
#
nothing to commit (create/copy files and use "git add" to track)
[git@stand control2]$ ls
[git@stand control2]$ git pull origin master
remote: Counting objects: 34, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 34 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (34/34), done.
From /var/lib/git/control
 * branch master -> FETCH_HEAD
[git@stand control2]$ ls
environment.conf hieradata manifests modules 

Two useful commands that you should know at this point are git log and git show. The git log command will show you the log entries from Git commits. Using the log entries, you can run git show to piece together what your fellow developers have been doing. The following snippet shows the use of the following two commands in our example:

[git@stand control2]$ git log
commit 316a391e2641dd9e44d2b366769a64e88cc9c557
Author: Git Repository Owner <git@example.com>
Date: Sat Sep 26 19:13:41 2015 -0400

 initial commit
[git@stand control2]$ git show 316a391e2641dd9e44d2b366769a64e88cc9c557
commit 316a391e2641dd9e44d2b366769a64e88cc9c557
Author: Git Repository Owner <git@example.com>
Date: Sat Sep 26 19:13:41 2015 -0400

 initial commit

diff --git a/environment.conf b/environment.conf
new file mode 100644
index 0000000..c39193f
--- /dev/null
+++ b/environment.conf
@@ -0,0 +1,18 @@
+# Each environment can have an environment.conf file. Its settings will only
+# affect its own environment. See docs for more info:
+# https://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html
...

The git show command takes the commit hash as an optional argument and returns all the changes that were made with that hash.

Now that we have our code in the repository, we need to create a production branch for our production code. Branches are created using git branch. The important concept to be noted is that they are local until they are pushed to the origin. When git branch is run without arguments, it returns the list of available branches with the currently selected branch highlighted with an asterisk, as shown here:

[git@stand ~]$ cd /tmp/control
[git@stand control]$ git branch
* master
[git@stand control]$ git branch production
[git@stand control]$ git branch
* master
 production 

This sometimes confuses people. You have to checkout the newly created branch after creating it. You can do this in one step using the git checkout -b <branch_name> command, but I believe using this shorthand command initially leads to confusion. We'll now check our production branch and make a change. We will then commit to the local repository and push to the remote, as shown here:

[git@stand control]$ git checkout production
Switched to branch 'production'
[git@stand control]$ git branch
 master
* production
[git@stand control]$ cd hieradata/hosts/
[git@stand hosts]$ sed -i -e 's/watch your step/best behaviour/' client.yaml 
[git@stand hosts]$ git add client.yaml 
[git@stand hosts]$ git commit -m "modifying welcome message on client"
[production 74d2ff5] modifying welcome message on client
 1 file changed, 1 insertion(+), 1 deletion(-)
[git@stand hosts]$ git push origin production
Counting objects: 9, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 569 bytes | 0 bytes/s, done.
Total 5 (delta 1), reused 0 (delta 0)
To /var/lib/git/control.git
 * [new branch] production -> production n

Now, in our second copy of the repository, let's confirm that the production branch has been added to the origin, using git fetch to retrieve the latest metadata from the remote origin, as shown here:

[git@stand hosts]$ cd /tmp/control2/
[git@stand control2]$ git branch -a
* master
[git@stand control2]$ git fetch
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From /var/lib/git/control
 * [new branch] master -> origin/master
 * [new branch] production -> origin/production
[git@stand control2]$ git branch -a
* master
 remotes/origin/master
 remotes/origin/production 

It is important to run git fetch routinely, to take a look at the changes that your teammates may have made and branches that they may have created. Now, we can verify whether the production branch has the change we made. The –a option to git branch instructs Git to include remote branches in the output. We'll display the current contents of client.yaml and then run git checkout production to see the production version, as shown here:

[git@stand control2]$ grep welcome hieradata/hosts/client.yaml 
welcome: 'Production Node: watch your step.'
[git@stand control2]$ git checkout production
Branch production set up to track remote branch production from origin.
Switched to a new branch 'production'
[git@stand control2]$ grep welcome hieradata/hosts/client.yaml 
welcome: 'Production Node: best behaviour.'

As we can see, the welcome message in the production branch is different from that of the master branch. At this point, we'd like to have the production branch in /etc/puppetlabs/code/environments/production and the master branch in /etc/puppetlabs/code/environments/master. We'll perform these commands, as the root user, for now:

[root@stand ~]# cd /etc/puppetlabs/code/
[root@stand code]# mv environments environments.orig
[root@stand code]# mkdir environments
[root@stand code]# cd environments
[root@stand environments]# for branch in production master
> do
> git clone -b $branch /var/lib/git/control.git $branch
> done
Cloning into 'production'...
done.
Cloning into 'master'...
done. 

Now that our production branch is synchronized with the remote, we can do the same for the master branch and verify whether the branches differ, using the following command:

[root@stand environments]# diff production/hieradata/hosts/client.yaml master/hieradata/hosts/client.yaml 
2c2
< welcome: 'Production Node: best behaviour.'
---
> welcome: 'Production Node: watch your step.'

Running Puppet on client in the production environment will now produce the change we expect in /etc/motd, as follows:

Production Node: best behaviour.
Managed Node: client
Managed by Puppet version 4.2.2 

Note

If you changed hiera.yaml for the single tree example, change it to the following:

:datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata"

Run the agent again with the master environment, to change motd, as shown here:

[root@client ~]# puppet agent -t --environment master
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for client.example.com
Info: Applying configuration version '1443313038'
Notice: /Stage[main]/Virtual/Exec[set tuned profile]/returns: executed successfully
Notice: /Stage[main]/Base/File[/etc/motd]/content: 
--- /etc/motd 2015-10-01 22:23:02.786866895 -0700
+++ /tmp/puppet-file20151001-12407-16iuoej 2015-10-01 22:24:02.999870073 -0700
@@ -1,3 +1,3 @@
-Production Node: best behaviour.
+Production Node: watch your step.
 Managed Node: client
 Managed by Puppet version 4.2.2

Info: Computing checksum on file /etc/motd
Info: /Stage[main]/Base/File[/etc/motd]: Filebucketed /etc/motd to puppet with sum 490af0a672e3c3fdc9a3b6e1bf1f1c7b
Notice: /Stage[main]/Base/File[/etc/motd]/content: content changed '{md5}490af0a672e3c3fdc9a3b6e1bf1f1c7b' to '{md5}8147bf5dbb04eba29d5efb7e0fa28ce2'
Notice: Applied catalog in 1.07 seconds 

Our standalone Puppet master is now configured such that each branch of our control repository is mapped to a separate Puppet environment. As new branches are added, we have to set up the directory manually and push the contents to the new directory. If we were working in a small environment, this arrangement of Git pulls will be fine; but, in an enterprise, we would want this to be automatic. In a large environment, you would also want to define your branching model to ensure that all your team members are working with branches in the same way. Good places to look for branching models are http://nvie.com/posts/a-successful-git-branching-model/ and https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows. Git can run scripts at various points in the commitment of code to the repository—these scripts are called hooks.

Git hooks

Git provides several hook locations that are documented in the githooks man page. The hooks of interest are post-receive and pre-receive. A post-receive hook is run after a successful commit to the repository and a pre-receive hook is run before any commit is attempted. Git hooks can be written in any language; the only requirement is that they should be executable.

Each time you commit to a Git repository, Git will create a hash that is used to reference the state of the repository after the commit. These hashes are used as references to the state of the repository. A branch in Git refers to a specific hash, you can view this hash by looking at the contents of .git/HEAD, as shown here:

[root@stand production]# cat .git/HEAD
ref: refs/heads/production

The hash will be in the file located at .git/refs/heads/production, as shown here:

[root@stand production]# cat .git/refs/heads/production 
74d2ff58470d009e96d9ea11b9c126099c9e435a

The post-receive and pre-receive hooks are both passed three parameters via stdin: the first is the commit hash that you are starting from (oldrev), the second is the new commit hash that you are creating (newrev), and the third is a reference to the type of change that was made to the repository, where the reference is the branch that was updated. Using these hooks, we can automate our workflow. We'll start using the post-receive hook to set up our environments for us.

Using post-receive to set up environments

What we would like to happen at this point is a series of steps discussed as follows:

1. A developer works on a file in a branch.
2. The developer commits the change and pushes it to the origin.
3. If the branch doesn't exist, create it in /etc/puppetlabs/code/environments/<branch>.
4. Pull the updates for the branch into /etc/puppetlabs/code/environments/<branch>.

In our initial configuration, we will write a post-receive hook that will implement steps 3 and 4 mentioned previously. Later, we'll ensure that only the correct developers commit to the correct branch with a pre-receive hook. To ensure that our Puppet user has access to the files in /etc/puppetlabs/code/environments, we will use the sudo utility to run the commits, as the Puppet user.

Our hook doesn't need to do anything with the reference other than extract the name of the branch and then update /etc/puppetlabs/code/environments, as necessary. To maintain the simplicity, this hook will be written in bash. Create the script in /var/lib/git/control.git/hooks/post-receive, as follows:

#!/bin/bash
PUPPETDIR=/etc/puppetlabs/code/environments
REPOHOME=/var/lib/git/control.git
GIT=/bin/git
umask 0002
unset GIT_DIR

We will start by setting some variables for the Git repository location and Puppet environments directory location. It will become clear later why we set umask at this point, we want the files created by our script to be group writable. The unset GIT_DIR line is important; the hook will be run by Git after a successful commit, and during the commit GIT_DIR is set to ".". We unset the variable so that Git doesn't get confused.

Next, we will read the variables oldrev, newrev, and refname from stdin (not command-line arguments), as shown in the following code:

read oldrev newrev refname
branch=${refname#*\/*\/}
if [ -z $branch ]; then
 echo "ERROR: Updating $PUPPETDIR"
 echo "       Branch undefined"
 exit 10
fi

After extracting the branch from the third argument, we will verify whether we were able to extract a branch. If we are unable to parse out the branch name, we will quit the script and warn the user.

Now, we have three scenarios that we will account for in the script. The first is that the directory exists in /etc/puppetlabs/code/environments and that it is a Git repository, as shown:

# if directory exists, check it is a git repository
if [ -d "$PUPPETDIR/$branch/.git" ]; then
  cd $PUPPETDIR/$branch
  echo "Updating $branch in $PUPPETDIR"
  sudo -u puppet $GIT pull origin $branch
  exit=$?

In this case, we will cd to the directory and issue a git pull origin <branchname> command to update the directory. We will run the git pull command using sudo with -u puppet to ensure that the files are created as the Puppet user.

The second scenario is that the directory exists but it was not created via a Git checkout. We will quit early if we run into this option, as shown in the following snippet:

elif [ -d "$PUPPETDIR/$branch" ]; then
  # directory exists but is not in git
  echo "ERROR: Updating $PUPPETDIR"
  echo "       $PUPPETDIR/$branch is not a git repository"
  exit=20

The third option is that the directory doesn't exist yet. In this case, we will clone the branch using the git clone command in a new directory, as the Puppet user (using sudo again), as shown in the following snippet:

else
  # directory does not exist, create
  cd $PUPPETDIR
  echo "Creating new branch $branch in $PUPPETDIR"
  sudo -u puppet $GIT clone -b $branch $REPOHOME $branch
  exit=$?
fi

In each case, we retained the return value from Git so that we can exit the script with the appropriate exit code at this point, as follows:

exit $exit

Now, let's see this in action. Change the permissions on the post-receive script to make it executable (chmod 755 post-receive). Now, to ensure that our Git user can run the Git commands as the Puppet user, we need to create a sudoers file. We need the Git user to run /usr/bin/git; so, we put in a rule to allow this in a new file called /etc/sudoers.d/sudoers-puppet, as follows:

git ALL = (puppet) NOPASSWD: /bin/git *

In this example, we'll create a new local branch, make a change in the branch, and then push the change to the origin. Our hook will be called and a new directory will be created in /etc/puppet/environments.

[root@stand ~]# chown puppet /etc/puppetlabs/code/environments 
[root@stand ~]# sudo -iu git
[git@stand ~]$ ls /etc/puppetlabs/code/environments
master production
[git@stand ~]$ cd /tmp/control
[git@stand control]$ git branch thomas
[git@stand control]$ git checkout thomas
Switched to branch 'thomas'
 1 files changed, 1 insertions(+), 1 deletions(-)
[git@stand control]$ sed -i hieradata/hosts/client.yaml -e "s/welcome:.*/welcome: 'Thomas Branch'/"
[git@stand control]$ git add hieradata/hosts/client.yaml
[git@stand control]$ git commit -m "Creating thomas branch"
[thomas 598d13b] Creating Thomas branch
 1 file changed, 1 insertion(+)
[git@stand control]$ git push origin thomas
Counting objects: 9, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 501 bytes | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
To /var/lib/git/control.git
 * [new branch] thomas -> thomas
remote: Creating new branch thomas in /etc/puppetlabs/code/environments
remote: Cloning into 'thomas'...
remote: done.
To /var/lib/git/control.git
 b0fc881..598d13b thomas -> thomas
[git@stand control]$ ls /etc/puppetlabs/code/environments
master production thomas

Our Git hook has now created a new environment, without our intervention. We'll now run puppet agent on the node to see the new environment in action, as shown here:

[root@client ~]# puppet agent -t --environment thomas
…
Notice: /Stage[main]/Base/File[/etc/motd]/content: 
--- /etc/motd 2015-10-01 22:24:03.057870076 -0700
+++ /tmp/puppet-file20151001-12501-1y5tl02 2015-10-01 22:55:59.132971224 -0700
@@ -1,3 +1,3 @@
-Production Node: watch your step.
+Thomas Branch
…
Notice: Applied catalog in 1.78 seconds 

Our post-receive hook is very simple, but it illustrates the power of automating your code updates. When working in an enterprise, it's important to automate all the processes that take your code from development to production. In the next section, we'll look at a community solution to the Git hook problem.

Puppet-sync

The problem of synchronizing Git repositories for Puppet is common enough that a script exists on GitHub that can be used for this purpose. The puppet-sync script is available at https://github.com/pdxcat/puppet-sync.

To quickly install the script, download the file from GitHub using curl, and redirect the output to a file, as shown here:

[root@stand ~]# curl https://raw.githubusercontent.com/pdxcat/puppet-sync/4201dbe7af4ca354363975563e056edf89728dd0/puppet-sync >/usr/bin/puppet-sync
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 7246 100 7246 0 0 9382 0 --:--:-- --:--:-- --:--:-- 9373
[root@stand ~]# chmod 755 /usr/bin/puppet-sync

To use puppet-sync, you need to install the script on your master machine and edit the post-receive hook to run puppet-sync with appropriate arguments. The updated post-receive hook will have the following lines:

#!/bin/bash
PUPPETDIR=/etc/puppetlabs/code/environments
REPOHOME=/var/lib/git/control.git

read oldrev newrev refname
branch=${refname#*\/*\/}
if [ -z "$branch" ]; then
  echo "ERROR: Updating $PUPPETDIR"
  echo "       Branch undefined"
  exit 10
fi

[ "$newrev" -eq 0 ] 2> /dev/null && DELETE='--delete' || DELETE=''
sudo -u puppet /usr/bin/puppet-sync \
  --branch "$branch" \
  --repository "$REPOHOME" \
  --deploy "$PUPPETDIR" \
  $DELETE 

To use this script, we will need to modify our sudoers file to allow Git to run puppet-sync as the Puppet user, as shown:

git ALL = (puppet) NOPASSWD: /bin/git *, /usr/bin/puppet-sync *

This process can be extended, as a solution, to push across multiple Puppet masters by placing a call to puppet-sync within a for loop, which SSHes to each worker and then runs puppet-sync on each of them.

This can be extended further by replacing the call to puppet-sync with a call to Ansible, to update a group of Puppet workers defined in your Ansible host's file. More information on Ansible is available at http://docs.ansible.com/.

To check whether puppet-sync is working as expected, create another branch and push it back to the origin, as shown:

[root@stand hooks]# sudo -iu git
[git@stand ~]$ cd /tmp/control
[git@stand control]$ git branch puppet_sync
[git@stand control]$ git checkout puppet_sync
Switched to branch 'puppet_sync'
[git@stand control]$ sed -i hieradata/hosts/client.yaml -e "s/welcome:.*/welcome: 'Puppet_Sync Branch, underscores are cool.'/"
[git@stand control]$ git add hieradata/hosts/client.yaml 
[git@stand control]$ git commit -m "creating puppet_sync branch"
[puppet_sync e3dd4a8] creating puppet_sync branch
 1 file changed, 1 insertion(+), 1 deletion(-)
[git@stand control]$ git push origin puppet_sync
Counting objects: 9, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 499 bytes | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
remote: .--------------------------------------------- PuppetSync ---
remote: | Host : stand.example.com
remote: | Branch : puppet_sync
remote: | Deploy To : /etc/puppetlabs/code/environments/puppet_sync
remote: | Repository : /var/lib/git/control.git
remote: `------------------------------------------------------------
To /var/lib/git/control.git
 e96c344..6efa315 puppet_sync -> puppet_sync

In a production environment, this level of detail for every commit will become cumbersome, puppet-sync has a quiet option for this purpose; add –q to your post-receive call to puppet-sync to enable the quiet mode.

Puppet environments must start with an alphabetic character and only contain alphabetic characters, numbers, and the underscore. If we name our branch puppet-sync, it will produce an error when attempting to execute puppet agent -t –environment puppet-sync.

Using Git hooks to play nice with other developers

Up to this point, we've been working with the Git account to make our changes. In the real world, we want the developers to work with their own user account. We need to worry about permissions at this point. When each developer commits their code, the commit will run as their user; so, the files will get created with them as the owner, which might prevent other developers from pushing additional updates. Our post-receive hook will run as their user, so they need to be able to use sudo just like the Git user. To mitigate some of these issues, we'll use Git's sharedrepository setting to ensure that the files are group readable in /var/lib/git/control.git, and use sudo to ensure that the files in /etc/puppetlabs/code/environments are created and owned by the Puppet user.

We can use Git's built-in sharedrepository setting to ensure that all members of the group have access to the repository, but the user's umask setting might prevent files from being created with group-write permissions. Putting a umask setting in our script and running Git using sudo is a more reliable way of ensuring access. To create a Git repository as a shared repository, use shared=group while creating the bare repository, as shown here:

git@stand$ cd /var/lib/git
git@stand$ git init --bare --shared=group newrepo.git
Initialized empty shared Git repository in /var/lib/git/newrepo.git/

First, we'll modify our control.git bare repository to enable shared access, and then we'll have to retroactively change the permissions to ensure that group access is granted. We'll edit /var/lib/git/control.git/config, as follows:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        sharedrepository = 1

To illustrate our workflow, we'll create a new group and add a user to that group, as shown here:

[root@stand ~]# groupadd pupdevs
[root@stand ~]# useradd -g pupdevs -c "Sample Developer" samdev [root@stand ~]# id samdev
uid=1002(samdev) gid=1002(pupdevs) groups=1002(pupdevs)

Now, we need to retroactively go back and change the ownership of files in /var/lib/git/control.git to ensure that the pupdevs group has write access to the repository. We'll also set the setgid bit on that directory so that new files are group owned by pupdevs, as shown here:

[root@stand ~]# cd /var/lib/git
[root@stand git]# find control.git -type d -exec chmod g+rwxs {} \;
[root@stand git]# find control.git -type f -exec chmod g+rw {} \;
[root@stand git]# chgrp -R pupdevs control.git

Now, the repository will be accessible to anyone in the pupdevs group. We now need to add a rule to our sudoers file to allow anyone in the pupdevs group to run Git as the Puppet user, using the following code:

%pupdevs ALL = (puppet) NOPASSWD: /bin/git *, /usr/bin/puppet-sync *

If your repo is still configured to use puppet-sync to push updates, then you need to remove the production directory from /etc/puppetlabs/code/environments before proceeding. puppet-sync creates a timestamp file (.puppet-sync-stamp) in the base of the directories it controls and will not update an existing directory by default.

With this sudo rule in place, sudo to samdev, clone the repository and modify the production branch, as shown:

[root@stand git]# sudo -iu samdev
[samdev@stand ~]$ git clone /var/lib/git/control.git/
Cloning into 'control'...
done.
[samdev@stand ~]$ cd control/
[samdev@stand control]$ git config --global user.name "Sample Developer"
[samdev@stand control]$ git config --global user.email "samdev@example.com"
[samdev@stand control]$ git checkout production
Branch production set up to track remote branch production from origin.
Switched to a new branch 'production'
[samdev@stand control]$ sed -i hieradata/hosts/client.yaml -e "s/welcome: .*/welcome: 'Sample Developer made this change'/"
[samdev@stand control]$ echo "Example.com Puppet Control Repository" >README
[samdev@stand control]$ git add hieradata/hosts/client.yaml README 
[samdev@stand control]$ git commit -m "Sample Developer changing welcome"
[production 49b7367] Sample Developer changing welcome
 2 files changed, 2 insertions(+), 1 deletion(-)
 create mode 100644 README
[samdev@stand control]$ git push origin production
Counting objects: 10, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 725 bytes | 0 bytes/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To /var/lib/git/control.git/
 74d2ff5..49b7367 production -> production

We've updated our production branch. Our changes were automatically propagated to the Puppet environments directory. Now, we can run Puppet on a client (in the production environment) to see the changes, as shown:

[root@client ~]# puppet agent -t
Notice: /Stage[main]/Base/File[/etc/motd]/content: 
--- /etc/motd 2015-10-01 23:59:39.289172885 -0700
+++ /tmp/puppet-file20151002-12893-1pbrr8a 2015-10-02 00:01:24.552178442 -0700
@@ -1,3 +1,3 @@
-Production Node: best behaviour.
+Sample Developer made this change
…
Notice: Applied catalog in 0.95 seconds

Now, any user we add to the pupdevs group will be able to update our Puppet code and have it pushed to any branch. If we look in /etc/puppetlabs/code/environments, we can see that the owner of the files is also the Puppet user due to the use of sudo, as shown here:

[samdev@stand ~]$ ls -l /etc/puppetlabs/code/environments
total 12
drwxr-xr-x. 6 root root 86 Sep 26 19:46 master
drwxr-xr-x. 6 puppet puppet 4096 Sep 26 22:02 production
drwxr-xr-x. 6 root root 86 Sep 26 19:46 production.orig
drwxr-xr-x. 6 puppet puppet 4096 Sep 26 21:42 puppet_sync
drwxr-xr-x. 6 puppet puppet 4096 Sep 26 21:47 quiet
drwxr-xr-x. 6 puppet puppet 86 Sep 26 20:48 thomas

Not playing nice with others via Git hooks

Our configuration at this point gives all users in the pupdevs group the ability to push changes to all branches. A usual complaint about Git is that it lacks a good system of access control. Using filesystem ACLs, it is possible to allow only certain users to push changes to specific branches. Another way to control commits is to use a pre-receive hook and verify if access will be granted before accepting the commit.

The pre-receive hook receives the same information as the post-receive hook. The hook runs as the user performing the commit so that we can use that information to block a user from committing to a branch or even doing certain types of commits. Merges, for instance, can be denied. To illustrate how this works, we'll create a new user called newbie and add them to the pupdevs group, using the following commands:

[root@stand ~]# useradd -g pupdevs -c "Rookie Developer" newbie
[root@stand ~]# sudo -iu newbie

We'll have newbie check our production code, make a commit, and then push the change to production, using the following commands:

[newbie@stand ~]$ git clone /var/lib/git/control.git
Cloning into 'control'...
done.
[newbie@stand ~]$ cd control
[newbie@stand control]$ git config --global user.name "Newbie"
[newbie@stand control]$ git config --global user.email "newbie@example.com"
[newbie@stand control]$ git checkout production
Branch production set up to track remote branch production from origin.
Switched to a new branch 'production'
[newbie@stand control]$ echo Rookie mistake >README
[newbie@stand control]$ git add README
[newbie@stand control]$ git commit -m "Rookie happens"
[production 23e0605] Rookie happens
 1 file changed, 1 insertion(+), 2 deletions(-)

Our rookie managed to wipe out the README file in the production branch. If this was an important file, then the deletion may have caused problems. It would be better if the rookie couldn't make changes to production. Note that this change hasn't been pushed up to the origin yet; it's only a local change.

We'll create a pre-receive hook that only allows certain users to commit to the production branch. Again, we'll use bash for simplicity. We will start by defining who will be allowed to commit and we are interested in protecting which branch, as shown in the following snippet:

#!/bin/bash

ALLOWED_USERS="samdev git root"
PROTECTED_BRANCH="production"

We will then use whoami to determine who has run the script (the developer who performed the commit), as follows:

user=$(whoami)

Now, just like we did in post-receive, we'll parse out the branch name and exit the script if we cannot determine the branch name, as shown in the following code:

read oldrev newrev refname
branch=${refname#*\/*\/}
if [ -z $branch ]; then
 echo "ERROR: Branch undefined"
 exit 10
fi

We compare the $branch variable against our protected branch and exit cleanly if this isn't a branch we are protecting, as shown in the following code. Exiting with an exit code of 0 informs Git that the commit should proceed:

if [ "$branch" != "$PROTECTED_BRANCH" ]; then
  # branch not protected, exit cleanly
  exit 0
fi

If we make it to this point in the script, we are on the protected branch and the $user variable has our username. So, we will just loop through the $ALLOWED_USERS variable looking for a user who is allowed to commit to the protected branch. If we find a match, we will exit cleanly, as shown in the following code:

for allowed in $ALLOWED_USERS
do
  if [ "$user" == "$allowed" ]; then
    # user allowed, exit cleanly
    echo "$PROTECTED_BRANCH change for $user"
    exit 0
  fi
done

If the user was not in the $ALLOWED_USERS variable, then their commit is denied and we exit with a non-zero exit code to inform Git that the commit should not be allowed, as shown in the following code:

# not an allowed user
echo "Error: Changes to $PROTECTED_BRANCH must be made by $ALLOWED_USERS"
exit 10

Save this file with the name pre-receive in /var/lib/git/puppet.git/hooks/ and then change the ownership to git. Make it executable using the following commands:

[root@stand ~]# chmod 755 /var/lib/git/control.git/hooks/pre-receive
[root@stand ~]# chown git:git /var/lib/git/control.git/hooks/pre-receive 

Now, we'll go back and make a simple change to the repository as root. It is important to always get in the habit of running git fetch and git pull origin <branch> when you start working on a branch. You need to do this to ensure that you have the latest version of the branch from your origin:

[root@stand ~]# sudo -iu samdev
[samdev@stand ~]$ pwd
/home/samdev
[samdev@stand ~]$ ls
control
[samdev@stand ~]$ cd control
[samdev@stand control]$ git branch
 master
* production
[samdev@stand control]$ git fetch
[samdev@stand control]$ git pull origin production
From /var/lib/git/control
 * branch production -> FETCH_HEAD
Already up-to-date.
[samdev@stand control]$ echo root >>README
[samdev@stand control]$ git add README
[samdev@stand control]$ git commit -m README
[production cd1be0f] README
 1 file changed, 1 insertion(+)

Now, with the simple changes made (we appended our username to the README file), we can push the change to the origin using the following command:

[samdev@stand control]$ git push origin production
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 324 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To /var/lib/git/control.git/
 b387c00..cd1be0f production -> production

As expected, there are no errors and the README file is updated in the production branch by our post-receive hook. Now, we will attempt a similar change, as the newbie user. We haven't pushed our earlier change, so we'll try to push the change now, but first we have to merge the changes that samdev made by using git pull, as shown here:

[newbie@stand control]$ git pull origin production
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /var/lib/git/control
 * branch production -> FETCH_HEAD
Auto-merging README
CONFLICT (content): Merge conflict in README
Automatic merge failed; fix conflicts and then commit the result.

Our newbie user has wiped out the README file. They meant to append it to the file using two less than (>>) signs but instead used a single less than (>) sign and clobbered the file. Now, newbie needs to resolve the problems with the README file before they can attempt to push the change to production, as shown here:

[newbie@stand control]$ git add README
[newbie@stand control]$ git commit -m "fixing README"
[production 4ab787c] fixing README

Now newbie will attempt to push their changes up to the origin, as shown in the following example:

[newbie@stand control]$ git push origin production
Counting objects: 5, done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 324 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: ERROR: Changes to production must be made by samdev git root
To /var/lib/git/control.git
 ! [remote rejected] production -> production (pre-receive hook declined)
error: failed to push some refs to '/var/lib/git/control.git'

We see the commit beginning—the changes from the local production branch in newbie are sent to the origin. However, before working with the changes, Git runs the pre-receive hook and denies the commit. So, from the origin's perspective, the commit never took place. The commit only exists in the newbie user's directory. If the newbie user wishes this change to be propagated, he'll need to contact either samdev, git, or root.