Console Login

Private Git Hosting with Gitolite on CentOS 6

The problem

A team that keeps proprietary code on a public hosting service, or on a server far from its developers, tends to run into three things:

  • Slow pushes and fetches. Every git push or git fetch over SSH starts with a connection handshake and several protocol round trips. A distant server makes each of them noticeably slower.
  • Unclear control over the data. Source code, test fixtures and database dumps checked into a repository can contain personal data. Where the server sits, and who operates it, then becomes a compliance question. In the EU/EEA of 2012 the relevant frame is Directive 95/46/EC and, in Norway, the Personopplysningsloven.
  • Clumsy access management. Plain Git over SSH has no notion of per-repository permissions. Each developer needs a shell account, or everyone shares one account and one set of rights.

This article builds a small, key-only Git server on CentOS 6 using Gitolite 2.x. It also covers what to check when it is slow, and how to back it up.

Why it happens

Latency

Network round-trip time grows with distance and the number of hops, and no server software can remove it. Git over SSH needs a TCP connection, an SSH key exchange, authentication and then a few request/response exchanges. A server on the same continent, or in the same country, as the developers shortens each of those steps.

Check the real figure from a workstation instead of assuming it:

ping -c 10 git.example.com
ssh -v [email protected] true

Disk I/O

Server-side Git work (packing objects for a clone, git gc, repacking after large pushes) reads and writes many files. When several developers fetch a large repository at once, a slow or shared disk shows up as high I/O wait. In that state even the SSH daemon responds slowly. Look at it with iostat -x 5 from the sysstat package, or at the wa column in top. Faster storage and a scheduled git gc both help. No fixed speed-up figure applies to every setup.

Access control

Git itself delegates authentication and authorisation to the transport. Over SSH that means operating-system users and file permissions. A tool such as Gitolite sits in front of Git. Every developer connects as the same operating-system user, and the SSH key decides who they are and what they may do.

Source of truth

The Gitolite project documents its requirements and its setup model in its own repository and manual:

  • The Gitolite repository and README confirms the server requirements: a Unix system with sh, Git 1.6.6 or later, Perl 5.8.8 or later and OpenSSH 5.0 or later. It also confirms that the hosting user is dedicated and has no existing pubkey access. Users, repositories and rules are changed by pushing to the special gitolite-admin repository, never by hand on the server. That repository contains the conf and keydir directories.
  • The install and setup page confirms that the administrator's public key is copied to the server under the admin's own name. A key file called alice.pub creates the user alice. It also confirms the split between the install step (put the code in place) and the setup step (create the files, directories and the admin repository).
  • The step-by-step setup page confirms two things. The administrator's key must not be added to the hosting user's authorized_keys by hand. A working setup can be checked from the workstation with git ls-remote against gitolite-admin.

Those pages describe the current Gitolite generation, which uses gitolite setup. The Gitolite 2.x series that was current in early 2012 follows the same model but uses the gl-system-install and gl-setup commands. Check the README of the release tag you check out. CentOS 6 ships Git 1.7.1, Perl 5.10.1 and OpenSSH 5.3, which satisfy the minimum versions above.

Fix options

OptionStrengthsWeaknesses
Bare repositories with one shell account per developerNothing extra to install; standard OpenSSH and file permissionsAccounts to manage per person; per-repository rights depend on Unix groups; developers get a shell
One shared git account with many keys in authorized_keysSimple; no extra softwareNo per-user or per-repository rules; every key can reach every repository; removing a person means editing the file by hand
Gitolite on a dedicated userPer-repository and per-user rules in a text file; no shell for developers; small Perl-only footprint; configuration lives in a Git repositoryCommand-line administration only; learning the rule syntax
Web-based suites (Gitorious, GitLab)Web UI, merge requests, user self-serviceA Ruby application stack to install, patch and back up; much more to secure
A hosted serviceNo server to runCode leaves your control; latency and jurisdiction depend on the provider

Why this fix

Gitolite is the best fit when the goal is a private server with real access rules and a small attack surface. Developers never receive a shell. The forced-command mechanism in authorized_keys limits each key to Git operations. Rules are plain text, kept in a Git repository, so every permission change is committed, reviewable and reversible. The software needs only Git, Perl and OpenSSH, all of which are in the CentOS 6 base repositories. The server still holds ordinary bare repositories, so migrating away later means copying directories.

The shared-account option is the weakest for more than a handful of people, because it cannot separate repositories or users. A web suite is the right choice only when the team needs a browser interface and can maintain the extra stack.

Step by step

These commands are given as instructions. This article does not reproduce program output. Success is described in words after each step.

1. Prepare the server (as root)

yum -y update
yum -y install git perl openssh-clients
useradd -m git
passwd -l git

The git account is created without a usable password, so the passwd -l step is mostly a safeguard. The account is reachable only through su from root until Gitolite installs the first key. If key logins are later refused for the locked account, check that UsePAM yes is set in /etc/ssh/sshd_config. That is the CentOS default.

2. Create the admin key (on the workstation)

ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa_git_admin
scp ~/.ssh/id_rsa_git_admin.pub root@<server-ip>:/tmp/admin.pub

The file name decides the Gitolite user name, so admin.pub creates the user admin. The key must be readable by the git user. The default file mode of scp is usually enough.

3. Install and set up Gitolite 2.x (on the server)

su - git
mkdir -p $HOME/bin
git clone https://github.com/sitaramc/gitolite
cd gitolite
git tag -l
git checkout <latest-v2-tag>
src/gl-system-install $HOME/bin $HOME/share/gitolite/conf $HOME/share/gitolite/hooks
export PATH=$HOME/bin:$PATH
export EDITOR=vi
gl-setup /tmp/admin.pub

Replace <latest-v2-tag> with the newest 2.x tag listed by git tag -l. gl-setup opens ~/.gitolite.rc in the editor. The defaults are acceptable, so save and quit. A successful setup creates ~/repositories/gitolite-admin.git and writes the forced-command entry for the admin key into ~/.ssh/authorized_keys. Add export PATH=$HOME/bin:$PATH to ~/.bash_profile if the shell is used again later.

4. Point the workstation at the server

Host git-server
    HostName <server-ip>
    User git
    IdentityFile ~/.ssh/id_rsa_git_admin
    IdentitiesOnly yes

Put this block in ~/.ssh/config. Then check access and clone the admin repository:

git ls-remote git-server:gitolite-admin
git clone git-server:gitolite-admin

If the key is accepted, the first command lists references and the second creates a gitolite-admin directory containing conf and keydir.

5. Add a developer and a repository

cp /path/to/lars.pub gitolite-admin/keydir/lars.pub

Add this to conf/gitolite.conf:

repo    project-viking
        RW+     =   admin
        RW      =   lars

Then commit and push:

cd gitolite-admin
git add .
git commit -m "Add repo project-viking and user lars"
git push

The server-side hook compiles the rules and creates the empty bare repository. Lars can then run git clone git@<server-ip>:project-viking with his own key. RW+ permits non-fast-forward pushes and ref deletion. RW does not, which protects history from accidental rewriting.

6. Back up

Save the following as /root/backup_git.sh:

#!/bin/bash
set -e
DATE=$(date +%Y-%m-%d)
TAR_FILE="/backup/git-backup-$DATE.tar.gz"
tar -czf "$TAR_FILE" -C /home/git .
scp "$TAR_FILE" user@backup-server:/storage/

Archiving the whole home directory captures repositories, .gitolite.rc and the SSH key entries. Make the script executable, create the target directory, and schedule it:

mkdir -p /backup
chmod 700 /root/backup_git.sh
crontab -e

Add this line to the crontab:

30 2 * * * /root/backup_git.sh

The scp step needs a passwordless key for the root user on the backup host. Test a restore on another machine before relying on the archive.

Limits and alternatives

  • Consistency. A plain tar of live repositories can capture a repository mid-push or mid-git gc. For busy servers, take a filesystem or LVM snapshot first, or schedule the backup in a quiet window. A git clone --mirror of each repository is a slower alternative that always produces a valid copy.
  • Version drift. The setup shown here belongs to the Gitolite 2.x generation. A later major version changes the install and setup commands, and the migration path should be read from the project documentation before upgrading.
  • No web interface. There is no browser, no issue tracker and no merge request workflow. Pair it with a read-only web viewer such as gitweb if one is needed, or choose a web suite.
  • Latency. Hosting near the developers shortens round trips. It does not speed up the work Git does on the developer's own machine.
  • Compliance. Server location is only one input. Retention, access logging, encryption of backups and the legal basis for storing personal data still have to be handled. A repository should not contain personal data in the first place.
  • Passwords and keys. Keep password logins disabled for the whole server with PasswordAuthentication no in sshd_config once the admin key works. RSA keys of 2048 bits were the accepted baseline in 2012, and 4096 bits is a reasonable option for new keys.