Day 3 - 14 Tips for Git Giddiness in 2014

Written By: J. Paul Reed (@soberbuildeng)
Edited By: Adam Compton (@comptona)

Whether you're a developer or a systems administrator, it's inconceivable you haven't heard of Git by now.

Statements like "You can use any version control system you want, just as long as it starts with a 'g' and ends with an 'it'" illustrate the exploding popularity and ubiquity of the version control system Linux's Linus developed in a weekend.

Despite this, organizations making the transition from the previous generation of version control systems (and sometimes no version control at all) face many challenges.

Here are 14 tips—ten for the individual Git user, and four for organizations moving to Git—to make 2014 your best year using Git yet!

1. Be an "Explicit Git User"

   A developer, tongue in cheek, once told me: "Git operates on the principle of most-surprise." That is often true, especially when Git's developers change the semantics of core commands every major release or so.

   One way to help reduce mistakes is to be (what my colleague Pete Cheslock dubbed) an "Explicit Git User." In other words, move away from using Git's default command arguments in daily workflows and always specify what you intend.

   Examples include:

   • Specify the repository and refspec arguments to git push (especially when using --force).
   • Favor git fetch (with the repository and refspec arguments!) over git pull; pull is a fetch, followed by a merge, so doing it this way separates the act of getting upstream content from integrating it into your local repository, which allows you to explicitly specify how you want to do that.

   You'll be glad you stopped relying on default command arguments, especially when they change in a major release.

2. Learn How To Pull In Upstream Changes Without Creating a Merge Commit

   Pretty much every Git repository contains comments like Merge branch 'master' of repository_name into master. These are merge commits, automatically generated by Git when you've made changes on your local branch and you pull in upstream changes. Git treats these as divergent histories that must be merged together using—drumroll—a merge commit! Previous version control systems would add conflict markers in the file and make you resolve them, but without operating against the repository.

   The issue here is clarity; this commit adds zero value and clutters up the repository's history. It also makes graphical views of the repository history more confusing.

   A common way to avoid it: use the --rebase option with git pull. The downside is this requires a clean working directory, but that's a good practice when pulling anyway. It also assumes the simplistic usecase where you have made local commits, you want to integrate upstream commits that occurred in the remote repository, and you want behavior that is similar to the CVS or Subversion-workflow you may be used to.

   You can make this behavior the default by adding the following to your .gitconfig:[branch]
      autosetuprebase = always

3. Learn to Rebase Your Commits to Tell a Story

   The next two tips are around Git's rebase functionality, which can be daunting and dangerous, but also powerful. Restricting yourself in the ways you use rebasing will help keep you clear of situations that are the makings of a Git horror story. One of those uses is rewriting history to tell a meaningful story to future developers (including, possibly, yourself).

   This method is recommended on feature branches, where you may have done a bunch of work, but the units of that work and the commit messages associated with them are less than useful. When I have this problem, I use the oldest-ancestor alias (see tip #7) to rewrite the commits to tell the story of my feature branch. On that branch, I run: git rebase -i $(git oldest-ancestor). I can then reorder and squash commits into something useful. Being an explicit Git user, when squashing commits, I reorder them first, complete the rebase, and then perform a separate rebase to squash the commits into a meaningful unit of work. This allows me to --abort the rebase if I get into an unexpectedly complex situation.

   When complete, my feature branch is ready to push upstream. If I've pushed it before—and this is where you see most of the warnings telling you not to use rebase at all, evar!—it will require a --force push; any others who've worked on my branch will need to know that I've rewritten history. Because of that, when working collaboratively on a feature branch, I will wait until the feature is ready for final review or to be merged before doing this.

4. Learn to Rebase Your Branch to Clarify/Constrain the Scope of Your Changes (And Keep the Build Green)

   Once your feature branch is ready to merge back to the parent code-stream, it's always been a best practice to merge in all the changes from the upstream branch into your local branch first. This reduces conflicts when you merge back, and if you've been off on a branch for a long time, allows you to make sure the code still builds and passes unit tests.

   In Git, this can be done by merging the parent's code down into your feature branch before merging back. But there are advantages to achieving the same result with a rebase. (The biggest, for me, is that the operation takes place by "replaying" all of my commits on top of the current code; this can often make resolving conflicts simpler, since I can do it in smaller chunks, as opposed to all at once.)

   To do so, make sure you have the latest copy of the upstream branch. Then, on your feature branch, use git rebase UPSTREAM_BRANCH. This will, in effect, pull in all the upstream changes, stopping for you to resolve any conflicts as your feature branch commits are replayed.

   (If you're using Git Flow, this is even easier! Use git flow feature rebase.)

5. Do Not Store Artifacts in Git (Even Lil' Ones)

   Previous version control systems handled binary artifacts well enough that it was common practice to dump JARs, shared-objects, and other dependent artifacts used in the software build process (or shipped in the product) into source control, right alongside the code. While there's nothing fundamentally wrong with using source control to track binaries—Pixar famously versions entire movies using Perforce—the industry is moving away from the practice, in lieu of proper artifact management, using tools specifically designed for the task.

   Because Git (currently) requires users to grab an entire copy of the repository to work, storing binary artifacts, even small ones, can balloon the size of the repository over time, causing times for clones and other costly Git operations to increase significantly. git rm'ing these binaries does not fix the problem, unless you undertake repository surgery (a service I am hired surprisingly often for).

   In the future, Git may address these issues, but proper artifact management provides a number of other benefits and is the future: resist the temptation to commit these types of files and put your binaries, even the small ones, under real artifact management.

6. Submodules are Horrible... Except When They're Not

   Git's submodule feature has a largely bad reputation and it's not entirely undeserved. A stand-in for Subversion's externals, the feature breaks many standard Git conventions and often feels not-fully baked. (See: removing a submodule.)

   Having said that, submodules can serve a useful purpose: stitching multiple, internal repositories within a singular organization. One of the standard patterns spurred by Git is emphasis on smaller, narrowly scoped repositories, often at the library or module level. Repositories built of submodules, used to build the complete product can reduce a lot of the complexity of trying to stitch these repositories together yourself. But, they should point to repositories your organization manages; pointing to random content on Github is a recipe for future sadness.

7. Learn [to Love] Git Aliases

   One of Git's most useful features is the ability to add commands to its lexicon, via the alias feature. Learn to love this feature: it allows you to create shortcut commands for complex operations.

   For instance, in my work I often need to find the common ancestor of a feature branch and master; to do this, I found an alias I called oldest-ancestor; it runs

   bash -c 'diff -u <(git rev-list --first-parent "${1:-master}") <(git rev-list --first-parent "${2:-HEAD}") | sed -ne \"s/^ //p\" | head -1'

   (Taken from StackExchange).

   Problem solved! Even commonly used simple operations can be aliased. One of my other favorites: mapping reset HEAD -- to unstage.

   Check out git-extras for more; I stole many of my favorites from Wil Moore's dotfiles.

8. Take Time to Learn Git's Primitives

   If your job requires you to use Git more than three times per week, do yourself a favor and take time to learn Git's primitive objects: the blob, tree, commit, and tag objects.

   A firm understanding of how these objects relate to each other and how they're used to construct histories of source code development lines will help you understand why Git behaves differently than previous-generation source control systems.

9. Be Careful What You Read On the Internet

   When running into brick walls with Git, engineers (including myself) invariably run to Google to find an answer. Take caution with those search results.

   In my experience, when untangling any reasonably complex Git problem (merges especially!), a search for the solution will return 20% of answers that look promising, but upon further reading, the language I used ("branch", "parent", "diff chunk", etc.) in my search is being used in a different context in the answer, and thus isn't helpful. About 40% of answers would be useful, if they applied to my team's repository structure and expectations. 10% of them will be just plain incorrect. This leaves me with just 30% of answers that are actually useful in the environment in which I'm using Git.

10. Take it Easy on Each Other (And Yourself!)

    New Git users: in my training sessions, I find it important to let new users know: if Git seems difficult and confusing, it is. Much has been written about Git's cognitive inconsistencies.

    It's one of the few software development tools that makes it too easy to destroy your own (and others') content! For all the bandying we do in the software industry about the importance of user interface consistency, usability, and polish, Git came roaring out of the gate without an emphasis on any of these. Any usability expert will tell you this it's the tool's fault, not the user's. Git is continuing to evolve and improve, but if you ever feel frustrated or confused by Git, you are not alone.

    Seasoned Git users: I know developers who love to trade stories on how Git has screwed them. They seem to see it as a badge of honor that they blew their (and sometimes their team's) foot off with the tool. Sometimes they've been able to recover; sometimes not-so-much. While regaling these warstories, some label those who find Git confusing or counterintuitive to be incompetent, "not a craftsperson," or just plain stupid.

    If this describes you: stop it. There are many capable, competent developers, QA engineers, techpub writers, artists, and others who take great care in their work, but totally do not care about the version control tool they use; they just want to get their job done. The attitude that everyone must obsess over the intricacies of a single counterintuitive tool required to get work done does nothing to help Git's adoption. Plus, you come off as an unempathetic jerk.

For organizations adapting their development environment and release processes to Git:

1. Your Team Needs a Git "Language Lawyer"

   The concept of a "language lawyer" came out of the necessity to have someone on the team who knew all the little crevices of complicated languages, like C++, where the standard was continually evolving, compilers supported different features, and platform oddities made development complex and error prone. Given Git's constant evolution, differing versions (especially on long-term support distributions), and platform oddities (*cough*Windows*cough*), current-day Git is at least as complicated as C++.

   Your team should have a "Git language lawyer." That person should follow the Git releases, read release notes and possibly even follow the mailing lists. They should track the state of the art of Git's supporting tools.

   This provides a resource for your organization to help frame discussions optimizing your team's Git workflows and be a source of deep technical knowledge when team members, old and new, run into problems.

2. Make Time Early On to Discuss and Decide on Workflows

   An anti-pattern I commonly see is a subset of developers dragging the rest of the team into using Git and the repository quickly becoming a "bed-headed, tangled mess" of content, branches, and commits.

   This is due to Git supporting so many workflows. Failing to discuss and agree upon a workflow is a recipe for disaster. Some issues your team should come to agreement on include:

   • Are you going to use Git Flow? (This may sound like a stupid question, but most organizations I work with start using Git Flow and pretty quickly evolve its workflow into something else. It's a fine tool, but, for instance, Github doesn't use it, so it's a conversation worth having.)
   • For integrating content, is your team going to use a repository forking model or a branch/merge model?
   • If you're relying on branching, what name-space standards will you use? How will you clean up branches?
   • When merging, will you use merge commits or squash feature-branch merges to a single commit? What about feature branches that contain (or squash to) a single commit?

   These are just a few of the questions your team's "Git language lawyer" will be able to help discover what's best for your team.

3. Use a Git Repository That Provides Guardrails

   Certain codelines—develop, master, release/hotfix branches—are intrinsically more important than feature and personal branches. They should be treated as such and have special policies around structure and naming (so they're easy for everyone to find), who can create them, how content flows into them, and whether or not their history can be rewritten.

   There are a number of repository management tools to help with this: gitolite, Gerrit, Gitlab, Stash. Use one of them.

   (Readers may note: hosted-Github isn't listed; it doesn't provide the ability to disable force-pushes. Github Enterprise has a form of the feature. Bitbucket, also a hosted-Git provider, includes this functionality.)

4. Understand the 'D' in DVCS

The D in DVCS is commonly expanded as "decentralized" or "distributed." I find neither to be accurate.

In the real world in which we develop code, Github and company "repositories of record" provide obvious counterexamples to the "decentralized" descriptor. And while Git repositories technically are distributed, the term had a specific (and different) meaning than commonly used in computer science and if your DVCS implements fully-functional shallow clones, any claimed benefits of "decentralized" or "distributed" disappear.

What do I think the 'D' stands for? Disconnected. Git allows you to do all of the operations we generally care about offline, including creating personal codeline histories which may not be important to the larger group.

So why is the distinction even worth making? Many claim Git is better because everyone has a backup and so nothing will ever be lost. This is a precarious argument. (And, as Git evolves, it may not even be true!)

More than that, it gives the impression that our (centralized!) repositories of record—and all of the services around them that produce our builds—aren't critical infrastructure, and are replaceable by a random development laptop with no impact on the organization's ability to develop and ship software. I know of few, if any, environments where that is practically true, and so that impression is disingenuous.

Upshot: Git is great... but it doesn't mean we can start ignoring our (centralized) repositories of record and their accompanying services.

It would surprise me little if some of these tips and tricks generated discussion and even disagreement. I've found version control and code-line management of interest for over fifteen years now, and while I have many feelings about Git, I can't argue that it's done more in the past six years to make our industry aware of and care about these issues than release engineering has for... well, probably ever.

Hopefully at least a few of these will make you and your team's Git usage more productive, easier, and more fun in 2014!

Happy Holidays to all! (And to all, a clean merge!)

Day 20 - Github Gist

This article was written by Phil Hollenback (@philiph)

I assume everyone is familiar with the idea of a pastebin - a website for sharing text fragments with an emphasis on code fragments. Pastebins have been around since 2002, according to Wikipedia. They're an incredibly useful resource for sharing textual data and are something we, as sysadmins, need to do on an almost continual basis. However, there are several problems with some existing pastebin implementations:

• lack of command-line integration
• no version control
• no privacy settings

I recently came across a new (to me, anyway) alternative to the traditional pastebin: github gists. The following is a description of how gists work and how they differ from traditional pastebin clippings. I'll also describe some ways you can collaboratively edit gists with one or more people.

What's a Gist?

A gist is simply a text clipping with optional syntax highlighting, the same as you would find in any other pastebin. You can go look at some right now to get the idea.

So, why would you want to use this instead of a traditional pastebin? Pick a file (say, a perl script) and hold on to your socks:

$ gist test.pl
https://gist.github.com/737292

That's it! You just created a syntax-highlighted text clipping anyone on the internet can view.

Unfortunately, there is some up-front work to get this all set up. You can't just post anonymous gists to github.com like you can with some pastebins. I'll detail that setup info below. And, here's the really exciting part: there's an emacs script to automate all of this!

Initial Setup

As I mentioned, you have to have a github account to create gists (or to comment on existing gists). The good news is that's free and just takes a moment to set up. Once you have your account created, go to your account page and click on Account Admin. You will find your API token on this page. Take a moment to copy that down as you will need it to set up your command-line gist client.

You should also click on SSH Public Keys in the account settings page and upload your ssh public key. you're going to need this to edit gists shortly. Did I mention that gists are version controlled with git?

I'm assuming you have the git client installed for your linux or mac box already, if you don't have that go get it now as you will be using git a lot for all this. One thing that was not clear to me, initially, was how to set up your local git config for gist access. This is controlled by your ~/.gitconfig file, which will look something like this:

[user]
    name = <your name>
    email = <your email>
[github]
    user = <your github username>
    token = <your api token>

You can actually read and write from this file via git config on the command-line, like this:

git config --global github.user username
git config --global github.token blah

the gist command-line and emacs clients use this mechanism to read from your ~/.gitconfig.

Once you have this all configured, download and install the gist command-line client. I used the gem install gist install method which worked just fine. Verify your setup works by creating a gist, as above.

Now What?

At this point you've got a simple, command-line pastebin client which is a pretty useful thing. For example, suppose you want to demonstrate some code to someone on twitter. Instead of mucking with pasting your code into a regular pastebin website, feed your script directly to the commandcommand-st client. Right here you've got an url you can paste into your tweet. If the viewer of your gist goes through the small hoop of creating their own github account, they can leave comments about your gist too.

Don't worry, though - there's lots of other ways to use gists. For example, there's an emacs interface to gists!

Emacs Mode

The emacs interface for gists is gist.el. It supports mostly the same options as the regular command-line client with a few twists. For example, you can use gist-list to select from and open one of you public gists.

I've been using the emacs gist interface quite heavily to share gists with others. For example, if someone tells me 'check out my gist 741773', I can just hit <Meta>-x gist-fetch<RET>741773 to pop that gist right into an emacs buffer.

Unfortunately the emacs mode suffers from some glitches due to problems with ssl access in emacs. I had to hack on gist.el somewhat myself to get it working with Aquamacs on my mac. Thus while I'm pretty excited about gist.el, it's not really ready for primetime.

Markdown

In addition to plain text and programming language markup, gists also support Markdown. Actually they support Github Flavored Markdown, which includes a few small tweaks of the original Markdown language.

I assume most readers are familiar with Markdown, but if you aren't, it's a simple way to write structured ASCII text that can be easily turned into HTML or other document formats. The beauty of Markdown is it's completely readable as straight ASCII as well as HTML.

To force interpretation of your gists as markdown, use the .md file extension on the file you upload to create a gist. When you view your gist on github you will see it all dressed up with headers and bullets and everything.

Private Gists

By default, gists are public. This is the standard convention for pastebins - everyone can see what you post. This usually works just fine. However, if you want to protect your information, you can create a private gist. There are two differences between private and public gists:

1. public gists show up on the gist main page.
2. public gists use easily guessable sequence numbers, private ones use hash identifiers.

For #2, public gists have incremented IDs like 73962 while private gists use hashes like d17b2652f7896c795723. In practice, this makes it difficult to guess the ID (and URL) of a private gist. Note there is no real security here in the form of access controls - if someone obtains your private gist ID, they can access it. Thus, don't use private gists for passwords or other sensitive information.

However, private gists work just great for information you want to protect but isn't super critical. I would feel fine pasting config files as private gists, for example.

With the gist command-line client, use the -p switch to create a private gist, or use git-config to set your default gist posting mechanism to private. If you are going to use gist as a pastebin to share system information such as config files and scripts, you should probably use private gists by default. The emacs interface supports similar functionality.

Using git for Gists

As I mentioned earlier, gists are stored in a git repository on github. That means you can use them to collaborate on a documentation project. Here's the workflow:

1. Create a gist through web interface, cli, etc.
2. Give your friend Joe the url to that gist on github.
3. Joe visits that url and clicks 'fork' to get his own repository
4. Joe makes edits to his forked copy of your gist
5. Joe commits his changes to his repo, gives you his private clone url
6. cd into your local repository on your computer
7. Merge Joe's changes into yours with git pull <Joe's private clone url> master
8. commit your merged changes to your repo with git commit -a and git push

That's it! You're now collaborating with someone on a shared script, config file, markdown document, or whatever. Also, since this is a distributed version control service, your collaborator can always fork your gist and start modifying their own copy.

Remember that the gist web interface supports comments, so if you don't want to do a full collaboration with someone, they can always just leave gist comments instead (although commenters do need github accounts). Note that comments don't seem to be exposed in the git repository, unfortunately.

I've focused on single-file gists in this description, but note that gists can contain multiple files. You can add additional files via the web interface or by creating additional files in your local git repository. This is another important difference from traditional pastebins.

Why Should I Care About This?

As a sysadmin, I'm excited about this tool for a number of reasons. Mainly, I have a need to share scripts, config files, and the like with other sysadmins. Currently, that involves emails or traditional cut-n-paste pastebins. Neither of these solutions are very satisfactory.

What I want is a way to create public and private pastebins from the command-line and share those via a URL. I also want a way to mark up and collaborate on text files. Finally, it would be pretty handy if those files were automatically version-controlled and stored somewhere out on the internet for me.

Oh, also, that tool better not cost me anything, because I'm cheap and/or poor. Hey look, github gists support all those features! That's why I've started using gists instead of the old pastebins. The command-line and emacs integration are the real power of gists. Gists are a direct interface between your terminal and the cloud, all wrapped up in a sysadmin-friendly package.

Further Reading

• A good blog writeup about gists.
• The gist api.
• The gist homepage.