Prologue

After many years of research, I've finally come to an important conclusion:

Writing completion scripts sucks.

In consideration of this, I have organized these sections in order of greatest importance to the topic (excepting this first section), so the devoted reader can stop reading this article at whatever point you lose interest.

Like me, you probably began using ZSH as a result of increasingly rabid advocacy stirring your soul into dissatisfaction with the inflexible completion system of other shells (c.f bash).

At that time, ZSH's comprehensive documentation was the best source of information on the topic. ZSH's man pages are written in encyclopedic reference style while also containing many digressions, anecdotes and 'end notes' (some of which have footnotes themselves) in a medium generally lacking both. This provides the manpages a unique distinction for a technical document - that of supporting range of interpretations and the creation of a robust semi-scholarly enterprise of interpreting the "true" meaning of the documentation.

Since then, the ZSH guide (a separate document) has been updated with details of the new completion system. Although it contains some of the structural details required for quickly writing correct completion scripts, completion details common to many utilities aren't included.

In response, I've tried to compile some principles, mailing-list wisdom and other disambiguation for this hairy topic.

The structure of standard argument completion.

ZSH's completion system is contextual, sensitive to the point at which it is invoked, consequently a common pattern is to progressively backtrack to current point, determining what options and what subcommands have been set thus far, pruning the selection of completion options.

If you've never written completion scripts before, you might not yet have been exposed to the ultimate structure of option parsing that completion scripts (generally) successfully distill. It will be a profitable use of your time to get a high-level view of completion script's structure unintentionally defy it and spend time working against, rather than with the grain of the system.

  1. An entry point (e.g git) will processes the command line string.
  2. A function for each subcommand such as diff, typically wrapping _arguments
  3. One function for each type of object, referred to as a tag in ZSH nomenclature.
  4. In some cases, functions to handle special values, files, sockets, fifos.

The actual mechanics of achieving this typically mean popping off the head/first word of the line, feeding that word into a case statement, calling the appropriately specialized function for whatever completion option has been passed in.

Another important feature of ZSH's completion system is that there are many special utility functions, variables and options that make many common tasks easier, a great deal of writing completion scripts is to use these appropriately.

Git is a good command to analyse, not just because it is comprehensive, but also because it reflects many common option handling patterns; Git is run by a single command, git, followed by either flags or an argument giving the particular git command, again followed by options to that command. A high level view of what-must-be-completed-when in git demonstrates it's recursive nature immediately.

The structure of _arguments

The function _arguments has been described as having `the syntax from hell', but with the arguments already laid out in front of you it doesn't look so bad. The are three types of argument: options to _arguments itself, arguments saying how to handle options to the command (i.e. `p4 diff'), and arguments saying how to handle normal arguments to the command.

The following is an excerpt from the p4 diff completion function (in turn excerpted from the manual) and covers some important fundamentals of argument parsing.

(( $+functions[_git-diff] )) ||
_git-diff() {
      _arguments -s : \ 
          '-f[diff every file]' \ 
          '-t[include non-text files]' \ 
          '(-sd -se -sr)-sa[opened files, different or missing]' \ 
          '(-sa -se -sr)-sd[unopened files, missing]' \ 
          '(-sa -sd -sr)-se[unopened files, different]' \ 
          '(-sa -sd -se)-sr[opened files, same as depot]' \ 
          '-d-[select diff option]:diff option:' \ 
  '((b\:ignore\ blanks c\:context n\:RCS s\:summary' \ 
    'u\:unified w\:ignore\ all\ whitespace))' \ 
          "*::file:_perforce_files"
  }

Flags

The excerpt above only has one flag, and it's the most meaningful flag as well, but I won't waste an opportunity to briefly cover some other very handy flags:

-s conveys that single-letter options are allowed, i.e. they can be combined as in -ft, although this doesn't prevent you from acceping multiple word options either. -w is related; in combination with -s it means that the options can stack even if one of them itself takes an argument.

For example, tar -cf $FILE could be processed in this way, as the option after -f indicates the file we'd be processing (further options AFTER -f would be valid as well)

-S is completely unrelated, it indicates that the completion function shouldn't complete options after --, which is a common UNIX 'pattern' to indicate options have ended.

The optspec

The long strings of option specification that follows the flags to _arguments and a colon are known known as ~optspec~s or option specification.

Option Naming

_arguments broadly supports 7 different option specification varieties, all of which can be directly followed by a bracketed explanation string.

specification description
*optspec Here, optspec is one of the remaining forms below
-/+optname Plus or Minus the option
-optname- The first argument must be supplied here
-optname+ The first argument must be supplied with a +

Utility Functions

Creating a dummy first argument

The following is an extract of the iproute2 argument handling in _ip

local args
args=(
  # Command word
  /$'[^\0]#\0'/
  'l*ink:configure network device:$link_cmds' \
  'addrlabel:manage addrlabel:$addrlabel_cmds' \
  'a*ddr:manage protocol address:$addr_cmds' \
)
_regex_arguments _command

_pick_variant to add options depending upon the version of a program.

local arguments
# We supply a regex to _pick_variant, in this case checking gor the string `gnu`
if ! _pick_variant gnu=gnu unix --help; then
  arguments=('-g[This flag only works on gnu distributions of this binary]')
else
  arguments=('-a[Otherwise this flag is available]')
fi

Match an ambiguous clause with _guard

The _guard can break between two tags, dependent upon the regex; if this doesn't seem extraordinarily useful to you, you're not alone – In the body of existing ZSH completion scripts, _guard is typically used an the action for the specification pased into _arguments and similar functions.

The zshcompsys manpage itself describes behavior reminiscent of the completion behavior of fc(1) _guard

As an example, consider a command taking the options -n and -none, where -n must be followed by a numeric value in the same word.

zshcompsys(4)

The _fc completion demonstrates this here:

if [[ -n $state ]]; then
  zstyle -s ":completion:${curcontext}:" list-separator sep || sep=--
  if [[ -z ${line:#*=*} ]] && compset -P '*='; then
    _message -e replacements 'replacement'
  elif [[ -prefix [0-9] ]]; then
    events=( ${(0)"$(printf "%-${#HISTNO}.${#HISTNO}s $sep %s\0" "${(kv)history[@]}")"} )
    _wanted -2V events expl "$state_descr" compadd -M "B:0=" -ld events - \
        "${events[@]%% *}"
  elif [[ -prefix - ]]; then
    for num cmd in "${(kv@)history}"; do
      (( num=num - HISTNO ))
      events+=( "${(r.1+$#HISTNO.)num} $sep $cmd" )
    done
    _wanted -2V events expl "$state_descr" compadd -ld events - \
        "${events[@]%% *}"
  else
    _wanted events expl "$state_descr" compadd -S '' - \
        ${${history%%[=[:IFS:]]*}:#[0-9-]*} || _guard "[0-9]#" event
  fi
fi && ret=0

Examples

A statement about these examples should be made here

Delimited values with final option

A common scenario that occurs in commands such as libcap's capability manipulation toolchain, bintools and coreutils is the requirement to complete a list of arbitrary keywords, each with a unix-style (equal sign) option after each one.

An example of such a command is exemplified by setcap


% setcap cap_sys_rawio,cap_sys_pacct=
-- operator -- e -- effective i -- inheritable p -- permitted

You might initially look at the chmod completion, and this would get you far, however the completion script itself is quite long. The core of the unix options completion lies in the following.

list_terminator='*[=]' # Corresponds to `=` 
delimiter=',' # The character that delimits the list
options=("e:effective", "i:inheritable", "p:permitted") # Valid options
case $state in
  # compset -P checks if we've reached a user entering a $list_terminator
  if compset -P $list_terminator; then
    _describe -t options "options" options
  else # Otherwise complete from these list of items.
    _values -s $delimiter items 
      'foo[Description of foo]' \
      'bar[Description of bar]'
  fi
  ;;
esac

Operating system specific flags with $OSTYPE

local arguments
arguments=('-b[Base argument]')
# We might add additional arguments based on the operating system
if [[ "$OSTYPE" = (freebsd*|darwin*) ]]; then
  arguments+=('-m[OSX or FreeBSD Specific Flag]')
fi
if [[ $OSTYPE = solaris* ]]; then
  arguments+=('-s[Solaris specific flag]')
fi
if [[ $OSTYPE = linux* ]]; then
  arguments+=('-l[Linux specific flag]')
fi

Completion from a dynamic list

There are two ways to go about this. Both require that you create a function that calls compadd with the list of words you want completed.

typedef -a _tmux_words
_tmux_list() {
   compadd -a _tmux_words
}

Up to you to figure out how to populate the _tmux_words array. The function that eventually calls compadd can do as much other work as you like to decide whether to call compadd at all; see for example the _expand_alias function in the zsh distribution. 1

With that in place, you can do either:

  1. Create a key binding that invokes it, leaving normal completion alone.

    compdef -k _tmux_list complete-word ^XT

  2. Add a function to your "completer" style.

    zstyle ':completion:*' completer _complete _tmux_list _correct

Don't use the above zstyle literally; find the one you are presently using and insert _tmux_list at the point where you want those words tried as possible completions.

Caching variables during completion

Depending on whether you mean all completions for the current command line or just all repetitions of completion for the same word (e.g., cycling through a menu) there may be different approaches to this. Within completion on a single word, you can look at the _oldlist completer for an example.

Based on your additional explanation, though, I suspect that's not what you're after, but the basic idea is still the same: Create a function which you reference at the beginning of the completer zstyle. That function tests (somehow) whether the cached state needs to be refreshed.

Bart Schaefer describes a crude procedure to cache the value value of $HISTNO and then reload the cache if it has changed.

_xrcache() {
  if (( $_xr_HISTNO != $HISTNO ))
  then
    _xr_HISTNO=$HISTNO
    _xr_output=$(xrandr -q)
  fi
  return 1 # always "fail" so other completers are tried
}
zstyle ':completion:*' completer _xrcache _oldlist _expand _complete # etc.

Manual ordering of completion alternatives

You can prevent alphabetical sorting by passing -V and the matchname: compadd -V unsorted - $revarray

Bart Schaefer also discusses compadd -V unsorted -a revarray for large arrays:

Notable zstyle options

Hidden completion list

This sort of question occassionally appears on newsgroups from time to time:

I want to have the alternatives offered by consecutive presses of alt-e, and I don't want the alternatives to be listed below the command line. To achieve this, I have had to set the option BASH_AUTO_LIST. If this option is not set, a list of alternatives is displayed as soon as I hit alt-e (and at the same time the first alternative is put on the command line, which is good). But I don't want this option to be set globally. I have not been able to figure out how to make this menu NOT appear for this particular completion, but without setting the global option. Is there a way to achieve this?

The answer is to set the hidden zstyle, which can be done like this:

zstyle ':completion:*list-comp:*' hidden all

But hidden is looked up from _description which you don't call. You could add _wanted around the compadd but all the hidden style actually does is cause the -n option to be passed to compadd which you could do directly.

Style and Convention

ZSH completion scripts are (fortunately) never given the opportunity to evolve into the complex balls of mud that a 'real' programming environment affords; consequently there is much less attention given to the stylistic debates that are tied to other languages.

This said, there are a few, largely unwritten, rules and conventions that are

Descriptions

Always use description. This is important. Really. Always use descriptions. If you have just written down a compadd without a $expl[@] (or equivalent), you have just made an error. Even in helper functions where you use a $@: if you can't be sure that there is a description in the arguments, add one. You can (and, in most cases, should) then give the description you generated after the $@. This makes sure that the, probably more specific, description given by the calling function takes precedence over the generic one you have just generated.

And it really isn't that complicated, is it? Think about a string people might want to see above the matches (in singular – that's used throughout the completion system) and do:

local expl

_description tag expl <descr>
compadd "$expl@]" - <matches ...>

Note that this function also accepts -V and -J, optionally (in the same word) preceded by 1 or 2 to describe the type of group you want to use. For example:

_description tag expl '...'
compadd "$expl[@]" -1V foo - ...    # THIS IS WRONG!!!

is not the right way to use a unsorted group. Instead do:

_description -1V tag expl '...'
compadd "$expl[@]" - ...

and everything will work fine.

Also, if you are about to add multiple different types of matches, use multiple calls to _description and add them with multiple calls to compadd. But in almost all cases you should then add them using different tags anyway, so, see above.

And since a tag directly corresponds to a group of matches, you'll often be using the tags function that allows you to give the explanation to the same function that is used to test if the tags are requested (again: see above). Just as a reminder:

_wanted [ -[1,2]V | -[1,2]J ] <tag> expl <descr> <cmd> ...

and

_requested [ -[1,2]V | -[1,2]J ] <tag> expl <descr> [ <cmd> ... ]

is all you need to make your function work correctly with both tags and description at the same time.

Terminology

  • spec : Argument Specification
  • tag : The 'varieties' of types of objects that are valid completions, e.x a command that takes a set of permissions OR a file as it's next argument.

Variables

  • $state: The canonical variable for processing which tag you are in.
  • $expl: An idiom for options normally given to compadd at some point, typically an array
  • $descr: Argument description variables

External Resources

Footnotes:

1
I picked _expand_alias because it's explicitly designed to be usable as either a key binding or a completer entry. Note #compdef at the top of the source file.