Can we use Commands *and* (global) Flags? What about a 'default' command?

[GwynethLlewelyn]

Hi there! 👋

I was actually a bit surprised that nobody asked this before...

I'm using the latest version of v2, even though I'm quite eager to "upgrade" to v3... as soon as at least some documentation is finished 😁

So, here is my apparently simple question, which, however, I'm not able to implement in a reasonable way.

I'm trying to do a tool that will perform some operations on images. Because I like the git approach, I thought that it would be nice to have some commands (some of which have their own specific flags), as well as some global flags, which all commands can use.

So consider the commands compress, resize, and convert. All can optionally take up to two arguments, i.e. the filename of an existing image, and the output filename. That's the part that works.

Some commands need to have a few extra flags. For instance, convert has an additional (optional) flag, -g, which allows the filetype to be specified (strictly speaking, this might not be needed). That works well, so

./mytool convert -g webp test.jpg test.webp
./mytool convert test.jpg test.webp

work, because webp is the default image format to convert to.

Similarly, the resize option has its own set of flags to set height and width, both of which are fine as well (compress doesn't have any flags, I think, but it may in the future).

So far, so good. I can confirm this on the Action: for each command, and check what it got, and it's as expected.

Now, the problem starts when the flag is actually at the end!

./mytool convert test.jpg test.png -g png

Uh-oh. What happens now is that the -g flag is never 'seen' by convert. Instead, the command reports that it has four arguments, not two. And since only the two first are acknowledged by the code, well, this sort of works but... the image will be converted to webp, since that's the default anyway.

This can still be sort-of-fixed by changing the help to say, "flags before filenames" or something to the effect.

But now I have another issue. What about the global flags?

For the moment, I have two crucial global flags defined: -d (for the debugging level, multiple -d increases that), and -k. The latter is, of course, an API key (which can also be retrieved via the environment). Suppose that we'd tried the previous command with the -g at the end, and we became confused, because we expected a PNG image, and we got a WebP instead. What's going on? Let's increase the debug level:

./mytool convert test.jog test.png -g png -d

Ok. So, this will give us zero errors. The image is still a WebP! Weird!

Maybe at some point the user figures out, well, probably the -d cannot be at the end. Let's try this:

$ ./mytool convert -d test.jpg test.png -g png
2024/10/01 21:09:47 flag provided but not defined: -d

Uh oh — say what?! What is this error?!

"Let me see," the user thinks. "Maybe -d is not the right flag, let's look at help..." And they type:

$ ./mytool convert -h test.jpg test.png -g png
2024/10/01 21:11:30 No help topic for 'test.jpg'. Did you mean "help"?
$ ./mytool convert help test.jpg test.png -g png
2024/10/01 21:11:35 No help topic for 'test.jpg'. Did you mean "help"?
$ ./mytool convert --help test.jpg test.png -g png
2024/10/01 21:11:42 No help topic for 'test.jpg'. Did you mean "help"?

Arrrrrrgh! And, in despair:

$ ./mytool convert --help
NAME:
   mytool convert - Convert between image types

USAGE:
   mytool convert [command options]

DESCRIPTION:
   You can use the API to convert your images to your desired image type. `mytool` currently supports converting between WebP, JPEG, and PNG. When you provide more than one image type in your convert request, the smallest version will be returned to you.
   Image converting will count as one additional compression.

OPTIONS:
   --image-type type, -g type [ --image-type type, -g type ]  Valid image types are: `webp`, `png`, `jpg` (default: "webp")
   --help, -h                                                 show help

... which gives a very useful description, sure, but a very wrong one.

All right, I can correct the texts, that's not a problem. But the above example show that this is not how commands and subcommands work on, say, git, or gpg, or even go (to an extent... go also has its quirks).

Because the problem comes when you've got tons of global flags to parse — they must all come before the command (which is awkward and not intuitive), except for the flags that are specific to that subcommand — these must come after the command, which, well, is just a recipe for trouble. Trouble when answering support requests, that is.

To make matters even slightly worse... where do I actually place the globals?

Consider the following. Besides having filenames as arguments (which is a 'normal' way of expressing things), sometimes it might be more useful to, say, use them as parameters passed by flags (I use -i for the input file and -o for the output file), such as in the following:

./mytool -d -k agh -i file1 convert

Although I dislike having the command at the end of everything, instead of at the start, like a 'normal' git-like CLI is supposed to work, the command above is fine, and works exactly as expected.

Since I set compress as default, this also means that omitting the command will, indeed, do the compress action.

But what if someone mistakenly adds a typo?

./mytool -d -k agh -i test.jpg convret

I would expect one of the two to happen:

1. It detects that there is no valid command, so defaults to compress which is set as the default; or
2. It (correctly) identifies this as an invalid (sub)command and throws an error, shows the help, or perhaps even both.

Instead, none of the above happens.

What happens was tricky for me to debug. With an invalid command, what gets called instead is the Flags Action handler (!). And to be even more devious: the filename is set to test.jpg (since that comes from the flag level), but the convret typo is assumed to be the first argument passed to the command line (which, if you think a bit about it, it makes sense from the perspective of the parser). Since the first argument, by definition, is the input filename, it overrides whatever comes after -i (thus, exactly the opposite of what one would expect), but instead tries to open the convret file instead, which of course doesn't exist and throws a strange error...

Eventually, one might argue that this last error will make the user double-check on their command line and find the typo. But it would be nice to catch it much earlier on!

That said...

If I understand things correctly, mixing & matching Flags and Commands is Not A Good Idea™, right?

I believe I'll simply add flags instead of commands, and avoid the mess. Perhaps this is something that will get changed under v3?... it would certainly be nice!

[abitrolly]

Oookay. So this is the tool.

usage: tool convert [-g gype] <filein> <fileout>

And this works.

tool convert -g webp test.jpg test.webp
tool convert test.jpg test.webp

But flag at the end doesn't work.

tool convert test.jpg test.png -g png

Without example in https://github.com/urfave/cli/tree/main/examples, no "flag at the end" in the docs https://cli.urfave.org/v3/examples/flags/ and no "flag in the end" issues https://github.com/urfave/cli/issues?q=is%3Aissue%20flag%20in%20the%20end%20in%3Atitle%20 and no code to run, I can not inspect how and if it supposed to work and in which v.

[Skeeve]

Correct me, if I'm wrong, but if you "like thegit approach", why don't you accept that cli does it the in the git approach.

Looking at git's man page I see:

       git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|-P|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--super-prefix=<path>] [--config-env=<name>=<envvar>]
           <command> [<args>]

So the global flags clearly are before the command.

Looking at git help remote also shows, that all flags come before any arguments.

[abitrolly]

@Skeeve don't put the burden on proof on me. :D -h is listed as a global flag and works after the command.

[Skeeve]

Okay, @abitrolly, but it's the only example I know of.

Usually I hide, from the help texts, the -h/ --help flag as I consider it general knowledge. But let's not digress here.

Generally speaking, I know of very few programs, where the flags can appear after the arguments. Usually flags alway come first.

[abitrolly]

@Skeeve I am certain there are a lot of tools where the order of flags and args is not important, because Python argparse from its stdlib doesn't impose such restriction by default.

[Skeeve]

Okay. I never use python so I trust you on that.

[abitrolly]

@Skeeve I stolen this code from the internet, which was probably generated by some GPT that stole it from somewhere else/

import argparse

parser = argparse.ArgumentParser(description="Description of the program")

parser.add_argument("positional_argument", type=str, help="Required command")
parser.add_argument("-f", "--flag", action="store_true", help="True or False")
parser.add_argument("-o", "--option", help="Attach a value to 'option'")

args = parser.parse_args()

print(args.positional_argument, args.flag, args.option)

Output:

➜  python ar.py
usage: ar.py [-h] [-f] [-o OPTION] positional_argument
ar.py: error: the following arguments are required: positional_argument
➜  python ar.py -f zzed
zzed True None
➜  python ar.py zzed -f
zzed True None
➜  python ar.py zzed
zzed False None

[GwynethLlewelyn]

Gosh, I'm getting old, since, a year later, I was going to ask exactly the same thing, and even raise an issue... but then, like any well-behaved netizen, I searched around for a bit more, and re-found this discussion from last year.

Well... a year has passed, I've moved on to urfave/cli v3, and the issue sort of persists, but with a few catches.

I realise that I gave git as an example, but I shouldn't, since, in fact, git has the most insanely convoluted sequence of commands and arguments and parameters-as-commands and... anyway, a bad example. Slightly worse than GNU Pretty Good Privacy, gpg, which is also a nightmare. Or the counter-intuitive way in which openssl works, which has different formatting/arguments/parameters, depending on who actually wrote one of the several submodules.

Then we have the likes of ImageMagick and ffmpeg, where both rely on a very specific sequence — and a very rigid one! — in the way parameters get processed. I can't say which is worse, but ImageMagick goes to extremes when you essentially use the command-line to do some scripting, just by applying chains of commands and parameters in a specific way. But then... it's magic!

Instead, I should have given as a better example rclone (which everybody knows) or keybase or cells-client (both of which nobody know, except the handful of geeks using them), and I suppose that the ipfs client could be considered a good example. All share one thing in common — they were all written in Go. I didn't check what they used to parse commands/subcommands/flags. But all of them seem quite reasonable in their approach.

Soooooo back to my original question...

In the context of urfave/cli, and assuming a CLI application using commands (and possibly subcommands) for independent actions, what, exactly, is the role of the 'root' or 'top level' command?

The flags defined there seem to propagate to commands and subcommands as well, that is, you might have some flags that are valid across the whole command/subcommands tree structure. -h/´help´, of course, but that's like a 'superglobal' (as well as -v/--version) which is handled differently (such as being automatically hardcoded with reasonable defaults, which you can override, but most people won't bother to do that).

Next, I thought (the manual implicitly says so) that we'd had the arguments and flags defined at the 'root' level. These I assumed (erroneously) that they would be available on all levels below 'root'. And, indeed, this seems to be the case with, say, my general-purpose flag to change the debug level: it was defined at the 'root' level, but it seems to work on the other levels as well (to be confirmed).

The other thing are the arguments. Here, my main issue was having a command by default at the 'root' level. In that specific case, what happens is that the arguments get wiped out (i.e., they show up empty — they get 'consumed' by the default handler, and, once that command is called, it has no arguments left). This can be reproduced easily, and I would like to consider that it's not such an 'edge case'.

One way of getting them back is, of course, to re-define them at each command (and subcommand). In my case, I just define a []cli.Argument in advance with all the arguments common to all levels, and then it's just a question of assigning them to the Arguments: field, command by command. It's a bit of a pain, but it's doable. It might be slightly tougher if you have to append more arguments to the 'global' ones, but that's doable as well.

Possibly this is a very inefficient way of doing this, but, in general, on CLIs I don't worry too much on the resources (time, space...) consumed during the parsing phase, especially if the CLI is not interactive, but one-shot. Purists will claim that the parsing inefficiency will become a pain if you're calling your application on a tight loop, driven by a shell script that reads a directory with a million entries. In that case, I'd have to figure out if it's worth an optimisation or not (after all, the compiler might already be using pointers to the structure, which gets allocated only once — assuming, that is, that this structure remains immutable and does not get expanded later with more arguments...).

Doing this with flags is a bit trickier (since different commands will probably have different sets of flags; however, some might be common to all commands), but, as said, I haven't encountered that issue with flags, only with arguments. As said, flags defined at the 'root' level, at least so far as I've tested, seem to behave as if they're 'global'.

That said, I continue to have the issue with the command set as default: if the command is omitted, the correct default command is called, but the arguments are now wrong — in a subtle way.

To be sure that it's just urdave/cli's own argument list that gets 'lost', I temporarily rewrote my code to read the arguments from os.Args[]. They're all there — none is missing. And, indeed, I could parse them from os.Args[] — but that defeats the whole point, doesn't it?

Consider this example:

$ tool convert test.jpg test.png -g png

That works now (under v3.6.1), so there is no problem with the 'flag at the end', even if it's a flag defined at the 'root' level.

But assuming that "convert" is set as the default command, and that test.png does not exist yet, then what happens nex is more interesting:

$ tool test.jpg test.png -g png
error: could not find `test.png` for opening

So. what happened here is that the pointer for 'next element on the command line' (or whoever this is actually been implemented) is not advancing correctly. On a first pass, the first token is retrieved from the command line: test.jpg. Since this is neither a flag nor an argument, the default command is called. But urdave/cli behaves as if the command was test.jpg and advances to the next token that is not a flag.

Under v3.6.0 and v3.6.1, this is quite consistent, i.e.

$ tool  -g png --lots-of-more-flags-here -a="some might be written with an equal sign" test.jpg test.png
error: could not find `test.png` for opening

urdave/cli is not confused by any of the above. It 'knows' what are flags, and what are arguments. However, possibly due to the way the tokenizer works (I haven't checked!), it parses non-flags as 'arguments' and considers the first argument to be the command — and all subsequent arguments are, well, arguments. This is a very reasonable assumption. It only fails in the special case where there is a default command, and the user has omitted it on the command line...

Note that an amusing side effect of the default command is that anything that wasn't recognised as a valid command will be accepted, but in any of those cases, this 'bug' is not present (i.e., the token for the invalid command will be consumed as expected, leaving the arguments in the right position to be retrieved correctly).

Finally, this is suspiciously similar to what happens in #2223 — but that issue is about empty flags, while this one is about empty commands.

This is where the setup seems to be done for the DefaultCommand:

cli/command.go

Line 337 in 41334d4

if cmd.DefaultCommand != "" {

The parsing logic seems to stem from here:

cli/command_run.go

Line 266 in 2240690

argsWithDefault := cmd.argsWithDefaultCommand(args)

It's not quite obvious where the issue is. My best guess is that there is something that happens here, when the current command under consideration is called with the default command instead:

cli/command_run.go

Line 273 in 2240690

} else if cmd.parent == nil && cmd.DefaultCommand != "" {

Possibly when we reach this point with an empty command on the line (which would be the case) and run Command() with the 'replaced' empty command with a new one, the logic gets confused, and things like cmd.Args().Slice(), which gets called a few lines later might have the wrong arguments, namely, the first element in that slice is presumed to be the actual command to be run. My doubts and hesitations come from the simple fact that a seriously botched implementation should trigger an endless loop, but this clearly does not happen.

Oh, and for the sake of completude, I'm trying to fix this fork of mine.

[GwynethLlewelyn]

All right, let's try again. Consider an image conversion tool (like ImageMagick, for instance). It has, say, three options:

1. convert — this converts between file formats
2. resize — this changes the original size, scaling up or down
3. optimise — this compresses an image further

The tool is to be invoked like this:

$ tool [command] [flags] input-file.iii output-file.ooo

Assume that just specifying the .ooo extension is enough to figure out the format.

So, for instance, to convert a JPEG into a PNG:

$ tool convert test.jpg test.png

Each command, of course, may have its own set of flags, e.g. resize would have height/width parameters, while optimise might have a compression level (or a degree of lossy-ness, depending on the file format).

All of the above work flawlessly out of the box.

However, in 90% of the cases, let's assume that users simply want to do a conversion, and don't want to bother with special parameters requiring flags, so I assume that the default command is convert. This means I'm expecting the following to work:

$ tool test.jpg test.png

But what I get instead is:

$ tool test.jpg test.png
error: could not find `test.png` for opening

Note the subtleness here: it skips the first argument — it assumes that the first argument is a command, figures out that it's not one of the three valid commands (of course not: it is, after all, a filename!), so it replaces the first argument with convert, uses the second argument as the input file and tries to open it. Obviously, this file will not exist yet, so it panics when attempting to open a non-existing file.

In other words, tool test.jpg test.png gets (internally) replaced by tool convert test.png, and since test.png does not exist, it throws an error.

What I expected to happen was that the default command could be omitted, and the system would just proceed with calling convert without consuming the first argument, that is, internally calling tool convert test.jpg test.png.

---

As a side note: I don't understand how exactly the DefaultCommand and the CommandNotFound handler are supposed to work. I forked my own code to try to figure out what was happening in either case (and all possible combinations I could think of), and neither seems to get properly called as expected.

[abitrolly]

I see. I would do my custom argument parsing for this case. At least splitting the whole command line into commands and their arguments.

[dearchap]

Actually I think this is a bug. Let me check the tests

[dearchap]

yes its a bug.

[dearchap]

@GwynethLlewelyn Please try the fix in #2228