Quirky zsh behavior with file descriptors

Playing around with file descriptors, doing things like:

exec {fd}>&2
echo test >&"${fd}"

Does anyone know why my shell exits when I close a descriptor?

exec "${fd}">&-

Closes the shell (or possibly the tmux panel).

By chance are you closing fd 0 (ie. common handle to the tty/stdin)?

  1. Is the exec "${fd}">&- being run directly at an Interactive terminal or instead within a Script run at an interactive?
  2. Can it be caused by just running the exec "${fd}">&- without the earlier redirections; Preferably check/provide/confirm testing an example of the ${fd} manually written as the specific fd number that causes the issue (is it failing only with 0 or is it something else)? Whatā€™s the minimalist steps to reproduce and which file descriptor number (we may need to check what is being closed)?

It is worth noting that if you close fd 0 (stdin) which is the tty for an interactive shell that bash will then logout as the next time it checks for command input then yyparse() a few frames past the reader_loop will have found an EOF and triggers the current interactive shell to exit.

For example, running just the following:

exec 0>&-

will lead to close(0) if that shellā€™s pid is straceā€™d from another terminal:

$ strace -fvy -e fcntl,close,write,clone,execve -p 18709
strace: Process 18709 attached
write(2</dev/pts/20>, "echo $$", 7)     = 7
write(2</dev/pts/20>, "\10\10\10\10\10\10xec 0>&-", 14) = 14
write(2</dev/pts/20>, "\n", 1)          = 1
fcntl(0</dev/pts/20>, F_GETFD)          = 0
fcntl(0</dev/pts/20>, F_DUPFD, 10)      = 10</dev/pts/20>
fcntl(0</dev/pts/20>, F_GETFD)          = 0
fcntl(10</dev/pts/20>, F_SETFD, FD_CLOEXEC) = 0
close(0</dev/pts/20>)                   = 0
close(10</dev/pts/20>)                  = 0
clone(strace: Process 18821 attached
....

(side note: it wonā€™t fully fork/replace if no <command> is passed to exec, which is intended in your example, but means we may not need to use -f or another terminal for straceā€™ing; though it can be convenient to do so)

and the interactive bash shell will later exit when it comes back around to check on any input and sees EOF

# gdb --batch  -p 26075 -ex 'break exit_or_logout' -ex 'continue' -ex 'bt' -ex 'quit'
0x00007ff45a3969a0 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Breakpoint 1 at 0x470380: file ./exit.def, line 106.

Breakpoint 1, exit_or_logout (list=list@entry=0x0) at ./exit.def:106
106                              last_shell_builtin == logout_builtin ||
#0  exit_or_logout (list=list@entry=0x0) at ./exit.def:106
#1  0x0000000000470515 in exit_builtin (list=list@entry=0x0) at ./exit.def:69
#2  0x0000000000428a98 in handle_eof_input_unit () at ./parse.y:5733
#3  yyparse () at ./parse.y:428
#4  0x000000000041e03a in parse_command () at eval.c:229
#5  0x000000000041e0fc in read_command () at eval.c:273
#6  0x000000000041e2fc in reader_loop () at eval.c:138
#7  0x000000000041c9de in main (argc=1, argv=0x7fff35380ed8, env=0x7fff35380ee8) at shell.c:759
A debugging session is active.

More specifically down this code path:

(gdb) 

Breakpoint 1, close () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0  close () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000000046ef76 in _evalfile (filename=filename@entry=0x1694930 "/root/.bash_logout", flags=flags@entry=9)
    at evalfile.c:166
#2  0x000000000046f477 in maybe_execute_file (fname=fname@entry=0x4bc08d "~/.bash_logout", 
    force_noninteractive=force_noninteractive@entry=1) at evalfile.c:315
#3  0x0000000000470363 in bash_logout () at ./exit.def:163
#4  0x00000000004703a4 in exit_or_logout (list=list@entry=0x0) at ./exit.def:148
#5  0x0000000000470515 in exit_builtin (list=list@entry=0x0) at ./exit.def:69
#6  0x0000000000428a98 in handle_eof_input_unit () at ./parse.y:5733
#7  yyparse () at ./parse.y:428
#8  0x000000000041e03a in parse_command () at eval.c:229
#9  0x000000000041e0fc in read_command () at eval.c:273
#10 0x000000000041e2fc in reader_loop () at eval.c:138
#11 0x000000000041c9de in main (argc=1, argv=0x7ffea0aedee8, env=0x7ffea0aedef8) at shell.c:759

(gdb) f 7
#7  yyparse () at ./parse.y:428
428                               handle_eof_input_unit ();
(gdb) l
423             |       yacc_EOF
424                             {
425                               /* Case of EOF seen by itself.  Do ignoreeof or
426                                  not. */
427                               global_command = (COMMAND *)NULL;
428                               handle_eof_input_unit ();
429                               YYACCEPT;
430                             }
431             ;
432

One other side note is that >&- assumes 1>&-; and <&- assumes 0<&- if by chance you end up using those ever without a specified fd. (or if "${fd}" in exec "${fd}>&- evaluates/expands to empty). Though in your example above it is using ...>&-, thus my guess that ${fd} might be set to 0, or the example posted might not match exactly what is being done (trying to simplify for our convenience). Also, there may be other fd that bash is using around the time of exec.

  1. I should ask what the current/eventual goal is if the issue isnā€™t that itā€™s closing fd 0. Youā€™ll likely find that stdin/0 can be closed when used within a script (ie. one run by a separate bash process which is not in interactive mode), if that happens to be desired. :slight_smile:

If Iā€™ve missed something or my guess is incorrect then just let us know and maybe a bit more of an example; and check which fd is being used(closed) in that command and we might be able to reproduce it or suggest more specific troubleshooting to clarify.

TL;DR: I suspect that ${fd} is 0 thus exec "${fd}>&>- would close the stdin handle to the terminalā€™s current tty, which is ok if your application/script writes directly to the tty device I suppose; however, for an interactive bash shell prompt it is the equivalent of closing the thing the bash prompt is reading your typing from, and after running the exec it checks for the next command youā€™ll type but the fd itā€™s reading commands/input from was closed, so it exits. If itā€™s not an interactive terminal you could close stdin/0 but at the moment I suspect itā€™s an interactive terminal.

5 Likes

Wow, bravo on an extensive post.

{fd} in this case is not ${fd}. exec {var}>&whatever will assign the first available file descriptor over 9 to $var. It is not defined before the exec statement.

Also, to answer your question, I was running this in zsh interactively.

Oh! zsh oops! I was wondering about the quotes used in the later "${fd}">&- and was not sure if the last line or prior was left out as a typo and forgot about the bash 4.1 style use of {var}>&whatever.

Iā€™m less familiar with the internals of zsh but may poke at it a bit later. Just for quick reference, with the exact syntax seen below the interactive bash shell does not exit, though I do see zsh (as launched below) does exit.

Some confusion for bash (though note it does not exit below)

[root@host ~]# ls -l /proc/$$/exe
... /proc/1662/exe -> /usr/bin/bash
[root@host ~]# exec {fd}>&2
[root@host ~]# echo $fd
10
[root@host ~]# exec "${fd}">&-
-bash: exec: 10: not found
-bash: printf: write error: Bad file descriptor
[root@host ~]# echo $?
-bash: echo: write error: Bad file descriptor
-bash: printf: write error: Bad file descriptor
[root@host ~]# exit

Interesting- in this case of zsh itā€™s exiting despite fd 11:

[root@host ~]# zsh -l
[root@host]~# ls -l /proc/$$/exe
... /proc/2221/exe -> /usr/bin/zsh
[root@host]~# exec {myfd}>&2
[root@host]~# echo $myfd
11
[root@host]~# exec "${myfd}">&-
bash: 11: command not found...
zsh: command not found: 11
[root@host ~]# echo $?
127
[root@host ~]# ls -l /proc/$$/exe
... /proc/2161/exe -> /usr/bin/bash

Another variant, just out of curiosity:

[root@host ~]# ls -l /proc/$$/exe
... /proc/1204/exe -> /usr/bin/bash
[root@host ~]# zsh -l
[root@host]~# ls -l /proc/$$/exe
... /proc/1265/exe -> /usr/bin/zsh
[root@host]~# exec "">&-
zsh: permission denied:
# echo $?
126
[root@host ~]# ls -l /proc/$$/exe
... /proc/1204/exe -> /usr/bin/bash

Those quick tests were for RHEL/CentOS 7:

# rpm -qa zsh bash
zsh-5.0.2-34.el7_8.2.x86_64
bash-4.2.46-34.el7.x86_64

I may have something mixed up still (re:quotation use), and there may be version specific behavior, though the above is a difference at least. I may poke at this again out of curiosity to see how itā€™s getting there. Unsure of things like if zsh replaces and then exits with error code when it runs into and fd issue (regardless of which) though Iā€™d guess it would not execve or similar just for redirection.

2 Likes

Before looking too deep, testing with just {newfd}>&- is happy with zsh and bash. (no quotes in the close; like the below)

<Iā€™m too young for links!>/Doc/Release/Redirection.html

The following shows a typical sequence of allocation, use, and closing of a file descriptor:

integer myfd
exec {myfd}>~/logs/mylogfile.txt
print This is a log message. >&$myfd
exec {myfd}>&-
Note that the expansion of the variable in the expression >&$myfd occurs at the point the redirection is opened. This is after the expansion of command arguments and after any redirections to the left on the command line have been processed.

Error output in the previous comment suggesting bash at least things itā€™s getting a command to run vs the #>&# syntax. Tracing zsh shows it also trying to execve a file named the filedescriptor.

Working example (no exit with zsh):

[root@host ~]# ls -l /proc/$$/exe
... /proc/5642/exe -> /usr/bin/bash
[root@host ~]# zsh -l
[root@host]~# ls -l /proc/$$/exe
... /proc/5702/exe -> /usr/bin/zsh
[root@host]~# exec {myfd}>&2
[root@host]~# echo test >&"${myfd}"
test
[root@host]~# echo test >&${myfd}
test
[root@host]~# exec {myfd}>&-
[root@host]~# echo $?
0
[root@host]~# ls -l /proc/$$/exe
... /proc/5702/exe -> /usr/bin/zsh

So at least that behaves better :slight_smile:

As noted, with zsh test below itā€™s trying to run that:

[root@localhost ~]# zsh -l
[root@localhost]~# echo $$
6793
[root@localhost]~# exec {myfd}>&2
[root@localhost]~# echo test >&${myfd}
test
[root@localhost]~# exec "${myfd}">&-
bash: 11: command not found...
zsh: command not found: 11
...
execve("/usr/bin/11", ["11"], .....
....

Just for reference, though itā€™s only to note that with quotes it does match behavior of it suspecting itā€™s a command and not just redirection.

# man zshbuiltins | grep -E '\bexec\b' -A 6
       exec [ -cl ] [ -a argv0 ] simple command
              Replace the current shell  with  an  external  command
              rather  than  forking.  With -c clear the environment;
              with -l prepend - to the argv[0] string of the command
              executed  (to  simulate  a login shell); with -a argv0
              set the argv[0] string of the command  executed.   See
              the section `Precommand Modifiers'.

# man bash | grep -E '^\s*\bexec\b \[' -A 16
       exec [-cl] [-a name] [command [arguments]]
              If command is specified, it replaces  the  shell.   No
              new  process  is  created.   The  arguments become the
              arguments to command.  If the -l option  is  supplied,
              the shell places a dash at the beginning of the zeroth
              argument passed to command.   This  is  what  login(1)
              does.   The  -c  option  causes command to be executed
              with an empty environment.  If  -a  is  supplied,  the
              shell  passes  name as the zeroth argument to the exeā€
              cuted command.  If command cannot be executed for some
              reason,  a  non-interactive  shell  exits,  unless the
              shell option execfail is enabled,  in  which  case  it
              returns failure.  An interactive shell returns failure
              if the file cannot be executed.   If  command  is  not
              specified, any redirections take effect in the current
              shell, and the return status is 0.  If there is a  reā€
              direction error, the return status is 1.

Why zsh (at least in my case) is exiting versus bash not exiting if they both think theyā€™re running a commandā€¦ curious! That I will leave checking on that and maybe those other behaviors for tomorrow.

2 Likes

@spiders this is actually my current problem if you have any insightsā€¦

1 Like

Ah. hrmā€¦ I see what you mean.

So the amusing part is that emulation for ksh does work, as it had the feature before bash didā€¦ haha :thinking:

emulate -LR ksh

Some documentation seemed to be in man zshbuiltins (emulate) and man zshmisc (redirection with varid style), though not specifics about this observation.

However, it does seem possible to toggle some shopt after running emulate so that ignorebraces is off (though braceexpand can also change behaviors:

#!/usr/bin/env zsh                                                                                                                                                                                                              

set -euo pipefail                                                                                             

emulate -LR bash

#setopt braceexpand                                                                                                    
unsetopt ignorebraces

#for i in {1..10}; do                                                                                                  
  exec {fd}> >( tee /dev/stderr | logger )

  echo "FD: ${fd}" >&"${fd}"
#done

(Yay! It seems we can still set the shell option after emulate.)

Below is not a fully broken down explanation of misc findings or definitive conclusion, but some notes for now. At the moment Iā€™m unsure if itā€™s intended for bash emulation defaults or an oversight (now that its the future~), or common bash defaults per distro, etc. Itā€™d be recommended to look into the effects for your existing code if considering using the above approaches. But it might help jumpstart further investigation. If there was something within zsh that needed to be fixed for this to work, Iā€™m unsure it would be worth relying on what versions all users might have, etc. (Though it may be the same for all emulation.)

Oh, also on RHEL/CentOS 7, zsh didnā€™t like the pipefail; I moved on to testing on Fedora32 for the sake of checking closer versions.

Quick example of the behavior with the normal options (no emulation):

[user@host zsh]$ shopt  | grep brace
[user@host zsh]$ zsh -l
[user@host]~/src/zsh% exec {myvar}>&2
[user@host]~/src/zsh% exec {myvar}>&-
[user@host]~/src/zsh% setopt ignorebraces
[user@host]~/src/zsh% exec {myvar}>&-    
bash: {myvar}: command not found...
[user@host zsh]$

(also the other side-behavior is seen above; failing exec exits interactive zsh)
(as expected, both options also have their respective effect on echo test{,ing} in non emulated zsh)

(hoped for a spoiler tag to collapse verbose post content)

Below this point is mostly some notes on what was checked and pastes if revisiting.

I can see with some targeted gdb breakpoints where zsh parses the words of each line, and see within par_simple() where it skips over that part of io redirection expansion depending on the options and also howtokstr was setup prior to that point as either "\217fd\220" or {fd}.

Checking on the introduction of the {varid} feature in upstream helped (more than initial looking for use of EMULATE_KSH and SH in hopes of seeing something easy missing/extra for ksh emulation vs bash. Then focusing on *REDIR_VARID* and similar macros and then the trails of redir and exec.c.

Making a longer explanation a shorter story for the moment:

  • I broke on execve() in gdb and got a list of functions of interest, then broke on those in the working scenario, got rough paths of concern and more targeted breakpoints.
  • Realized ksh emulation worked, looked over upstream for uses of EMULATE_KSH and for SH as well, hoping Iā€™d see something missing/extra for ksh emulation. This didnā€™t quite pan out as ā€˜quick answerā€™, so moved on but that can be revisited.
  • Found the sections for execcmd_exec() and par_cmd/par_simple that were working on redirection, noted use of IGNOREBRACES at one critical point and realized itā€™s worth testing forcing things afterwards. (but note the change in tokstr in relation to Inbrace; that difference happens prior; ie. part of the root lies elsewhere)

Misc snippets and examples of difference in behavior, mostly pasted so I have them for later if revisiting.

593        /* Do process substitutions */
(gdb)
3594        if (redir)
3595            spawnpipes(redir, nullexec);                                                                           
3596
3597        /* Do io redirections */
3598        while (redir && nonempty(redir)) {
3599            fn = (Redir) ugetnode(redir);                                                                          
3600
3601            DPUTS(fn->type == REDIR_HEREDOC || fn->type == REDIR_HEREDOCDASH,
3602                  "BUG: unexpanded here document");                                                                
3603            if (fn->type == REDIR_INPIPE) {
(gdb)
3604                if (!checkclobberparam(fn) || fn->fd2 == -1) {
3605                    if (fn->fd2 != -1)
3606                        zclose(fn->fd2);                                                                           
3607                    closemnodes(mfds);                                                                             
3608                    fixfds(save);                                                                                  
3609                    execerr();                                                                                     
3610                }
3611                addfd(forked, save, mfds, fn->fd1, fn->fd2, 0, fn->varid);                                         
3612            } else if (fn->type == REDIR_OUTPIPE) {
3613                if (!checkclobberparam(fn) || fn->fd2 == -1) {
(gdb) break 3594
Breakpoint 5 at 0x55555558aa91: file exec.c, line 3594.

Initial area of concern (running picky bash emulation):

(gdb)
1875                if (tok == TYPESET)
(gdb)
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
(gdb)
1921                    if (postassigns) {

(gdb) break 1878
Breakpoint 6 at 0x5555555d17c1: file parse.c, line 1878.

....

Breakpoint 6, par_simple (nr=0, cmplx=0x7fffffffd564) at parse.c:1878
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
(gdb) p tokstr
$17 = 0x7ffff7fc6428 "exec"

(gdb) n
1921                    if (postassigns) {

(gdb) c
Continuing.

Breakpoint 6, par_simple (nr=0, cmplx=0x7fffffffd564) at parse.c:1878
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
(gdb) p tokstr
$18 = 0x7ffff7fc6478 "{fd}"
(gdb) n
1921                    if (postassigns) {

...

Breakpoint 1, execcmd_exec (state=0x7fffffffd5a0, eparams=0x7fffffffd1e0, input=0, output=0, how=18, last1=2,
    close_if_forked=-1) at exec.c:2791
2791    {

[Detaching after fork from child process 3340547]
./test-OLD.sh:10: command not found: {fd}
[Inferior 1 (process 3340503) exited with code 0177]

(note the ā€œ{fd}ā€ above, and skiping L:1978)

And the working ksh emulation scenario:

(gdb) set args ./test-OLD-KSH.sh
(gdb) r
Starting program: /usr/bin/zsh ./test-OLD-KSH.sh

....

(gdb) c
Continuing.

Breakpoint 6, par_simple (nr=0, cmplx=0x7fffffffd554) at parse.c:1878
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
(gdb) p tokstr
$37 = 0x7ffff7fc6428 "exec"
(gdb) n
1921                    if (postassigns) {
(gdb) c
Continuing.

Breakpoint 6, par_simple (nr=0, cmplx=0x7fffffffd554) at parse.c:1878
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
(gdb) p tokstr
$38 = 0x7ffff7fc6478 "\217fd\220"
(gdb) n
1881                    char *eptr = tokstr + strlen(tokstr) - 1;                                                      

(gdb) l
1876                    intypeset = is_typeset = 1;                                                                    
1877
1878                if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
1879                {
1880                    /* Look for redirs of the form {var}>file etc. */
1881                    char *eptr = tokstr + strlen(tokstr) - 1;                                                      
1882                    char *ptr = eptr;                                                                              
1883
1884                    if (*ptr == Outbrace && ptr > tokstr + 1)
1885                    {

(at the end above it enters this section for {var}> parsing)

Various uses, and the emulation masking in option table (to check later):

$ grep IGNOREBRACES . -R
./Doc/Zsh/options.yo:pindex(IGNOREBRACES)
./Doc/Zsh/options.yo:pindex(NOIGNOREBRACES)
./Doc/intro.ms:\fIIGNOREBRACES\fP turns off csh-style brace expansion.
./Src/lex.c:        if (isset(IGNOREBRACES) || sub)
./Src/lex.c:        if ((isset(IGNOREBRACES) || sub) && !in_brace_param)
./Src/lex.c:        if (unset(IGNOREBRACES) && !sub && bct > in_brace_param)
./Src/lex.c:    } else if (unset(IGNOREBRACES) && !sub && lexbuf.len > 1 &&
./Src/lex.c:             (unset(IGNOREBRACES) && unset(IGNORECLOSEBRACES) &&
./Src/parse.c:      if (!isset(IGNOREBRACES) && *tokstr == Inbrace)
./Src/subst.c:      if (unset(IGNOREBRACES) && !(flags & PREFORK_SINGLE)) {
./Src/zsh.h:    IGNOREBRACES,
./Src/options.c:{{NULL, "ignorebraces",       OPT_EMULATE|OPT_SH},       IGNOREBRACES},
./Src/options.c:{{NULL, "braceexpand",        OPT_ALIAS}, /* ksh/bash */ -IGNOREBRACES},
./Src/options.c:    /* I */  IGNOREBRACES,
./Src/Zle/zle_tricky.c:         (unset(IGNOREBRACES) &&
./Src/Zle/zle_tricky.c:    if (!isset(IGNOREBRACES)) {

(may look at the emulation flags in options.c and how theyā€™re used)
(may have to return to the other REDIR macros and emulation flags)

I suspect a few tweaks and recompiling can confirm regarding the default options above.

------------------------------------
EDIT: oh, you sneaky macrosā€¦ OPT_SH was also EMULATE_SH

$ grep OPT_SH . -R
./Src/options.c:#define OPT_SH          EMULATE_SH
./Src/options.c:#define OPT_ALL         (OPT_CSH|OPT_KSH|OPT_SH|OPT_ZSH)
./Src/options.c:#define OPT_BOURNE      (OPT_KSH|OPT_SH)
./Src/options.c:#define OPT_BSHELL      (OPT_KSH|OPT_SH|OPT_ZSH)
./Src/options.c:{{NULL, "bsdecho",            OPT_EMULATE|OPT_SH},       BSDECHO},
./Src/options.c:{{NULL, "ignorebraces",       OPT_EMULATE|OPT_SH},       IGNOREBRACES},
./Src/options.c:{{NULL, "octalzeroes",        OPT_EMULATE|OPT_SH},       OCTALZEROES},

I missed that the first time aroundā€¦ need morning coffee~ :slight_smile:

EDIT2: And just checking from set -o perspectiveā€¦

If we remove the -R we can get the original zsh -x to go through, or can set -x after the emulate; but, with no braces option changes, we can confirm set -o | grep brace output and then check with unset:

### with -R and no unset/set of anything (ie. default emulation)
$ zsh -x ~/test-ZSH.sh
+.../test-ZSH.sh:5> set -euo pipefail
+.../test-ZSH.sh:10> emulate -LR bash
braceccl              off
noignorebraces        off
ignoreclosebraces     off
.../test-ZSH.sh:18: command not found: {fd}

### without -R (-x passes), note 'noignorebraces' after unset
$ zsh -x ~/test-ZSH.sh
+.../test-ZSH.sh:5> set -euo pipefail
+.../test-ZSH.sh:10> emulate -L bash
+.../test-ZSH.sh:14> unsetopt ignorebraces
+.../test-ZSH.sh:15> set -o
+.../test-ZSH.sh:15> grep brace
braceccl              off
noignorebraces        on
ignoreclosebraces     off
+.../test-ZSH.sh:20> echo 'FD: 14'

You can test a few combinations of with or without -R reset and when setting opts before and after if curious, though Iā€™ll avoid documenting all of that here. I believe braces ones are reset always after emulate bash in my case. (bash has braceexpand on by default).

2 Likes

Got a good answer here. Apparently there isnā€™t a bash emulate mode in zsh at allā€¦ The command enables bourne/posix emulation. Youā€™re right about IGNORE_BRACES being the culprit. I think the best solution for me is to use emulate -R zsh and then change flags as issues arise.

1 Like

Ahh. Nice! A more historically and zsh informed comment. A snippet from the userā€™s larger comment on the stackexchange for anyone that runs into this in thread in the future or has followed along:

echo {fd}< /dev/null is required by POSIX to output {fd} (though thatā€™s going to change in future versions of the standard(link-removed)), so zsh disables it in sh emulation. It doesnā€™t in ksh emulation, so thatā€™s the one youā€™ll want to use here.

As for which option is enabled in which emulation (zsh, sh, ksh, csh), see the <C> , <K> , <S> , <Z> next to each option description in the manual ( info zsh 'Description of Options' ).

What option affects the {fd}>... feature is specified in the description of that feature in the manual ( info zsh 'file descriptors, use with parameters' ): IGNORE_BRACES whose description ( info zsh IGNORE_BRACES ) has a <S> next to it meaning itā€™s enabled by default in sh emulation only.

This explains the code comments referencing ā€œour (k) flagā€ which I wondered about but could not find the right keyword when searching the zsh. I also originally tried with sh itself, thinking it could be sh and not bash emulation (to compare the failure), but saw a syntax error and not the behavior of trying to run {fd} as a command for the original example script (it may be more of a hybrid; certainly itā€™s emulated and not bash/sh itself). This is why my copies of the script started having -OLD and .sh in their names at one point. :sweat_smile: I did open the info page for a few seconds and then had flashbacks to always not remembering how to search all the broken out pages of grub for what I wanted that I had previously found, and then promptly closed itā€¦

It does seem to be in the zshall manpage, the ā€œflagā€ part of a comment threw me off when quickly checking.

COMPATIBILITY
     Zsh  tries  to  emulate sh or ksh when it is invoked as sh or
     ksh respectively; more precisely, it looks at the first  letā€
      ter  of  the name by which it was invoked, excluding any iniā€
      tial 'r' (assumed to stand for 'restricted'), and if that  is
      'b',  's'  or 'k' it will emulate sh or ksh.  Furthermore, if
      invoked as su (which happens  on  certain  systems  when  the
      shell  is  executed by the su command), the shell will try to
      find an alternative name from the SHELL environment  variable
      and perform emulation based on that.
...
      The following options are set if the shell is invoked  as  sh
      or  ksh: NO_BAD_PATTERN, NO_BANG_HIST, NO_BG_NICE, NO_EQUALS,
      NO_FUNCTION_ARGZERO,  GLOB_SUBST,  NO_GLOBAL_EXPORT,  NO_HUP,
      INTERACTIVE_COMMENTS,   KSH_ARRAYS,  NO_MULTIOS,  NO_NOMATCH,
      NO_NOTIFY, POSIX_BUILTINS, NO_PROMPT_PERCENT, RM_STAR_SILENT,
      SH_FILE_EXPANSION, SH_GLOB, SH_OPTION_LETTERS, SH_WORD_SPLIT.
      Additionally the BSD_ECHO and IGNORE_BRACES options  are  set
      if  zsh  is  invoked  as  sh.   Also,  the  KSH_OPTION_PRINT,
      LOCAL_OPTIONS, PROMPT_BANG, PROMPT_SUBST and  SINGLE_LINE_ZLE
       options are set if zsh is invoked as ksh.
...
DESCRIPTION OF OPTIONS
       In  the  following list, options set by default in all emulaā€
       tions are marked <D>; those set by default only in csh,  ksh,
       sh, or zsh emulations are marked <C>, <K>, <S>, <Z> as approā€
       priate.  When listing options (by `setopt', `unsetopt',  `set
       -o'  or  `set  +o'), those turned on by default appear in the
       list prefixed with `no'.  Hence (unless  KSH_OPTION_PRINT  is
       set),  `setopt'  shows all options whose settings are changed
       from the default.
...
       IGNORE_BRACES (-I) <S>
              Do not perform brace expansion.  For  historical  reaā€
              sons   this   also   includes   the   effect   of  the
              IGNORE_CLOSE_BRACES option.
....
OPTION ALIASES
       Some options have alternative names.  These aliases are never
       used for output, but can be  used  just  like  normal  option
       names when specifying options to the shell.

       BRACE_EXPAND
              NO_IGNORE_BRACES (ksh and bash compatibility)
...
SINGLE LETTER OPTIONS
...
       -I     IGNORE_BRACES
...

I more commonly use brace expansion (like test{,ing}) at a terminal (bash) then in scripts and had not been using {varid} expansion in the past and just avoid it if writing sh scripts (versus bash for hopes of compatibility).

The workaround of using bash emulation but enabling the shopt after starting emulation may be an uncommonly tested scenario, versus using ksh emulation, but likely is using the usual codepaths for zsh/ksh-emulation when braces processing is enabled. As you noted in the comments on stack exchange it may be a decision of using sh syntax or making sure using ksh emulation is ok for the existing code base (or using that shopt method further above if it seems appropriate).

Other observations that looked annoying:
(bash emulation for the first output and ksh for the later)

## checking options via conditional (bash emulation)
$ zsh -x ~/test-KSH.sh
+.../test-KSH.sh:5> set -euo pipefail
+.../test-KSH.sh:10> emulate -L bash
+.../test-KSH.sh:14> set -o
+.../test-KSH.sh:14> grep brace
braceccl              off
noignorebraces        off
ignoreclosebraces     off
+.../test-KSH.sh:15> [[ -o braceexpand ]]
+.../test-KSH.sh:16> [[ -o ignorebraces ]]
+.../test-KSH.sh:16> echo SET
SET
+.../test-KSH.sh:17> [[ -o noignorebraces ]]
+.../test-KSH.sh:20> '{fd}'
.../test-KSH.sh:20: command not found: {fd}

## checking options via conditional (ksh emulation)
$ zsh -x ~/test-KSH.sh
+.../test-KSH.sh:5> set -euo pipefail
+.../test-KSH.sh:10> emulate -L ksh
+.../test-KSH.sh:14> set -o
+.../test-KSH.sh:14> grep brace
braceccl              off
ignorebraces          off
ignoreclosebraces     off
+.../test-KSH.sh:15> [[ -o braceexpand ]]
+.../test-KSH.sh:15> echo SET
SET
+.../test-KSH.sh:16> [[ -o ignorebraces ]]
+.../test-KSH.sh:17> [[ -o noignorebraces ]]
+.../test-KSH.sh:17> echo SET
SET
+.../test-KSH.sh:22> echo 'FD: 14'
FD: 14

though at least if bash emulation is used and unsetopt ignorebraces is used then we do see noignorebraces set along side braceexpand.

After it wasnā€™t just closing fd 0 I got sucked in, but still, being more familiar with zsh internals and behavior is probably better for me. Thank you for offering up your issue so that weā€™re now all more informed of the caveat/emulation feature. And best of luck with adapting the script. :slight_smile:

1 Like

Thanks for all your help @spiders, if every new member were as dedicated as you, we would have taken over the world long ago.

Iā€™m going to merge all this zsh stuff out for posterity and because of how long it is.

1 Like

@spiders whatā€™s your back story? You possess knowledge of the intricacies at such a deep level it is fascinating.

2 Likes

@oO.o Haha. No problem. I checked the forums by chance and saw the topic title/post and thought I might be able to help out. Your post was what got me to sign-up (and I usually avoid creating accounts in new places if I can). So you get the credit for convincing me to do so (and maybe credit to the sticky thread and site design for exposing it).

@Dynamic_Gravity In this context (w/o the life story**) Iā€™m probably a mix of software maintenance and dev. Itā€™s very close to my wheelhouse. Iā€™ve been programming since I was little and I enjoy puzzles and learning / helping, so troubleshooting / reverse-engineering and exploring can be fun. Linux/opensource does very well to feed that passion.
(** Iā€™m tempted for our entertainment/introductions, but am a bit wary of dumping so much in public forums at the moment)

Sorry about the delayed follow up.


Much of it is having the list of tools, resources/contacts, and general familiarity to begin poking around, and then the persistence to keep digging your own wondrous rabbit hole. (and filling it with your favorite toys and bed time horror stories)

While Iā€™ve not looked inside zsh before and while it was only a tip of itā€™s iceberg (Iā€™m not now a zsh expert by any means) it can become useful in the future (certainly knowing of emulation mode and this behavior, and places in the larger docs, thank you for that), or at the least the motions are additional practice of exploring uncharted code bases.

{Tangent} this is why many forms of community-connectivity can be great, discovering new tools / methods / workflows / challenges. Itā€™s a lot of fun to stand behind a peer at their system/bench and share/realize new techniques/details; you donā€™t get that directly much without a good community. The field/workplace is the prime example (though limited to your own environment), but even a blog/mailing-list/journal (where only what was ~thought~ to have been said is presented) can still share those details/questions. And, similar for stackexchange and that user- everyone with their own experience/areas/interests coming together. {/Tangent}


Thanks for cleaning up the general thread. I tend to provide a trail of example commands/output during my journeys in case others can find them useful, so it can get fairly verbose and sometimes rambling / meandering if partially complete. (One downside of ongoing thread with multiple questions is older unanswered ones becoming lost; it is still worth it though. I worried my longer posts would pollute it a bit.)

Not a spoiler, just trying to lighten the wall of less important text
(I wanted to clarify that documentation was likely better for the later emulation issue than my eventual route here.)

After starting to look into zsh for the first behavior I kept going with the code side for the emulated {varid}> behavior, for a possible solution / workaround (and to match the bash examples) and since I donā€™t have as much user-side zsh familiarity (sometimes the code will speak faster if not immediately found in docs/online). Whereas the documentation on options and emulation types might have been enough if found earlier (and connection to braces expansion), though that can be part of the fun. Thus it grew a bit more than I expected for just a quick answer to a bigger small-linux-question thread. My starting with code can also act as a fallback (or find non-documented workarounds) if itā€™s a bug/unexplained after waiting for responses from more senior users of zsh.

Basically, already fairly familiar with some bash internals; tested a few things, pulled down debug symbols to check the behavior with gdb, then the zsh source from package manager (later from upstream git mirror). Read over it to figure out what it may be up to in those areas. Presented findings with evidence once understood. So I donā€™t know zsh deeply, but the tools and methods for poking around the problem area :+1:t2:


Also, I may have figured out the Discourse reply-to-topic vs the to-post-sub-thread and quoting with references now. It wasnā€™t a tree view of comments and quoting markup didnā€™t have the usual post id references/inline-linkage (though found the highlight->bbcode).

I may need to lurk/read a bit more to sort out if separate replies (ā€œdouble postingā€) to multiple past comments is acceptable (organized diverging conversations) vs quoting multiple in one response to avoid double posting :slight_smile:

3 Likes