Hi there. In this post, I will describe an interesting behavior, that I noticed in a while loop in one of my bash scripts about 5 years ago. I recently encountered something similar again. Fortunately, I solved it easily this time, but I think it deserves to be investigated in a post. So, what is this interesting behavior?
Let's consider a simple bash script with a while loop:
input.file is a simple text file which contains more than one lines. For example, a file which was simply created with ls -1 > input.file command. The script does read these lines from the file line by line and prints them with 3 second intervals. So far, so good. Another script, that does the same in a different way:
Even though the code is different, they are the same in the sense of black box approach according to its input and output. Finally, another example which does the same thing, too:
Even though a while loop is not used above, it still is an alternative to the previous examples as it generates the same output from the same input.
Nothing strange so far, but the strange behavior will show up, when I try to read from stdin in the loop. I read the input from stdin using read command. For example, I wait for a keypress after each line of input.file. In order to do this, I replace sleep command with read -t 10 -n 1 dummyvariable. The parameter t is the input timeout in seconds and the parameter n is the character count of the input. The variable name is not important because I will not use it anywhere. As mentioned before, my input files consists of ls -1 output (12 lines).
Arduino
Desktop
Documents
Downloads
Music
Pictures
Public
Templates
Videos
input.file
logs
test.sh
The execution of the script below should take 120 seconds from 10 seconds per line, if I don't press any key. The script, I am running:
But this is not executed as I expected. All the first characters of all lines got deleted except the first line and it ended in less than a second. But why?
To inspect the execution, I inserted a sleep 10 command under read and opened another bash tab.
When I checked the file descriptors of the running process, I saw that the zeroth descriptor, which has to be normally stdin, is redirected to input.file and this is actually me, doing it. As soon as I gave "input.file" as input to while, stdin changed. Therefore, I cannot use stdin as input source in while anymore.
This is not a problem or an error. That's why, I avoided using the word "problem" since the beginning of this post. This is totally an expected behavior, but still strange when a file descriptor just changes within a loop, considering loop practices of other programming languages like python or C.
When I replace sleep command with read in the second example, same behavior occurs in a different way. First line printed normally, but first characters are still missing in the rest of output. In this example, I used pipe instead of IO redirection but this resulted redirection of stdin to a pipe instead of a file under the hood. To examine this, I inserted a sleep command again after read command, ran again and checked file descriptors:
The main difference here is, that test.sh is executed multiple times. PID 15635 runs cat. Then the spawned process with ID 15637 executes while loop and the output of first process is sent to the stdin of the second process via pipe. In short, although the algorithm is different, it basically produces the same result.
In this third example, there are no pipe nor redirect. Using cat, the contents of input.file is dumped to memory and processed by for loop. As it can be seen below, the descriptors are not changed.
Hence, a for loop might be preferable over while in some cases. Below is another example of such behavior:
Let's assume, there are three users in this system named testuser01, ...02, ...03. When trying to change their passwords, passwd testuser01 command is executed at first iteration, then the string "testuser02" is entered as a password to testuser01 (as an input to passwd testuser01 command) and when passwd asked for the same password second time (for confirmation), "testuser03" is entered as input and so passwd fails. The loop is completed but passwd command has only been executed once.
So, how can this be done with while? In this case, file descriptor needs to be assigned manually. Frankly, I have only used this technique once before for a different purpose: Read n lines at a time using Bash. What's done here is a bit advanced but quite elegant at the same time as it shows how flexible bash is. By the way, the accepted answer is IMHO too complex to solve "read n lines" problem (see other answers) but same approach can be applied to this situation, I am mentioning: In the first example above, zeroth descriptor (which is stdin normally) was redirected to the file and stdin was allocated with tenth descriptor. In that case, I can guarantee that stdin gets the zeroth descriptor by opening the file with another descriptor using the method described in the answer.
Below is a snapshot of the file handles while the script is running:
The file was opened with tenth descriptor, just as I wanted and read command could read data from stdin successfully.
Let's consider a simple bash script with a while loop:
# first example
while read -r line; do
echo $line
sleep 3
done < input.file
while read -r line; do
echo $line
sleep 3
done < input.file
input.file is a simple text file which contains more than one lines. For example, a file which was simply created with ls -1 > input.file command. The script does read these lines from the file line by line and prints them with 3 second intervals. So far, so good. Another script, that does the same in a different way:
# second example
cat input.file | while read line; do
echo $line
sleep 3
done
cat input.file | while read line; do
echo $line
sleep 3
done
Even though the code is different, they are the same in the sense of black box approach according to its input and output. Finally, another example which does the same thing, too:
# third example
for line in $(cat input.file); do
echo $line
sleep 3
done
for line in $(cat input.file); do
echo $line
sleep 3
done
Even though a while loop is not used above, it still is an alternative to the previous examples as it generates the same output from the same input.
Nothing strange so far, but the strange behavior will show up, when I try to read from stdin in the loop. I read the input from stdin using read command. For example, I wait for a keypress after each line of input.file. In order to do this, I replace sleep command with read -t 10 -n 1 dummyvariable. The parameter t is the input timeout in seconds and the parameter n is the character count of the input. The variable name is not important because I will not use it anywhere. As mentioned before, my input files consists of ls -1 output (12 lines).
Arduino
Desktop
Documents
Downloads
Music
Pictures
Public
Templates
Videos
input.file
logs
test.sh
The execution of the script below should take 120 seconds from 10 seconds per line, if I don't press any key. The script, I am running:
while read -r line; do
echo $line
read -t 10 -n 1 dummyvariable
done < input.file
echo $line
read -t 10 -n 1 dummyvariable
done < input.file
But this is not executed as I expected. All the first characters of all lines got deleted except the first line and it ended in less than a second. But why?
To inspect the execution, I inserted a sleep 10 command under read and opened another bash tab.
[user@host ~]$ ps aux | grep test | grep -v grep
user 12001 0.0 0.0 222344 3612 pts/1 S+ 16:23 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/12001/fd
[SNIP]
lr-x------. 1 user user 64 Apr 29 16:26 0 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 16:26 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 10 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 16:26 255 -> /home/user/test.sh
user 12001 0.0 0.0 222344 3612 pts/1 S+ 16:23 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/12001/fd
[SNIP]
lr-x------. 1 user user 64 Apr 29 16:26 0 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 16:26 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 10 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 16:26 255 -> /home/user/test.sh
When I checked the file descriptors of the running process, I saw that the zeroth descriptor, which has to be normally stdin, is redirected to input.file and this is actually me, doing it. As soon as I gave "input.file" as input to while, stdin changed. Therefore, I cannot use stdin as input source in while anymore.
This is not a problem or an error. That's why, I avoided using the word "problem" since the beginning of this post. This is totally an expected behavior, but still strange when a file descriptor just changes within a loop, considering loop practices of other programming languages like python or C.
When I replace sleep command with read in the second example, same behavior occurs in a different way. First line printed normally, but first characters are still missing in the rest of output. In this example, I used pipe instead of IO redirection but this resulted redirection of stdin to a pipe instead of a file under the hood. To examine this, I inserted a sleep command again after read command, ran again and checked file descriptors:
[user@host ~]$ ps aux | grep test | grep -v grep
user 15635 0.0 0.0 222212 3340 pts/1 S+ 19:01 0:00 /bin/bash ./test.sh
user 15637 0.0 0.0 222212 1996 pts/1 S+ 19:01 0:00 /bin/bash ./test.sh
[user@host ~]$ pstree -p | grep test
| |-konsole(10108)-+-bash(10120)---test.sh(15635)---test.sh(15637)---sleep(15668)
[user@host ~]$ ls -la /proc/15635/fd /proc/15637/fd
/proc/15635/fd:
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:03 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:03 255 -> /home/user/test.sh
/proc/15637/fd:
[SNIP]
lr-x------. 1 user user 64 Apr 29 19:03 0 -> 'pipe:[289548]'
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1
user 15635 0.0 0.0 222212 3340 pts/1 S+ 19:01 0:00 /bin/bash ./test.sh
user 15637 0.0 0.0 222212 1996 pts/1 S+ 19:01 0:00 /bin/bash ./test.sh
[user@host ~]$ pstree -p | grep test
| |-konsole(10108)-+-bash(10120)---test.sh(15635)---test.sh(15637)---sleep(15668)
[user@host ~]$ ls -la /proc/15635/fd /proc/15637/fd
/proc/15635/fd:
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:03 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:03 255 -> /home/user/test.sh
/proc/15637/fd:
[SNIP]
lr-x------. 1 user user 64 Apr 29 19:03 0 -> 'pipe:[289548]'
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1
The main difference here is, that test.sh is executed multiple times. PID 15635 runs cat. Then the spawned process with ID 15637 executes while loop and the output of first process is sent to the stdin of the second process via pipe. In short, although the algorithm is different, it basically produces the same result.
In this third example, there are no pipe nor redirect. Using cat, the contents of input.file is dumped to memory and processed by for loop. As it can be seen below, the descriptors are not changed.
[user@host ~]$ ps aux | grep test
user 15955 0.0 0.0 222212 3336 pts/1 S+ 19:13 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/15955/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:13 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:13 255 -> /home/user/test.sh
user 15955 0.0 0.0 222212 3336 pts/1 S+ 19:13 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/15955/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:13 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:13 255 -> /home/user/test.sh
Hence, a for loop might be preferable over while in some cases. Below is another example of such behavior:
cat ^testuser /etc/passwd | cut -d':' -f 1 | while read -r line; do
sudo passwd $line
done
sudo passwd $line
done
Let's assume, there are three users in this system named testuser01, ...02, ...03. When trying to change their passwords, passwd testuser01 command is executed at first iteration, then the string "testuser02" is entered as a password to testuser01 (as an input to passwd testuser01 command) and when passwd asked for the same password second time (for confirmation), "testuser03" is entered as input and so passwd fails. The loop is completed but passwd command has only been executed once.
So, how can this be done with while? In this case, file descriptor needs to be assigned manually. Frankly, I have only used this technique once before for a different purpose: Read n lines at a time using Bash. What's done here is a bit advanced but quite elegant at the same time as it shows how flexible bash is. By the way, the accepted answer is IMHO too complex to solve "read n lines" problem (see other answers) but same approach can be applied to this situation, I am mentioning: In the first example above, zeroth descriptor (which is stdin normally) was redirected to the file and stdin was allocated with tenth descriptor. In that case, I can guarantee that stdin gets the zeroth descriptor by opening the file with another descriptor using the method described in the answer.
exec 10< input.file
while read line <&10; do
echo $line
read -t 10 -n 1 dummyvariable
done
while read line <&10; do
echo $line
read -t 10 -n 1 dummyvariable
done
Below is a snapshot of the file handles while the script is running:
[user@host ~]$ ps aux | grep test
user 17537 0.0 0.0 222212 3408 pts/1 S+ 20:26 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/17537/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 20:26 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 20:26 1 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 10 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 20:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 255 -> /home/user/test.sh
user 17537 0.0 0.0 222212 3408 pts/1 S+ 20:26 0:00 /bin/bash ./test.sh
[user@host ~]$ ls -la /proc/17537/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 20:26 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 20:26 1 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 10 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 20:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 255 -> /home/user/test.sh
The file was opened with tenth descriptor, just as I wanted and read command could read data from stdin successfully.
