FreeDOS Invalid Opcode Problem in VirtualBox and Transfering Files through Network

Hi there. I have written about FreeDOS dozens of times in previous posts and installed it many times as well, especially in disk and file system posts. I never had a problem with it before.

Earlier this year, I formatted my disk and installed Fedora. Since its repos are up-to-date, I had the chance to install latest versions of programs, I am using. One of these programs is VirtualBox (VBox). I was using v5.x before and now v6.1.20 is currently installed. Its most recent version was v6.1.22 while I am writing this post. However, FreeDOS has crashed, while booting it in new VBox. I was even unable to see the prompt, and had to figure out a solution to this. First half of this article will be about this solution.

I wanted to set up a new VM for the upcoming FAT32 blog post. While creating the VM, I chose DOS as machine type. In fact, machine type is automatically set to DOS, in case VM name contains "DOS". I created a 4 GB disk to guarantee that the disk is formatted as FAT32 (>2GB). I mounted FD12CD.iso and as soon as I powered the VM up, I got this "Invalid Opcode" error which I mentioned:

This error actually occurs during the execution of HIMEMX.EXE. First, I entered step by step (trace) boot mode by pressing F8 during the boot and found the origin of the error. Then I reset the VM again and skipped the step where HIMEMX.EXE is loaded, but this caused another problem with installation because the text user interface of installation is also created with a .bat file. I simply lost myself in trace mode between the commands which creates the menu and waits for user input to continue.

Apparently, this is a known bug and was reported in 2018 first. I never encountered this problem before because I installed FreeDOS previously on the old version of VBox. By the way, the previous version, I was using, was definitely 5.x but I can't remember exactly. And although there is a major version difference between them, this problem is first reported in VBox v5.2.10.

Solution

Frankly, I could not understand whether this bug is in FreeDOS or in VBox, because same FreeDOS boots in Vmware without any problem but on the other hand, it is announced in the bug ticket, that this bug will be solved in FreeDOS v1.3*. This problem has two different solutions: (1) booting the VM with FD12LGCY.iso instead of FD12CD.iso or (2) appending "raw" to the kernel parameters. Although I got a warning while booting legacy iso, I had no problem with the installation. The warning was: "Unfortunately, this method of installation is not supported on this hardware platform. Please try a different installation method." (screenshot above). I did not encounter any warning in vmware. Despite the above warning, I partitioned the disk first with fdisk when I landed to prompt, then rebooted the VM and started installation with setup command.

For the second option, I started the VM with FD12CD.iso and pressed Tab key at the "Welcome to FreeDOS 1.2" screen, appended "raw" with a leading space and started boot process by pressing Enter. This time, the installation wizard started automatically but since I needed to partition the disk first, I chose "Return to DOS" at step two and quit.

If partitioning is done by setup, it automatically creates 2x 2GB partitions. I do the partitioning manually to create a single partition. After seeing the prompt, I ran fdisk and answered the question about FAT32 support as "Yes" by pressing Y, I created a single 4 GB partition with 1, 1, Enter, Esc key combination and restarted the VM. Then, I selected "Erase and format drive C:" and "Full Installation" options in setup. Of course, setup command needs to be run first, in case the VM is booted with legacy CD. After installation is done, I removed CD and restarted the VM. Formatting disk manually isn't necessary, because for some unknown reason, the wizard formats the already formatted disk, anyway.

This is the solution of "Invalid opcode" problem.

* While I was writing this post, it was announced on FreeDOS mailing list that v1.3 RC4 will be released. The last time I checked, freedos.org was also updated.


Copying Files to FreeDOS VM
I encountered the second problem while trying to copy files to FreeDOS. Normally, I used to start the VM with Damn Small Linux (DSL)  and copy files with SSH from host. I couldn't connect to DSL because some insecure SSH ciphers were disabled in new Fedora (supposedly).

First of all, there was a constraint in this problem: In VBox, host (hypervisor) can't access to NATted VMs (or maybe I failed) but NATted VMs can access to their host. In vmware, for each virtual network, a network card (NIC) is created on host and it is possible to reach every subnet using the IP address of corresponding NIC however this isn't possible in VBox NAT. When using DSL, I used to initiate SSH session from VM (or create a Host-only Network to connect in opposite direction). Now, if I want to do the same, I have to update the ciphers in DSL. Let's keep this aside.

On the other hand, I didn't mention it before, but FreeDOS actually comes with many network tools: NIC device driver, TCP/IP stack, ethtools (packet analyzer), mtcp (tools like dhcp, irc, ftp, netcat, sntp etc.) ve sshdos. Even though, I am not very hopeful, I thought maybe I could ssh from FreeDOS to Fedora. If I can, I can scp too. I attached FreeDOS CD (doesn't matter legacy or normal) to install the tools, I mentioned and run

fdimples

command. This opens the text interface of FreeDOS package manager. I browsed to "Networking" and selected fdnet, mtcp and sshdos packages (with Tab and spacebar). Even though, curl and wget can also be used for file tranfer, the files are not on the internet but on my disk and I don't want to install httpd to my computer. FreeDOS has also rsync, but it doesn't support ssh.

I picked the packages, I need, and installed them via Tab, Tab and Enter key sequence. Then I removed the CD and rebooted the VM. During the boot, I saw "VirtualBox network detected" message and the IP assigned by DHCP. When I changed to SSH directory and gave sshdos command (image on the left), I saw that only 3des and blowfish ciphers are supported. Even though, I added "Protocol 2,1" and "Ciphers 3des-cbc" settings to /etc/ssh/sshd_config file in Fedora (host), I couldn't overcome "Unsupported remote protocol version" error.

Solution
I cannot connect to the host via ssh and I cannot connect to the VM from host either. First, I have to change network settings to connect to VM from my host:

VM network settings

At this step, I have three options:
1- Host-only Adapter: With this option, VM will be connected to a subnet in which the host also exists, but I skip this option as this network needs to be created first with VBox manager.
2- Bridged Adapter: With this one selected, VM will be connected to the physical subnet where the host is also connected. This is a hassle-free solution, but can be problematic in administered networks such as corporate networks. If there are more than one NIC on a machine, it is necessary to select the card which is connected to the network. Additionally, if this option is selected after NAT, running \FDOS\bin\fdnet.bat or restarting the VM is needed, to renew its IP.
3- NAT + Port Forwarding: When this option is enabled, a port is opened on the host and all packets are forwarded to VM through this port. To configure this, you should open Advanced options while NAT is selected and a free port number is given for each side on "Port Forwarding" window. Since DOS is running as guest OS, Guest port can be any port but it will be good to have a value greater than 1024 as Host port. Ports in use can be listed with netstat -tln command under linux (and it should be netstat -anl under windows). I set both ports to 1500.

 

 
After changing network settings, I can connect to VM but how can I transfer the data? There is an FTP server in mtcp package but I would rather use netcat instead of configuring that FTP server. On Fedora, nc was already installed (from nmap-ncat package). On Windows, it is shipped with nmap portable. Although telnet is sufficient for testing connection between host and guest, nc must be installed for file transfer. First, I run

nc -listen 1500

in MTCP directory for testing. 1500 is the "Guest Port" value here (in case Guest Port and Host Port are not equal). Then I run following command on host (for NAT option):

nc localhost 1500

For "Bridged Adapter" option, hosts IP has to be given instead of "localhost". After that, typed characters on host must be exactly shown in VM after pressing Enter. I closed nc with Ctrl+D and gave following command on DOS for file transfer:

nc -bin -listen 1500 > FILEN.AME

and I transferred the file with following command

nc localhost 1500 < FILEN.AME

on linux. Now, I can check their integrity with md5sum command on both machines and verify that they are same files.


Alternate Solution (SSH)

I changed the network configuration of the VM like before (NAT or Bridged). I additionally set "Guest Port" to 22 if NAT is used. This value should not mandatorily be 22, but nice to have (will explain why). I powered the VM with DSL and run sshd from DSL menu (image on the right). I also gave a password to dsl user using sudo passwd dsl command on terminal. Assigned IP address is displayed on the right of the desktop. I only need it for "Bridged Adapter" configuration. In NAT configuration, this IP should start with 10.0.2. but it is not necessary because I will be connecting to localhost anyway for NAT.

If the "Guest Port" is different than 22, I have to change sshd configuration using sudo nano /etc/ssh/sshd_config, find the line starting with "Port", change it to the "Guest Port" value (e.g. Port 1500). Then I save the config file and apply the change with sudo /etc/init.d/ssh restart command. Therefore, setting "Guest Port" to 22 in NAT settings is pretty much easier.

Even though, machines can ping eachother, SSH is in both directions problematic. I cannot connect to host from DSL. Although I listed supported ciphers on my machine using ssh -Q ciphers command and gave them to ssh command as arguments of -c parameter, I still could not connect. For some algorithms, I got "Unknown cipher type" error which means that ssh on DSL doesn't support these algorithms. Fedora is of course more up-to-date than DSL. Instead of trying to install missing ciphers on DSL, it would be easier to fix the problem on Fedora (I suppose). I was hoping that, Fedora still supports outdated algorithms for backward compatibility even though it doesn't allow connecting with them. Btw, ssh client on DSL is old that it doesn't even support -Q parameter. Anyway, my first try:

[user@host ~]$ ssh -p 1500 dsl@localhost
Unable to negotiate with 127.0.0.1 port 1500: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

Btw, the above command has to be like ssh dsl@<VM_IP> for "Bridged Adapter" configuration. DSL wants to use key exchange methods listed above, only. So, I entered it manually and tried again:

[user@host ~]$ ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -p 1500 dsl@localhost
Unable to negotiate with 127.0.0.1 port 1500: no matching cipher found. Their offer: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se

Now, it wants that only these listed ciphers to be used for connection. So, I manually entered one of them and tried again:

[user@host ~]$ ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -c aes128-cbc -p 1500 dsl@localhost
The authenticity of host '[localhost]:1500 ([127.0.0.1]:1500)' can't be established.
RSA key fingerprint is SHA256:[SNIP].
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
[SNIP]

It finally worked! In this case, I can copy the file with scp by passing same parameters to it but -p parameter of ssh corresponds to -P parameter of scp:

[user@host ~]$ scp -oKexAlgorithms=+diffie-hellman-group-exchange-sha1 -c aes128-cbc -P 1500 filen.ame dsl@localhost:~
dsl@localhost's password:
filen.ame                   100%  543   530.9KB/s   00:00  

Since I copied the file to the home directory of dsl user, I returned to DSL terminal and gave:

sudo su
mount /dev/hda1 /mnt
cp filen.ame /mnt
umount /mnt

commands to copy the file to the disk where FreeDOS in installed.

I came across another alternative solution on FreeDOS wiki: If the format of the virtual disk is .vhd (MS HyperV format) and if libguestfs-tools is installed, virtual disk can be mounted using guestmount command. However this package and its dependencies require 137 MB of space on Fedora. So, really not worth to install. On the other hand, Windows can open .vhd files like normal hard drives. Virtual disk files can also be converted with qemu-img command between vhd, vdi (VBox) and vmdk (vmware) formats but this would be the least preferable solution because it is over-complicated.


References:
(except the links within the text)
http://wiki.freedos.org/wiki/index.php/VirtualBox_-_Chapter_4
http://wiki.freedos.org/wiki/index.php/VirtualBox_-_Chapter_6
How to enable diffie-hellman-group1-sha1 key exchange on Debian 8.0?

Bash while Loop and Its Strange Behavior with Stdin

Hi there. In this post, I will describe an interesting behavior, that I noticed in a while loop in one of my bash scripts about 5 years ago. I recently encountered something similar again. Fortunately, I solved it easily this time, but I think it deserves to be investigated in a post. So, what is this interesting behavior?

Let's consider a simple bash script with a while loop:

# first example
while read -r line; do
    echo $line
    sleep 3
done < input.file

input.file is a simple text file which contains more than one lines. For example, a file which was simply created with ls -1 > input.file command. The script does read these lines from the file line by line and prints them with 3 second intervals. So far, so good. Another script, that does the same in a different way:

# second example
cat input.file | while read line; do
    echo $line
    sleep 3
done

Even though the code is different, they are the same in the sense of black box approach according to its input and output. Finally, another example which does the same thing, too:

# third example
for line in $(cat input.file); do
    echo $line
    sleep 3
done

Even though a while loop is not used above, it still is an alternative to the previous examples as it generates the same output from the same input.

Nothing strange so far, but the strange behavior will show up, when I try to read from stdin in the loop. I read the input from stdin using read command. For example, I wait for a keypress after each line of input.file. In order to do this, I replace sleep command with read -t 10 -n 1 dummyvariable. The parameter t is the input timeout in seconds and the parameter n is the character count of the input. The variable name is not important because I will not use it anywhere. As mentioned before, my input files consists of ls -1 output (12 lines).

Arduino
Desktop
Documents
Downloads
Music
Pictures
Public
Templates
Videos
input.file
logs
test.sh

The execution of the script below should take 120 seconds from 10 seconds per line, if I don't press any key. The script, I am running:

while read -r line; do
    echo $line
    read -t 10 -n 1 dummyvariable
done < input.file

But this is not executed as I expected. All the first characters of all lines got deleted except the first line and it ended in less than a second. But why?

To inspect the execution, I inserted a sleep 10 command under read and opened another bash tab.

[user@host ~]$ ps aux | grep test | grep -v grep
user   12001  0.0  0.0 222344  3612 pts/1    S+   16:23   0:00 /bin/bash ./test.sh

[user@host ~]$ ls -la /proc/12001/fd
[SNIP]
lr-x------. 1 user user 64 Apr 29 16:26 0 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 16:26 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 10 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 16:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 16:26 255 -> /home/user/test.sh

When I checked the file descriptors of the running process, I saw that the zeroth descriptor, which has to be normally stdin, is redirected to input.file and this is actually me, doing it. As soon as I gave "input.file" as input to while, stdin changed. Therefore, I cannot use stdin as input source in while anymore.

This is not a problem or an error. That's why, I avoided using the word "problem" since the beginning of this post. This is totally an expected behavior, but still strange when a file descriptor just changes within a loop, considering loop practices of other programming languages like python or C.

When I replace sleep command with read in the second example, same behavior occurs in a different way. First line printed normally, but first characters are still missing in the rest of output. In this example, I used pipe instead of IO redirection but this resulted redirection of stdin to a pipe instead of a file under the hood. To examine this, I inserted a sleep command again after read command, ran again and checked file descriptors:

[user@host ~]$ ps aux | grep test | grep -v grep
user   15635  0.0  0.0 222212  3340 pts/1    S+   19:01   0:00 /bin/bash ./test.sh
user   15637  0.0  0.0 222212  1996 pts/1    S+   19:01   0:00 /bin/bash ./test.sh

[user@host ~]$ pstree -p | grep test
 | |-konsole(10108)-+-bash(10120)---test.sh(15635)---test.sh(15637)---sleep(15668)

[user@host ~]$ ls -la /proc/15635/fd   /proc/15637/fd
/proc/15635/fd:
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:03 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:03 255 -> /home/user/test.sh

/proc/15637/fd:
[SNIP]
lr-x------. 1 user user 64 Apr 29 19:03 0 -> 'pipe:[289548]'
lrwx------. 1 user user 64 Apr 29 19:03 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:03 2 -> /dev/pts/1

The main difference here is, that test.sh is executed multiple times. PID 15635 runs cat. Then the spawned process with ID 15637 executes while loop and the output of first process is sent to the stdin of the second process via pipe. In short, although the algorithm is different, it basically produces the same result.

In this third example, there are no pipe nor redirect. Using cat, the contents of input.file is dumped to memory and processed by for loop. As it can be seen below, the descriptors are not changed.

[user@host ~]$ ps aux | grep test
user   15955  0.0  0.0 222212  3336 pts/1    S+   19:13   0:00 /bin/bash ./test.sh

[user@host ~]$ ls -la /proc/15955/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 19:13 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 1 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 19:13 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 19:13 255 -> /home/user/test.sh

Hence, a for loop might be preferable over while in some cases. Below is another example of such behavior:

cat ^testuser /etc/passwd | cut -d':' -f 1 | while read -r line; do
    sudo passwd $line
done

Let's assume, there are three users in this system named testuser01, ...02, ...03. When trying to change their passwords, passwd testuser01 command is executed at first iteration, then the string "testuser02" is entered as a password to testuser01 (as an input to passwd testuser01 command) and when passwd asked for the same password second time (for confirmation), "testuser03" is entered as input and so passwd fails. The loop is completed but passwd command has only been executed once.

So, how can this be done with while? In this case, file descriptor needs to be assigned manually. Frankly, I have only used this technique once before for a different purpose: Read n lines at a time using Bash. What's done here is a bit advanced but quite elegant at the same time as it shows how flexible bash is. By the way, the accepted answer is IMHO too complex to solve "read n lines" problem (see other answers) but same approach can be applied to this situation, I am mentioning: In the first example above, zeroth descriptor (which is stdin normally) was redirected to the file and stdin was allocated with tenth descriptor. In that case, I can guarantee that stdin gets the zeroth descriptor by opening the file with another descriptor using the method described in the answer.

exec 10< input.file
while read line <&10; do
    echo $line
    read -t 10 -n 1 dummyvariable
done

Below is a snapshot of the file handles while the script is running:

[user@host ~]$ ps aux | grep test
user   17537  0.0  0.0 222212  3408 pts/1    S+   20:26   0:00 /bin/bash ./test.sh

[user@host ~]$ ls -la  /proc/17537/fd
[SNIP]
lrwx------. 1 user user 64 Apr 29 20:26 0 -> /dev/pts/1
lrwx------. 1 user user 64 Apr 29 20:26 1 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 10 -> /home/user/input.file
lrwx------. 1 user user 64 Apr 29 20:26 2 -> /dev/pts/1
lr-x------. 1 user user 64 Apr 29 20:26 255 -> /home/user/test.sh

The file was opened with tenth descriptor, just as I wanted and read command could read data from stdin successfully.