Wednesday, December 16, 2020

Keyboard Scan Code and Keyboard Handler


Hi there. In this article, I will explain how keyboards work.


Keyboard Scan Code
When a key is pressed on the keyboard, the keyboard does not transmit to the computer, which key is pressed such as 'N', '5' or so. If this were the case, it would be impossible to change keyboard layouts to English, Turkish or Russian etc. Instead of that, keyboard sends a key sequence number to the computer like first key pressed, eighth key pressed and so on. These key sequence numbers are called keyboard scan codes. These codes are literally the sequence numbers of the keys on the (US) keyboard. For example scan code of ESC is 1, top row keys starting from 1 to Backspace have 2 .. 14, second row starting with Tab and ending with Enter have 15 .. 28 etc. Although " (double quote) key on the Turkish keyboard is left to the 1 key, its scan code is 41 because this key is located next to Enter in the middle row on a standard AT keyboard.

Original IBM AT Keyboard (Image source: https://de.wikipedia.org/wiki/Datei:AT_keyboard.jpg)

In fact, the scan code of ESC key is 1 even though, it is located on the left side of the US AT keyboard. This is because its predecessor XT keyboard has ESC key at the top left side (like we are using today). Please note that there are no F11 and F12 keys on both keyboards. I will cover this later.


The Evolution of the Keyboard
The standards of the keyboards we use today, go back to the IBM PC's Model F keyboard which was released in 1981 to the market. IBM PC XT, which was released in March 1983, also had the same keyboard. These models don't have a specialized keyboard controller so the keyboards are not programmable. Being programmable means capabilities such as hardware self-testing, resetting of setting parameters such as key repeat frequency. This keyboards only transmit the scan codes to 8048 peripheral chip on the motherboard.
 
IBM PC AT, which was released in 1984 to the market, came with PC AT keyboard (shown in the image above). With this model, IBM changed the scan code set used in in PC XT keyboard, and added 8042 keyboard controller to the mainboard. This controller was only responsible for the keyboard and programmable features mentioned above, are added to the keyboard. Due to the backward compatibility issues, these keyboards could be operated with XT protocol thanks to a switch added to the keyboard. SysRq key is also added to this model. 

The standard 101-key keyboard (Model M) is released in April 1986. Keys such as Insert, Home are added and two new function keys F11 and F12 were also introduced.

IBM Model M keyboard (Image source: https://en.wikipedia.org/wiki/File:IBM_Model_M.png)

And finally, Microsoft added 3 more keys to the keyboards with Windows95: left and right Windows keys and right click menu key.


Make Codes, Break Codes and Different Scan Code Sets
Keyboard communicates with the computer via 60h and 64h IO ports. 60h is the data port and 64h is the command port where keyboard controller commands are sent. When a key is pressed on the keyboard, it sends the scan code of the key pressed, to the 8042 controller on the mainboard. This code is called make code. When the key is released, the break code is sent. However there are three different code standards.

Note: 60h is the port to which PS/2 mouse is connected as well. 

The first standard, called Set 1, is the standard of PC XT, I mentioned above. This is left as compatibility mode. The second standard (Set 2) comes with PC AT. 8042 controller translates set 2 codes to set 1 codes and transmits them to OS through interrupt controller. But the behavior of 8042 can also be changed. Finally, set 3 was released in October 1983 with IBM PC 3270 and this set is partially compatible with set 2. Set 3 codes are also translated into the first code set by 8042 like set 2. In short, different code sets transmitted by the keyboard, appear as just a single set on the OS side. When I connect an oscilloscope to the data pin of the keyboard, the signal I see is not the same as the value read from port 60h (unless I have an XT keyboard).

I briefly mentioned make codes of set 1 above. Break codes are obtained by doing OR operation on make codes with 80h. e.g. make code of ESC is 01h, its break code is 81h. Make code of Enter is 1Ch, break code is 9Ch. With this consideration, it can be thought that a keyboard can have a maximum of 127 keys (without zero code). But some keys generate more than one scan code. These codes are called extended scan codes beginning with 0E0h. As an example of these keys are Enter and '/' (slash) keys on the numerical keyboard. Pressing them generates 0E0h 1Ch and 0E0h 35h respectively. The break code of a key with extended code is obtained by ORing the second byte with 80h, like 0E0h 9Ch and 0E0h 0B5h. By the way, although entire 0EXh block is actually reserved for extended keys, almost all extended code keys use 0E0h prefix and in practice there are a small number of keys using 0E1h prefix.

Scan codes can be viewed in linux using showkey command with -s parameter. Eg:


To view scan codes in DOS, I wrote a simple code. First of all, VirtualBox does not recognize extended scan code set. I developed and tested the code in vmware. The code runs also find in DOSBox, but I got some strange errors while developing it in TurboC in DOSBox (this might be also because of me). However, the code runs stable in DOSBox even than vmware.
 
In the screenshot above, I pressed Enter, F10, F11 and '/' on the numerical keyboard, in given order. These codes in the output are the codes translated into first set, obviously. I terminated the program by pressing ESC. There are two keyboard controller commands in this code which I am writing to 64h port. 0ADh deactivates the keyboard, and 0AEh activates it again. In the "Evolution of the Keyboard" section, I mentioned that there were ten function keys on XT keyboards. F11 and F12 were added with ATs. That's why, while the scan codes of F1 .. F10 keys are between 3Bh and 44h, the scan code of F11 is 57h and F12's is 58h. The source code of this program is below:

#include <stdio.h>

unsigned char readport()    {
    unsigned char r;
    asm    {
        mov     dx, 0x0064   // disable keyboard
        mov     al, 0xAD
        out     dx, al
        push    dx

        mov     dx, 0x0060   // read from keyboard port
        in      al, dx
        mov     r, al

        pop     dx
        mov     al, 0xAE     // enable keyboard
        out     dx, al
    }
    return r;
}


int  main(int argc, char* argv[])    {
    unsigned char r, r0;

    for( ;; )    {
        asm { cli }     // disable interrupts
        r = readport();
        if((r ^ 0xE0) < 0x10)   {
            // if the code read is an extended code
            printf("--- Ext. Key: %02X  ", r);
            // read one more byte:
            printf("Escaped Char: %02X  \n", r0 = readport());
        }
        else if (r0 != r)       {
            // if scan code is different than the previous one
            printf("%02X\n", r);
        }

        if(r == 0x01)   {       // escape with ESC key
            asm { sti }
            break;
        }

        r0 = r;
    }

    return 0;
}


In set 2, keys pressed are represented by one-byte codes. Codes in this set are mixed order, i.e. ESC is 76h, '1' is 16h, '2' is 1Eh and so on. The scan code layout is given below: 

Image source: Wikibooks

Codes starting with 0F0h in this image are break codes of set 2. As it can be understood, the break code is represented by the same make code, preceded by 0F0h. ESC is 76h and its break code 0F0h 76h. Extended scan codes also start with 0E0h in this set. e.g. make code of left windows key is 0E0h 1Fh and its break code is 0E0h 0F0h 1Fh. 0E0h precedes the release code as in set 1.

Interestingly, there are no extended keys in set 3. Scan code layout of set 3 is viewed with this link. Detailed information about these sets and conversion table can be accessed with this link.

8042 keyboard controller normally finds the type of keyboard at the boot time and converts the scan codes coming from the keyboard. None of the present-day OSes use the first set anymore. For example, in atkdb.c keyboard driver of linux, atkbd_select_set( ... ) function return either 2 or 3. The value returned by this function is written to the element "set" in the struct "atkbd".

8042 can be configured to which set the incoming codes will be translated into. Current set is queried by writing 0F0h, 0 to port 60h. The value 43h, 41h or 3Fh returns for the first, second or third set respectively. After sending 0F0h; 1, 2 or 3 can be written to 60h port. Thus 8042 will start sending the data in the requested code set. Of course, when you do this in DOS, keyboard driver will try to interpret the codes that are belonging to a different set and the system will be unusable until it is restarted. On the other hand, showkey -s command in linux switches the keyboard to raw mode temporarily.


Communication with the Keyboard
Keyboards were connected with five pin DIN connectors until IBM PC ATs. I think, nowadays it is impossible to find these keyboards except for the flea market (or dumpsters). IBM developed PS/2 connector in 1987. There still are people who remembers that, it was a six-pin connector, purple for keyboard and green for mouse. If they are not thrown away, they can be found in cellars or warehouses. Today, almost all of them are connected over USB.

Signalling in PC/2 and DIN connectors is exactly the same. Both have Vcc and GND as well as CLK and DATA pins. The data flows synchronously (bidirectional serial synchronous transmission) with CLK signal. With each keystroke, the keyboard transmits a scan code over DATA pin synchronously with CLK signal.

Although this information is not necessary to do anything on the computer, it may be necessary to use a PC keyboard with Arduino for example or with other development boards.


What Happens When a Key is Pressed?
I already mentioned that the scan code is sent to the controller when a key is pressed on the keyboard. So, let's take a look at what happens at the CPU level.

First, the keyboard controller signals the nineteenth pin (IRQ 1 pin) of the primary 8259 interrupt controller (aka IRQ controller) and notifies processor of the interrupt request. If the processor is in 8086 mode, it calls int 09. However this IRQ must be handled by another interrupt if the processor is running in protected mode. Changing interrupt number is achieved by reprogramming the PIC.

Normally, int 09 code is loaded by BIOS. If required, OS can write a wrapper for on it or change it completely. When BIOS is done (pre-boot phase), int 09 vector points to address 0F000h:0E987h. This interrupt deactivates the keyboard temporarily and reads the scan code from port 60h. After that, Int 15h/4Fh is called. This function is normally used to change the scan code of a key (for example swapping Ctrl and Fn key)*. Next steps vary depending on the key pressed.

There is a flag in BIOS data area (BDA) for Shift, Ctrl and Alt keys at the addresses 0040h:0017h and 0018h. There are also flags for CapsLock, NumLock, Pause etc. as well.

Sources: Ralf Brown Int. List and lowlevel.eu

 
A similar flag is found at 0040h:0096h and 0097h:

Source: Ralf Brown Interrupt List

If the pressed key is one of the special keys mentioned above, int 9 arranges the related bits on above flags. For example, when right shift is pressed, scan code 36h is read and bit 0 at the address 0417h is set. When this key is released, scan code 0B6h is read and bit 0 at the address 0417h is reset by the BIOS.

When a regular key without any special purpose is pressed, it is important to distinguish whether this key is pressed alone or with a combination of another key (like Ctrl for example). Key combinations are distinguished with the flags above. Int 09 uses a table to calculate the pressed keys ASCII code. Let's assume, only the key 'A' is pressed. The scan code read is 1Eh and the corresponding ASCII code is 61h for 'a' (small letter). However, when Shift+A is pressed, although the scan code to be read is still 1Eh, the shift key flag is already active (scan code of Shift key is still read from 60h port) and with that flag active, int 09 interprets its ASCII code as 41h (capital A). Same thing happens when CapsLock is on and Shift is not pressed. When Ctrl+A is pressed, ASCII code is translated as 01h. For this purpose, the conversion table in BIOS consists of 128 lines for 128 scan codes. Columns are added to this table for a key's different combinations with Shift, with Ctrl and so on.

Scan code to ASCII translation is done for almost every key and pressed key is stored in the keyboard buffer. Since keys such as function keys, Home, PgUp etc. do not have ASCII code, their ASCII codes are stored as 0 in the keyboard buffer. Keys like Ctrl, Alt or NumLock are not stored. Keyboard buffer is located at the 1Eh offset in BDA and is 32 bytes in size. Since one scan code and one ASCII code are stored for each pressed key, up to 16 keys can be stored here. This data structure is a ring buffer. Its starting address is defined at 1Ah and ending address is at 1Ch in BDA. Since this is a ring buffer, the address in 1Ah may be greater than the address in 1Ch. When a keystroke is buffered here, the word consisting of the scan and ASCII codes of the pressed key is written to the address pointed by 1Ch and this pointer is increased by 2. If the value gets greater than 3Eh, which is the end of ring buffer, 20h is subtracted from this value. The pointer starts pointing to the address 1Eh again. Subsequently pressed key can be written to the a lower memory address than the key pressed before. When the value in 1Ch will be equal to the value in 1Ah after increase, it means that the keyboard buffer is full. The user pressed already 16 keys, BIOS queued them, but OS did/could not read them from the buffer. Some BIOSes warn the user in this case with sound from the tiny speaker on the mainboard. The value in 1Ah is decreased by two for each key read from the buffer.

So far, the scan code has been read from the port, translated into ASCII and written into the buffer. After that, the keyboard is activated again by int 09 and the interrupt ends. It returns the execution back to the program which is running before the key is pressed.

I compiled these steps I explained above from VirtualBox's source code. VBoxBiosAlternative386.asm is assembly output of VBox BIOS. In the rightmost column of the file exist the memory address of the current line of code and source code file generating that assembly command. 0F000h:E987h, the address of int 09, starts at line 18 268 in this file and its source code is at the line 974 of orgs.asm file. Deactivation of the keyboard and int 15h/4Fh call happens here. Flags for extended scan codes are arranged here (line 1011) as well. Then, int09_function function is called which is in keyboard.c file (line 371). Assembly output of this function is on line 7501 in the VBoxBiosAlternative386.asm file. This function sets or resets the related flags in a big switch/case block, when a key with special function is pressed. When a regular key is pressed, it finds the corresponding ASCII code of the key from scan_to_scanascii structure defined in keyboard.c:90. Each row in this table contains the codes of a key in case it is pressed alone, with Shift, with Ctrl and with Alt, respectively. The last element of each row indicates that the key changes its state with lock keys. 20h means with NumLock, 40h means with CapsLock. The scan-ASCII code pair is then transferred to keyboard buffer by calling enqueue_key function in line 604. This function is at line 339 in same file (and in VBoxBiosAlternative386.asm:7454 as assembly output).

In fact, at the 80h and 82h offset of BDA, there is another pair of start and end pointers for an additional keyboard buffer. This pointers were defined with AT BIOS as an alternative to the 16-key limited buffer of XT BIOS. However when ATs came out, so many programs on the market were using the old buffer hard-coded, that these new pointers became dysfunctional over time. Even VBox doesn't care about these pointers.

* When a keyboard layout is loaded in DOS, int 15h/4Fh checks whether that scan code exists in a scan code - ASCII table corresponding to new keyboard layout. If the code exists in this table, int 15h/4Fh passes the scan and ASCII codes (instead of int 09) of the key into the keyboard buffer and bypasses the execution of the rest of int 09 code by resetting carry flag on return.


Reading from Keyboard Buffer
I did not mention above any way to read from the keyboard buffer, delivering the keystrokes to a user program or OS. Accessing to the keyboard port is not an efficient method for obvious reasons, e. g. keyboard may be a non-US keyboard and controlling all those special keys like Alt, CapsLock etc. makes the code unnecessarily complicated. In addition to buffering keystrokes, BIOS also provides functions for fetching them from the buffer. These functions are provided under int 16h.

The first function is the zeroth function called while AH=00. Its job is, if there is any buffered keystroke in keyboard buffer, to return it in AX and remove it from the buffer. If the buffer is empty, the function waits until a key is pressed. Next function, called while AH=01, returns the keystroke in AX if there is any buffered key but does not remove it from the buffer. If there is no key in the buffer, unlike the zeroth function, it ends without waiting for any input. On return, it sets the zero flag if no keystroke was read.
 
These two functions above, are XT functions and compatible with XT key set. In other words, keys such as F11, F12, Home, Insert etc. that came with the AT keyboard cannot be fetched with these functions. With PC AT series, functions 10h and 11h have been added to int 16h. The job of the function 10h is the same as 00, and it also reads the keystrokes of new keys of the AT keyboards. Likewise, function 11h is compatible with function 01, too. If the function 00 is called and F12 is pressed, nothing happens, however if function 10h is called, it will return the scan code of F12.

Similarly, 20h and 21h functions are implemented for 122-key keyboards, but these functions are not available on many computers because these keyboards are not common.


A Simple Demo
I wrote a simple code below to explain what I wrote about the keyboard buffer. The program prints the BDA in an infinite loop. Since the keyboard buffer is here, the buffering of keystrokes can be clearly seen in real time. Q terminates the program and ESC clears keyboard buffer. Source code is given below:

#include<stdio.h>
#include<stdlib.h>

#define MEMLINES 10
#define MEMSIZE (MEMLINES * 16)

// return current page
unsigned char getpage()     {
    unsigned char p;
   
    asm     {
        mov     ah, 0x0F
        int     0x10
        mov     p, bh
    }

    return p;
}

// return current cursor position
void getline(unsigned char *x, unsigned char *y)    {
    unsigned char tx, ty;
    unsigned char p = getpage();

    asm     {
        mov     ah, 0x03
        mov     bh, p
        int     0x10
        mov     tx, dl
        mov     ty, dh
    }

    *x = tx;
    *y = ty;
}

// set given cursor position
void setline(unsigned char x, unsigned char y)      {
    unsigned char p = getpage();

    asm     {
        mov     ah, 0x02
        mov     bh, p
        mov     dl, x
        mov     dh, y
        int     0x10
    }
}

// listen on port 60h and return keypress and key release values
char keypress()     {
    unsigned char r;

    asm     {
        in      al, 0x60
        mov     r, al
    }

    return r;
}

// clear keyboard buffer by copying last key pointer over first
// key pointer i.e. 0040h:001Ch <-- 0040h:001Ah
void clearKBbuffer()    {
    unsigned char far *startKBbuffer = (unsigned char far *)0x0041C;
    unsigned char far *endKBbuffer = (unsigned char far *)0x0041A;

    *startKBbuffer++ = *endKBbuffer++;
    *startKBbuffer   = *endKBbuffer;

}


int main(int argc, char* argv[])        {
    int i, j;
    unsigned char curX, curY;
    // memblock points to the start of BIOS Data Area
    unsigned char far *memblock = (unsigned char far *)0x00400;

    getline(&curX, &curY);

    while(1)    {

        for(i = 0; i < MEMLINES; i++)   {
            // print offsets here:
            printf("%06X  ", memblock + (i << 4));

            // hexadecimal block is printed below:
            for(j = 0; j < 16; j++)     {
                printf("%02X ", memblock[(i << 4) + j]);
                if(j == 7)  printf("- ");
            }

            printf("  ");

            // char block is printed below:
            for(j = 0; j < 16; j++)     {
                    if((memblock[(i << 4) + j] < 32) ||
                       (memblock[(i << 4) + j] > 128))
                        printf(".");
                    else
                        printf("%c", memblock[(i << 4) +j]);
            }

            printf("\n");
        }
       
        setline(curX, curY);

        if (keypress() == 0x10)     // if Q key is pressed
            break;                  // break the infinite loop
        if (keypress() == 1)    {   // if ESC key is pressed
            clearKBbuffer();        // clear keyboard buffer
            // break;
        }
    }
   
    return 0;
}


I wrote this program in TurboC. Since TurboC can be difficult to find, its executable file can be downloaded together with its source code using this link. This program can be run in a VM as well as in DOSBox. When running in DOSBox, getline() and setline() do not work properly when the prompt is at the bottom line. It needs to be run after a cls command. The output then looks fine.

The image on the right is the screenshot, when I run the program first. The start and end pointers 1Ah and 1Ch contains 38h. Since these are equal, this means the buffer is empty. The characters I used while typing the name of the executable, still remain in the buffer. There is an 'o' character at the address 1Eh, followed by a Tab key whose ASCII code is 9 and scan code is 0Fh. While I was typing the command, I typed "pro" instead of "project" and pressed Tab for completion (FreeDOS). Then I typed, "readmem" and pressed Tab again, but this time since there are both readmem.exe and readmem.obj files in the same directory, only a dot character appeared at the prompt. Then, I pressed 'e' and Tab again and the key with scan code 1Ch, which is Enter and thus I ran the program. The scan code 1Ch is at the offset 37h, therefore the pointers are pointing at 38h.

I pressed 'wertyu' keys in turn. 'wer' is located between 38h and 3Eh which is the end of buffer. Then the ring buffer rewinds to the beginning and the remaining 'tyu' keys are buffered between 1Eh and 24h. I clear the buffer by pressing Esc (by assigning the value in 1Ah to the value in 1Ch) then I type 'asdf' (image below). By doing this, 'wer' keys are overwritten by 'asd' keys and 'f' is written to the first byte of the buffer (1Eh, overwriting 't'). Finally, I quit by pressing 'Q' without clearing the buffer. Since, I did not fetched the keys using int 16h, when the program ends, 'asdf' and 'q' keys (which are already in buffer) are read by DOS and displayed at the prompt.


The things that can be done with the keyboard are not limited to these. What is explained here, can only be an introduction to keyboards. There are dozens of controller commands, not limited to activate/deactivate keyboard. With these command, for example keyboard can be reset or hardware test can be done. As I mentioned previously, PS/2 mouse is also connected through port 60h. The only difference is that while keyboard creates IRQ1, mouse creates IRQ12. Activating or deactivating mouse is also done by sending commands to port 64h.
 

References: 
In addition to the references given in the text:


Posted by at
Labels: , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)