HSC: Shoyler's Extremely Experimental Graphical Operating Library - SeeGOL

Author: Schuyler Martin [email protected]

About

SeeGOL (pronounced sea-gull) is my final graduate project. The end goal is to develop a graphics library/windowing system to 16-bit x86 processors (i386 and above).

Original Plan

I originally wanted to develop a library for the IBM PCjr. That has proven to be too difficult using modern tools. There is a branch, pcjr_8086_compat, which includes my attempt at converting gcc's i386 assembly to only valid instructions on the 8086.

Notes on the PCjr

The PCjr model I have has 128kb of RAM and a 3.5in floppy drive that can only read the older "1mb" IBM floppy format (as opposed to the much more common 2.44mb format most people are probably familiar with).

Setup

SeeGOL was originally built on a Fedora 24 workstation using gcc 6.3.1. Although it should be able to be built on almost any reasonable Linux system, I have run into issues with the custom image compression Python scripts running off of older versions of the Python 3 interpreter on Debian.

The Makefile is configured to warn and error-out on missing software build dependencies.

Dependencies

These programs are necessary to build and install

• dd
• make
• makedepend
• Python +3.5 (a Python 3 script is used to compress image data before being added to SeeGOL. It should work with any version of Python 3 but I did have the script crash once on a Debian machine running Python 3.4)
• ImageMagick (the convert program is used to down-sample image files) An emulator (such as QEMU) may also be helpful in running the OS.

Compiling from Source

To compile all of the x86 and C files and put them into a single flat binary:

make see_gol

To build the complete OS image, targetted to boot from a floppy:

make floppy.img

To remove all binaries and OS images:

make clean

Running with QEMU

There is a provided bash script so that the OS can be easily run with QEMU

./tools/i386_qemu.sh

Running with Physical Hardware

Assuming the floppy is mounted at /dev/fd0 (which is common on Linux systems) simply run:

make floppy

Otherwise dd the image file to the floppy:

dd if=/path/to/floppy.img of=/path/to/floppy

To just build the floppy image:

make floppy.img

Similarly, usb.img and usb make directives also exist. usb assumes the drive is at /dev/sdj (because that's the location my college desktop used).

After making any necessary changes to the boot order via a machine's BIOS settings, SeeGOL should be bootable from this floppy on almost any Intel x86 computer. Modern machines that do not have a floppy drive can usually boot See warning notes below for further information.

Warnings

Obviously this project comes without any warranties. The software provided has not been extensively tested across all possible hardware configurations. Although unlikely, since this OS does interact directly with the hardware, it could potentially damage it.

Run at your own risk.

Note on running this project on anything that isn't a PCjr

Although SeeGOL should run on any x86 system created after the PCjr (the x86 architecture is "backwards compatible to the dawn of time"), SeeGOL is not guaranteed to run properly on anything other than the PCjr. The main concern is that the PCjr uses the CGA graphics chip, as opposed to the much more common VGA chip. Most modern graphics cards, on-processor graphics chips, and emulators provide backwards-compatibility support for VGA. Although CGA and VGA both come from IBM and work in similar ways, VGA is not a direct descendant of CGA. Therefore the OS may not appear like it does on the PCjr across all systems or it may not work at all.

Directories

Here is a quick description of the top-level directories. Some of these directories have their own READMEs.

bin/

Stores binary objects (compiled code).

dgb_bin/

Stores compile-time debugging information. These are files that list the source code and compiled assembly instructions together with memory addresses.

docs/

Extensive documentation and presentations on the project.

img/

Stores the image files that can be dd'd to storage media and used to boot the OS.

media/

Contains media content not directly packaged into the OS.

old/

Deprecated code that is kept around as reference material.

src/

Source code directory for the project.

tools/

Includes scripts and other helpful tools used to run and debug the OS.

User Manual

SeeGOL comes with a built-in user manual. A help menu is accesible from both the main GUI menus and from SeeSH (the SeeGOL SHell).

Design

For a complete design overview, please see the docs/ directory.

Rolling Status Log

This a higher-level discussion of the project status while in development.

January 4th, 2017

• I decided to make the project public. I was going to wait until more had been completed but enough people have seemed interested in seeing what I had done that I figured I would publicly show my progress.
• I have booted SeeGOL on three platforms so far:
  • Modern x86 System (1st generation Core Series): Main function is reached but some functions defined outside of the main C file do not execute.
  • QEMU: Same behavior as with the physical x86 machine.
  • PCjr: Everything written in x86 assembly works. Nothing happens when the code written in C executes (more on that below).
• gcc does not compile to true 16-bit x86. Modern x86 systems will accept and run many 32-bit instructions in real mode despite not being in protected mode. To get around this, I think I may switch to using the "Amsterdam Compiler Kit" which claims it can compile to use only 16-bit instructions. If this does not work, I might be stuck writing some plugins for gcc.

January 17, 2017

• I am (for now) giving up on targeting the PCjr as the primary platform for this project. I can not find or implement a reliable tool set that will compile C to 8086 assembly. I will now be targeting i386 processors instead. SeeGOL should be able run on almost any modern x86 machine in Real Mode.

February 1, 2017

• I just solved a major bug with the kernel. It appears as though there is a difference between the .code16 and .code16gcc assembly directives. The former causes strange stack behavior and my best guess is that gcc is not properly handling segmentation addressing based on the initialization process in the bootloader. I can't find any documentation online that describes the directives but it makes a clear difference. I can now write messages to text video. I plan to improve the I/O tools shortly and then move to starting my work with the graphics modules.

February 13, 2017

• My graphics work from the Systems Programming class has been imported and significantly improved. Right now only Mode13h is supported but I have laid out the groundwork for other graphics modes. User programs (installed via the SeeSH shell interface) must access the graphics hardware through the Graphics Library (GL) package. This is actually enforced by design, unlike my work in Sys Prog.

February 27, 2017

A few major bugs/realizations came out of this past weekend, while laying the ground work for SeeGOL's image drawing capabilities.

• gcc's 16-bit compilation process does not use the segment registers. I previously thought that I was able to address 20 bits of memory (up to the A20 gate, 1mb barrier) but that is not the case. Since gcc does not manage the segment registers in Real Mode; I only have 64kb of memory to work with. I found this out the hard way while developing the image drawing code in the GL package, running out of memory while trying to import XPM files.
• XPM is a file format developed for X11 that stores image data as an indexed array of byte data. The file format is C code; you can simply include the file and to access an array of strings, each character byte representing a a value in a color look-up table. This would be an ideal image format for SeeGOL, as the image data can be baked into the OS, without the need of a reading image files...which would require a file system...that I don't have.
• However in my initial testing I found that the XPM file format is not space efficient...and ate up all my text memory. So I came up with a modified version of XPM: CXPM, Compressed X Pixel Map (the first proprietary element in SeeGOL that isn't prefixed with 'See'). To save space, the CXPM format is limited to less than 16 colors (each byte represents 2 pixels, identified by 4 bits) and then Huffman encoded in one dimension, per scanline. The run-length encoding greatly reduces the space used since so few colors are used. The full details of the spec are described currently in the cxpm.py file under src/res/. The spec may be further modified and I hope to write an OpenCV program that will quantize images and write them to the CXPM program, further automating the process. Currently I'm relying on Gimp and a Python script to convert color-quantized images to XPM and then to CXPM.
• One major (and painful to figure out) bug I ran into is that Python3 defaults to encoding strings in UTF-8. UTF-8 is a variable length encoding scheme and when I initially created CXPM files, it looked like text information was being saved correctly (confirmed through vim, less, cat, etc). However, when I compiled my code, I noticed that extra color information was being added to my test image, causing the images to "bleed out" of frame. Turns out that the UTF-8 encoding scheme was putting the extra bytes in my C strings and I was trying to interpret those bytes as color. To solve this issue, I switched the encoding scheme to 'latin-1', which is a fixed length 8-bit encoding scheme...so when I write 8 bits of bit-packed color information in Python, I get 8 bits and not occassionally 16 bits.

February 28, 2017

• After talking with a friend, I realized that there was no reason that CXPM needed to store pixel information in strings. I have now re-worked the format to store unsigned bytes instead of characters. This simplifies the encoding and decoding process and, in the process, solved multiple issues with encoding and decoding the image data as characters. I can now use 0 as a marker (i.e. the NULL byte) and I don't have to deal with issues in using newline characters (byte value of 10) in the string code. tl;dr I can store more color information while speeding up the decoding process.