Difference between revisions of "DOS memory management"

From Vogons Wiki
Jump to: navigation, search
m (→‎Upper memory)
(→‎Additional Remarks)
 
(21 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
With modern operating systems, all memory management you need to do is installing enough RAM. And that's it. The OS determines how to use it, and you usually don't need to worry about it at all.
 
With modern operating systems, all memory management you need to do is installing enough RAM. And that's it. The OS determines how to use it, and you usually don't need to worry about it at all.
−
In DOS, this is different. If you have ever tried DOS without paying attention to how your memory is organized, you may have noticed many programs fail to load with an 'Out of memory' error message. Even if you have four gigabytes of it! To use your RAM effectively in DOS you'll need to know how the x86 architecture splits up RAM in separate regions, what these different regions are for, and what you can do with them. Optimizing your system to reserve as much free space as possible for your programs is called '''memory management'''.
+
In DOS hardware and memory access is closely related to the standards defined by the x86 architecture. In DOS memory is split into different regions and there are several standards as EMS and XMS for accessing memory. To use the available RAM effectively in DOS you have to know how the x86 architecture splits up RAM, what these different regions are for. It is also helpful to know the most common memory management drivers. Optimizing your system to attain enough free memory for your programs is called '''memory management'''.
−
 
+
−
This page will try to explain, in simple language, all the technical terms you will encounter in DOS memory management, and provide you with practical information to get your favourite games and programs running. If you really want to understand how this all works, see the Wikipedia article on [http://en.wikipedia.org/wiki/DOS_memory_management DOS memory management], or use your web search engine of choice.
+
  
 +
This page will try to explain, in simple language, all the technical terms you will encounter in DOS memory management, and provide you with practical information to get your favorite games and programs running. If you really want to understand how this all works, see the Wikipedia article on [http://en.wikipedia.org/wiki/DOS_memory_management DOS memory management], or use your web search engine of choice.
  
 
== Types of memory ==
 
== Types of memory ==
 
=== Conventional memory ===
 
=== Conventional memory ===
−
Conventional memory, or base memory, is the most useful type of memory. All programs and TSRs load in conventional memory. Problem is, you only have 640KB of it.
+
Conventional memory or base memory is the memory range between 0 kb and 640 kb. Programs are loaded to this memory range. The available free memory can be lower as some drivers have to leave a part of their code in memory to handle e.g. hardware. This is called Terminate and Stay Resident (TSR). By default such drivers are placed in conventional memory.
−
(todo: expand)
+
The amount of free conventional memory is reported by CHKDSK or MEM.
  
 
=== Upper memory ===
 
=== Upper memory ===
−
The upper memory is all memory between the first 640KB and 1MB. It is normally marked as reserved memory for hardware, but memory management software can determine which parts are actually used and which are not, and enable you to access the unused areas as more-or-less 'normal' RAM. The unused parts of the upper memory area are referred to as Upper Memory Blocks.
+
The '''upper memory area''' ('''UMA''') is memory in the range between 640 kb and 1 Mb. By default there is no RAM in this range as it is reserved for use with hardware that is able to map own memory to this range. Usually present in this region is a part of the graphics cards RAM and the BIOS ROMs of the graphics card and mainboard. Additional hardware like mass storage controllers, network adapters... can use parts of the UMA for own BIOS ROMs or buffer RAM. The Extended Memory Specification (XMS) allows to map blocks of mainboard RAM in the remaining UMA memory range. These blocks are called '''Upper Memory Blocks''' (UMB) and are treated from DOS as High Memory. The default memory management driver that enables this mapping through XMS memory is EMM386.EXE. It has to be loaded in config.sys as second driver after HIMEM.SYS. The driver tries to detect common unused blocks and maps XMS memory there creating Upper Memory Blocks.
 +
By specifying DOS=UMB in config.sys MS-DOS allocates all UMBs through XMS and takes over memory management of them. this allows TSR-programs to be loaded to UMBs (in High Memory) in config.sys by using ''DEVICEHIGH=driver.sys'' or in autoexec.bat with the loadhigh statement, like ''LH driver.com''. If no contiguous free Upper Memory Block is available the driver will be loaded to Conventional Memory. Since UMA memory is managed in blocks the amount of free Upper Memory is usually larger than the largest contiguous free block. The external program MEM with parameter /C reports additionally the UMA RAM status.
  
 
=== High memory area ===
 
=== High memory area ===
−
HMA is the first 64KB above the 1MB limit. This can be a bit confusing if you're just getting started in memory management, since the name is similar to upper memory area and is usually referred to in the same context as upper memory. They are in fact separate regions, where the upper memory is '''below''' the high memory area. If you really want to know how this became a separate region, use your favourite search engine. All you really need to know is that you can load parts of DOS into it by specifying '''DOS=HIGH(,UMB)''' in your config.sys file.
+
The High Memory Area ('''HMA''') are the 64 kb directly above 1 Mb and are part of XMS. To use HMA it is required to load HIMEM.SYS in config.sys. By specifying the line '''DOS=HIGH''' in config.sys DOS is asked to load parts to HMA. Directly after an XMS driver is loaded in config.sys with DEVICE= and the HMA is still unoccupied DOS will move parts of code there. If DOS=HIGH was specified but loading to HMA fails ''HMA not available/loading DOS low'' is reported.
 +
The external program MEM reports if DOS is using the HMA.
  
 
=== Expanded memory ===
 
=== Expanded memory ===
−
(todo)
+
The '''Expanded Memory Specification''' ('''EMS''') is a standard developed by Lotus, Intel and Microsoft. Expanded Memory can be either memory on an memory expansion card or a part of the main memory. The specification describes that this memory can be used by mapping a 64 kB large part to the Upper Memory Area between 640 kB and 1 Mb. The 64 kB region within the Upper Memory Area is referred to as EMS Page Frame. The latest EMS 4.0 allowed to use up to 32 Mb as expanded memory. The most commonly used memory management driver implementing EMS is EMM386.EXE that has to be loaded in config.sys as second driver after HIMEM.SYS. EMM386 emulates expanded memory by using main memory beyond the High Memory Area. To achieve this it has to switch the CPU to Protected Mode. The amount of free main memory beyond 1 MB can be shared between EMS and XMS by adding the parameter AUTO to EMM386.EXE.
 +
In the case no EMS is required this feature can be deactivated by using the parameter NOEMS. This frees 64 kB in the Upper Memory Area for loading TSR-programs.
 +
 
 +
Versions:
 +
 
 +
EMS 3.0: max. 4 Mb
 +
 
 +
EMS 3.2: max. 8 Mb
 +
 
 +
EMS 4.0: max. 32 Mb
  
 
=== Extended memory ===
 
=== Extended memory ===
−
(todo)
+
The '''Extended Memory Specification''' ('''XMS''') describes an application interface that allows to copy memory between conventional and extended memory. Extended Memory is all main memory beyond 1 Mb. The most common memory management driver for this functionality is HIMEM.SYS that has to be loaded as first driver in config.sys. Accessing memory above the High Memory Area requires switching the CPU to Protected Mode, thus HIMEM.SYS requires at least a 286 CPU.
 +
 
 +
Versions:
 +
 
 +
XMS 2.0: max. 64 Mb
 +
 
 +
XMS 3.0: max. 4 Gb
 +
 
 +
 
 +
== Additional Remarks ==
 +
 
 +
While UMBs are a optional feature of XMS HIMEM.SYS does not support UMB. On bare MS-DOS this requires to load EMM386.EXE.
 +
 
 +
The DOS=HIGH and DOS=UMB statements in config.sys can be combined by writing DOS=HIGH,UMB. Specifying DOS=LOW can be used to prevent DOS from using the HMA.
 +
 
 +
Shadow-RAM is the RAM below the reserved UMA. Depending on chipset this RAM can be mapped in again in specific UMA memory ranges. A common use of Shadow-RAM is copying the BIOS ROMs to the Shadow-RAM below and mapping out the ROM afterwards. As the BIOS code is executed now from RAM instead of slow ROM the BIOS calls get accelerated. This functionality can usually enabled by BIOS (''Shadow System BIOS'' and ''Shadow Video BIOS'') or as parameter of a memory manager. Also the UMBPCI driver uses this feature to create UMBs through Shadow-RAM.
 +
 
 +
Architectural reasons force the memory for DMA transfers of ISA cards to be reserved below 1 Mb. This is usually of 16 kB size and reserved in UMA.
  
 
== Memory management ==
 
== Memory management ==
  
 
=== Getting more conventional memory ===
 
=== Getting more conventional memory ===
−
The most important and challenging aspect of memory management is to gain more free conventional memory. To do this, you will need to move as many drivers and TSRs as possible to UMBs. However, not all TSRs can load in upper memory, and even if they do they may not work correctly (usually in the form of completely crashing your system without giving any clues to a possible cause). If you notice any instability in your DOS system, try to move TSRs back into conventional memory to find the culprit. You may even need to use different hardware if the drivers for it won't load in upper memory.
+
A challenging aspect of DOS memory management can be to free enough conventional memory. Often games require a high amount of free conventional memory and sometimes also free EMS or XMS. To achieve this you have to use memory managers that allow e.g. enabling of UMBs and installation of APIs for XMS and EMS access.
 +
The availability of memory types depends on CPU and chipset.
  
−
(todo: expand)
+
On 286 CPUs special chipset support is required to enable UMBs. Rather well supported is the NEAT chipset. The HIMEM.SYS driver takes control of the A20-gate, makes HMA available and installs a XMS API.
 +
It is also possible to emulate a EMS page frame with the EMM286 driver. Some chipsets have jumpers to set XMS and EMS assignment of the main memory.
  
 +
386 and later CPUs feature an protected mode extension called Virtual 8086 mode that allows mimicking a 8086 memory layout for applications. With MS-DOS 5 a second memory manager was included called EMM386.EXE that uses the advanced features of 386 CPUs. It is loaded after HIMEM.SYS and enables UMBs and EMS emulation. This allows to load TSRs to UMBs freeing conventional RAM. The corresponding load statement in config.sys is DEVICEHIGH= and in autoexec.bat LOADHIGH which can be shortened to LH. EMM386 switches the CPU to Virtual 8086 mode.
 +
 +
(todo UMBPCI)
 +
 +
Generally memory managers must be loaded first. As rule of thumb TSRs taking more memory should be loaded before small TSRs to prevent memory fragmentation. Some drivers require a specific load order e.g. SMARTDRV.EXE after MSCDEX.EXE if the CD-ROM should be read cached. Some TSRs do not work correctly when loaded to upper memory, resulting in crashs or erratic system behavior. If you notice any instability in your DOS system, try to move TSRs back into conventional memory to find the culprit. You may look for driver alternatives or in worst case need to change hardware.
 +
Mainboard BIOS, graphic card BIOS and also additional cards as mass storage controllers or network cards use the memory area between 640K and 1 MB sharing space with UMBs. Especially newer graphic cards and mainboards are not designed with DOS in mind and are delivered with large BIOSes (> 32 kB). This can reduce available UMBs considerably. A classic DOS system has about 128 kB to 192 kB free UMBs for TSRs. Using EMS emulation requires a 64 kB area between 640K and 1 MB where the page frame of a virtual EMS card is mapped. If not required EMS emulation can be disabled freeing 64 kB of UMBs.
 +
 +
Another way is to use a utility that changes the start up files such that the amount of free conventional memory is maximal (e.g. MS-DOS Memmaker, QEMM Optimize). Typically these programs run several analysis reboots to gather information about the drivers and memory layout. In the end a optimized configuration is presented. The analysis depth and the final quality of the presented new boot configuration of the programs varies.
 +
(todo: expand)
  
 
== Software ==
 
== Software ==
−
(todo: list of memory managing software?)
+
'''HIMEM.SYS''': enables XMS (except UMBs) with 286 up to 15 Mb, with 386/486 up to 1023 Mb
 +
 
 +
Versions:
 +
 
 +
MS-DOS 5.0 : 2.77, max. 15 Mb XMS
 +
 
 +
MS-DOS 6.0 : 3.07, recognizes max. 4 Gb, 1023 MB XMS useable.
 +
 
 +
MS-DOS 6.2 6.21 6.22 : 3.10, recognizes max. 4 Gb, 1023 MB XMS useable.
 +
 
 +
'''EMM386.EXE''': introduced with MS-DOS 5.0, uses XMS to create UMBs in UMA, uses XMS to create EMS
 +
 
 +
'''MEMMAKER.EXE''': introduced with MS-DOS 6.0, utility to optimize free conventional memory
 +
 
 +
 
 +
'''QEMM386''': a complete package of memory management driver and utilities, last version 9.0 / QEMM97
 +
 
 +
'''QEMM386.SYS''' memory management driver implementing XMS, EMS, UMA, VCPI and DPMI, enhanced features include support for up to 256 MB RAM, QuickBoot, Stealth, dynamically load drivers from command line and specific support for popular drivers as DoubleSpace, Stacker. Additional utilities include
 +
 
 +
'''Optimize''' a powerful tool to optimize free conventional memory
 +
 
 +
'''DOS-UP''' for relocating DOS to high memory
 +
 
 +
'''Manifest''' a system information tool with focus on memory layout
 +
 
 +
'''QDPMI''' a DPMI 0.9 server
 +
 
 +
'''MagnaRAM''' a memory compression utility replacing a portion of Windows 3.1 virtual memory subsystem.
 +
 
 +
 
 +
'''UMBPCI''' Upper memory driver, partial replacement for EMM386.EXE. Uses Shadow-RAM instead of extended memory to create UMBs. Functionality is chipset dependent, see [http://www.uwe-sieber.de/umbpci_e.html UMBPCI homepage] for a list of supported chipsets. Also be aware of issues with ISA-DMA using the Shadow-RAM method that could lead to data corruption if not setup properly. This driver leaves the CPU in Real mode.
 +
 
 +
[http://support.microsoft.com/kb/95555 Overview of Memory-Management Functionality in MS-DOS]
 +
 
 +
[http://en.wikipedia.org/wiki/QEMM#DOS_equivalents Overview about features of different memory managers]

Latest revision as of 01:17, 12 April 2013

With modern operating systems, all memory management you need to do is installing enough RAM. And that's it. The OS determines how to use it, and you usually don't need to worry about it at all. In DOS hardware and memory access is closely related to the standards defined by the x86 architecture. In DOS memory is split into different regions and there are several standards as EMS and XMS for accessing memory. To use the available RAM effectively in DOS you have to know how the x86 architecture splits up RAM, what these different regions are for. It is also helpful to know the most common memory management drivers. Optimizing your system to attain enough free memory for your programs is called memory management.

This page will try to explain, in simple language, all the technical terms you will encounter in DOS memory management, and provide you with practical information to get your favorite games and programs running. If you really want to understand how this all works, see the Wikipedia article on DOS memory management, or use your web search engine of choice.

Types of memory

Conventional memory

Conventional memory or base memory is the memory range between 0 kb and 640 kb. Programs are loaded to this memory range. The available free memory can be lower as some drivers have to leave a part of their code in memory to handle e.g. hardware. This is called Terminate and Stay Resident (TSR). By default such drivers are placed in conventional memory. The amount of free conventional memory is reported by CHKDSK or MEM.

Upper memory

The upper memory area (UMA) is memory in the range between 640 kb and 1 Mb. By default there is no RAM in this range as it is reserved for use with hardware that is able to map own memory to this range. Usually present in this region is a part of the graphics cards RAM and the BIOS ROMs of the graphics card and mainboard. Additional hardware like mass storage controllers, network adapters... can use parts of the UMA for own BIOS ROMs or buffer RAM. The Extended Memory Specification (XMS) allows to map blocks of mainboard RAM in the remaining UMA memory range. These blocks are called Upper Memory Blocks (UMB) and are treated from DOS as High Memory. The default memory management driver that enables this mapping through XMS memory is EMM386.EXE. It has to be loaded in config.sys as second driver after HIMEM.SYS. The driver tries to detect common unused blocks and maps XMS memory there creating Upper Memory Blocks. By specifying DOS=UMB in config.sys MS-DOS allocates all UMBs through XMS and takes over memory management of them. this allows TSR-programs to be loaded to UMBs (in High Memory) in config.sys by using DEVICEHIGH=driver.sys or in autoexec.bat with the loadhigh statement, like LH driver.com. If no contiguous free Upper Memory Block is available the driver will be loaded to Conventional Memory. Since UMA memory is managed in blocks the amount of free Upper Memory is usually larger than the largest contiguous free block. The external program MEM with parameter /C reports additionally the UMA RAM status.

High memory area

The High Memory Area (HMA) are the 64 kb directly above 1 Mb and are part of XMS. To use HMA it is required to load HIMEM.SYS in config.sys. By specifying the line DOS=HIGH in config.sys DOS is asked to load parts to HMA. Directly after an XMS driver is loaded in config.sys with DEVICE= and the HMA is still unoccupied DOS will move parts of code there. If DOS=HIGH was specified but loading to HMA fails HMA not available/loading DOS low is reported. The external program MEM reports if DOS is using the HMA.

Expanded memory

The Expanded Memory Specification (EMS) is a standard developed by Lotus, Intel and Microsoft. Expanded Memory can be either memory on an memory expansion card or a part of the main memory. The specification describes that this memory can be used by mapping a 64 kB large part to the Upper Memory Area between 640 kB and 1 Mb. The 64 kB region within the Upper Memory Area is referred to as EMS Page Frame. The latest EMS 4.0 allowed to use up to 32 Mb as expanded memory. The most commonly used memory management driver implementing EMS is EMM386.EXE that has to be loaded in config.sys as second driver after HIMEM.SYS. EMM386 emulates expanded memory by using main memory beyond the High Memory Area. To achieve this it has to switch the CPU to Protected Mode. The amount of free main memory beyond 1 MB can be shared between EMS and XMS by adding the parameter AUTO to EMM386.EXE. In the case no EMS is required this feature can be deactivated by using the parameter NOEMS. This frees 64 kB in the Upper Memory Area for loading TSR-programs.

Versions:

EMS 3.0: max. 4 Mb

EMS 3.2: max. 8 Mb

EMS 4.0: max. 32 Mb

Extended memory

The Extended Memory Specification (XMS) describes an application interface that allows to copy memory between conventional and extended memory. Extended Memory is all main memory beyond 1 Mb. The most common memory management driver for this functionality is HIMEM.SYS that has to be loaded as first driver in config.sys. Accessing memory above the High Memory Area requires switching the CPU to Protected Mode, thus HIMEM.SYS requires at least a 286 CPU.

Versions:

XMS 2.0: max. 64 Mb

XMS 3.0: max. 4 Gb


Additional Remarks

While UMBs are a optional feature of XMS HIMEM.SYS does not support UMB. On bare MS-DOS this requires to load EMM386.EXE.

The DOS=HIGH and DOS=UMB statements in config.sys can be combined by writing DOS=HIGH,UMB. Specifying DOS=LOW can be used to prevent DOS from using the HMA.

Shadow-RAM is the RAM below the reserved UMA. Depending on chipset this RAM can be mapped in again in specific UMA memory ranges. A common use of Shadow-RAM is copying the BIOS ROMs to the Shadow-RAM below and mapping out the ROM afterwards. As the BIOS code is executed now from RAM instead of slow ROM the BIOS calls get accelerated. This functionality can usually enabled by BIOS (Shadow System BIOS and Shadow Video BIOS) or as parameter of a memory manager. Also the UMBPCI driver uses this feature to create UMBs through Shadow-RAM.

Architectural reasons force the memory for DMA transfers of ISA cards to be reserved below 1 Mb. This is usually of 16 kB size and reserved in UMA.

Memory management

Getting more conventional memory

A challenging aspect of DOS memory management can be to free enough conventional memory. Often games require a high amount of free conventional memory and sometimes also free EMS or XMS. To achieve this you have to use memory managers that allow e.g. enabling of UMBs and installation of APIs for XMS and EMS access. The availability of memory types depends on CPU and chipset.

On 286 CPUs special chipset support is required to enable UMBs. Rather well supported is the NEAT chipset. The HIMEM.SYS driver takes control of the A20-gate, makes HMA available and installs a XMS API. It is also possible to emulate a EMS page frame with the EMM286 driver. Some chipsets have jumpers to set XMS and EMS assignment of the main memory.

386 and later CPUs feature an protected mode extension called Virtual 8086 mode that allows mimicking a 8086 memory layout for applications. With MS-DOS 5 a second memory manager was included called EMM386.EXE that uses the advanced features of 386 CPUs. It is loaded after HIMEM.SYS and enables UMBs and EMS emulation. This allows to load TSRs to UMBs freeing conventional RAM. The corresponding load statement in config.sys is DEVICEHIGH= and in autoexec.bat LOADHIGH which can be shortened to LH. EMM386 switches the CPU to Virtual 8086 mode.

(todo UMBPCI)

Generally memory managers must be loaded first. As rule of thumb TSRs taking more memory should be loaded before small TSRs to prevent memory fragmentation. Some drivers require a specific load order e.g. SMARTDRV.EXE after MSCDEX.EXE if the CD-ROM should be read cached. Some TSRs do not work correctly when loaded to upper memory, resulting in crashs or erratic system behavior. If you notice any instability in your DOS system, try to move TSRs back into conventional memory to find the culprit. You may look for driver alternatives or in worst case need to change hardware. Mainboard BIOS, graphic card BIOS and also additional cards as mass storage controllers or network cards use the memory area between 640K and 1 MB sharing space with UMBs. Especially newer graphic cards and mainboards are not designed with DOS in mind and are delivered with large BIOSes (> 32 kB). This can reduce available UMBs considerably. A classic DOS system has about 128 kB to 192 kB free UMBs for TSRs. Using EMS emulation requires a 64 kB area between 640K and 1 MB where the page frame of a virtual EMS card is mapped. If not required EMS emulation can be disabled freeing 64 kB of UMBs.

Another way is to use a utility that changes the start up files such that the amount of free conventional memory is maximal (e.g. MS-DOS Memmaker, QEMM Optimize). Typically these programs run several analysis reboots to gather information about the drivers and memory layout. In the end a optimized configuration is presented. The analysis depth and the final quality of the presented new boot configuration of the programs varies. (todo: expand)

Software

HIMEM.SYS: enables XMS (except UMBs) with 286 up to 15 Mb, with 386/486 up to 1023 Mb

Versions:

MS-DOS 5.0 : 2.77, max. 15 Mb XMS

MS-DOS 6.0 : 3.07, recognizes max. 4 Gb, 1023 MB XMS useable.

MS-DOS 6.2 6.21 6.22 : 3.10, recognizes max. 4 Gb, 1023 MB XMS useable.

EMM386.EXE: introduced with MS-DOS 5.0, uses XMS to create UMBs in UMA, uses XMS to create EMS

MEMMAKER.EXE: introduced with MS-DOS 6.0, utility to optimize free conventional memory


QEMM386: a complete package of memory management driver and utilities, last version 9.0 / QEMM97

QEMM386.SYS memory management driver implementing XMS, EMS, UMA, VCPI and DPMI, enhanced features include support for up to 256 MB RAM, QuickBoot, Stealth, dynamically load drivers from command line and specific support for popular drivers as DoubleSpace, Stacker. Additional utilities include

Optimize a powerful tool to optimize free conventional memory

DOS-UP for relocating DOS to high memory

Manifest a system information tool with focus on memory layout

QDPMI a DPMI 0.9 server

MagnaRAM a memory compression utility replacing a portion of Windows 3.1 virtual memory subsystem.


UMBPCI Upper memory driver, partial replacement for EMM386.EXE. Uses Shadow-RAM instead of extended memory to create UMBs. Functionality is chipset dependent, see UMBPCI homepage for a list of supported chipsets. Also be aware of issues with ISA-DMA using the Shadow-RAM method that could lead to data corruption if not setup properly. This driver leaves the CPU in Real mode.

Overview of Memory-Management Functionality in MS-DOS

Overview about features of different memory managers

Retrieved from "https://www.vogonswiki.com/index.php?title=DOS_memory_management&oldid=1204"