415 lines
30 KiB
Plaintext
415 lines
30 KiB
Plaintext
HOW TO KICK OUT A MEMORY MANAGER
|
|
|
|
Warning : this article is meant for coders that are already experienced
|
|
enough with system programming. Sorry, but the concepts we'll have to use
|
|
are complicated enough to prevent me from explaining everything from the
|
|
very start. I think that this article will be long enough like that.
|
|
|
|
If you read this coding corner since its first issue (it was in Imphobia #7)
|
|
you should already know that one of my occupations was trying to find a way
|
|
to kick out the memory managers that likes to disable the use of the flat
|
|
real mode, or more generally to disable the acces to code privilege ring #0
|
|
(CPL0). Well, here is it.
|
|
|
|
The LOADALL option was not the good one. It is not supported on all types of
|
|
CPU. For example my intel 486DX2 does not support it. Moreover this instruct
|
|
seems to be a priviledged one, so it was impossible to use it to access to
|
|
CPL0.
|
|
|
|
However there must be a way to access to CPL0 with a memory manager loaded.
|
|
Do you think that Micro$oft Windows runs in enhanced 386 mode with a CPL3 ?
|
|
I don't. So there was only one possible solution : they used some kind of
|
|
undocumented interface to gain total control to the CPU.
|
|
|
|
This interface is known as the "Windows Global EMM Import Specification" and
|
|
we will just write GEMMIS in the rest of this text.
|
|
|
|
Here I need to thank my friend Cedric BERMOND aka LCA / Infiny for his great
|
|
help. He pointed out to me the Micro$oft Windows argument, and he told me
|
|
about the following references. He implemented his own system at the same
|
|
time as me, and we had a lot of chats about the problems we encountered and
|
|
the solutions we found. Without him the development of this system would
|
|
have been impossible. The code example included with this article is written
|
|
by me, but it's the result of a collaboration between us.
|
|
|
|
LCA will soon (before the Assembly'95) release his own protected mode memory
|
|
manager, featuring :
|
|
|
|
* user/supervisor protection system, with the user running at CPL0
|
|
|
|
* own linker, for a true flat segmentation model (for example the screen
|
|
really is at A0000 as opposed to pmode)
|
|
|
|
* paging disabled by default
|
|
|
|
* full debugger and disassembler system
|
|
|
|
Some references about the GEMMIS :
|
|
|
|
* Ralf Brown's interrupt list. Provides some quite detailed information
|
|
about the undocumented functions and structures that we will use, but with
|
|
no explanations at all. When I read this at the first time I believed that
|
|
this functions were a part of the Windows kernel and I skipped the rest. I
|
|
was wrong, this is some undocumented functions made FOR windows to allow
|
|
him to access to CPL0
|
|
|
|
* Dr Dobb's Journal, September 1994, Undocumented Corner,
|
|
"The Windows Global EMM Import Interface". This article provides some ex-
|
|
planations and gives the general layout we will have to use, but with a
|
|
lack of detail.
|
|
This problem becomes even more important today that the organizers of The
|
|
Assembly 95 said in a pre-invitation textfile that at this party the demos
|
|
will have to run with EMM386.EXE loaded. This sucks badly, however in this
|
|
article you have the solution. The only annoying thing is that I guess that
|
|
the organizer's reason is not a technical one, it is just because they want
|
|
to sell their damn CD's.... :(
|
|
|
|
Okay, now let's start with the serious things.
|
|
|
|
|
|
PART ONE - THE LAYOUT
|
|
|
|
We have two basic needs :
|
|
|
|
* switching to real, unprotected mode
|
|
|
|
* allocate some high memory
|
|
|
|
To satisfy this needs we will have to use the following layout :
|
|
|
|
* call int 2f undocumented function 1605h
|
|
"Micro$oft Windows - WINDOWS ENHANCED MODE & 286 DOSX INIT BROADCAST"
|
|
This call will fool the memory manager and he will believe that we are
|
|
Windows trying to initialize itself. The memory manager will suddenly become
|
|
very friendly with us. He will also provide us a pointer to a function that
|
|
we can call to switch to real, unprotected mode.
|
|
|
|
* before the switch we will allocate all available XMS and EMS memory
|
|
|
|
* In order to be able to use the allocated memory blocks we must know their
|
|
physical addresses. With the XMS it's easy, we just have to 'lock' them.
|
|
With the EMS it's harder because we have no documented way to know about
|
|
which physical pages are allocated to our EMS handle. That's no problem, the
|
|
switch to real mode function will return this information in a table. We
|
|
just need to know the physical address where this table will be, and we need
|
|
to use another undocumented function for that : we must do an IOCTL read on
|
|
the "EMMXXXX0" device, and the read will return the desired address along
|
|
with the version of the GEMMIS protocol. This function is referenced in RB's
|
|
interrupt list as "Memory Managers - GET EMM IMPORT STRUCTURE ADDRESS"
|
|
|
|
* then we use the switch to real mode function that was provided to us at
|
|
the first step. It fills us a table with all the information we could need
|
|
about the memory mapping and the pages allocated to our EMS handle, then it
|
|
gives us acces to the real mode.
|
|
|
|
* we need to analyse this table. It is located in high memory , so we will
|
|
have to switch to flat real mode (or in protected mode if you prefer it) in
|
|
order to access this table. It won't be a big problem : now we are in real
|
|
mode and the memory manager is not there to annoy us. But we must take care
|
|
of not enabling the interrupts : the UMB's and EMS page frames does not
|
|
exist anymore, and an interrupt would most likely hang the system.
|
|
|
|
* after that we have several ways to cope with the interrupts : the simplest
|
|
one is to be sure to NEVER call the any system interrupt during your demo,
|
|
the medium one is to switch back to V86 mode before any interrupt (it's easy
|
|
to do, we just have to use the same function that allowed us to acces to
|
|
real mode), and the hardest one (the one used by Micro$oft Windows) is to
|
|
code your own memory manager that will be able to run the system with the
|
|
same mapping that the DOS-memory manager used.
|
|
|
|
* When we have finished playing with the computer we must give it back to
|
|
the system. We just have to switch it back to V86 mode.
|
|
|
|
* Then we must call int 2f undocumented function 1606h
|
|
"Micro$oft Windows - WINDOWS ENHANCED MODE & 286 DOSX EXIT BROADCAST"
|
|
This will return the memory manager to his normal, unfriendly state.
|
|
|
|
PART TWO - THE DETAILS
|
|
|
|
* first step : call int 2f undocumented function 1605h "Micro$oft Windows -
|
|
WINDOWS ENHANCED MODE & 286 DOSX INIT BROADCAST" at the entry of this call
|
|
we must set ax=1605h (function number), and bx=cx=dx=si=ds=es=0. di will be
|
|
set of the Windows version number we want to simulate. For example we can
|
|
use 30ah for Windows version 3.10 then we call int 2fh. If the memory mana-
|
|
ger accepts to run windows he will return cx=0, in any other case we will
|
|
have to exit. The memory manager will return in ds:si a far pointer to a
|
|
modeswitch routine, used later to switch between V86 and real mode. It can
|
|
also return 0:0, and in this case we won't be able to enter real mode. The
|
|
memory manager will create a EMMXXXX0 device if there is not already one.
|
|
|
|
* then we have to allocate the memory. The simplest way is to allocate XMS
|
|
memory, because we can then use the documented XMS function 0ch to lock the
|
|
memory block and get his address. We want to allocate ALL the XMS memory and
|
|
perhaps it will already be fragmentated, so we should be able to handle
|
|
several XMS blocks. However some memory managers won't allow us to allocate
|
|
all the memory using XMS allocations, or sometimes they won't even support
|
|
the XMS standard. So we must also allocate some EMS memory. If we want to be
|
|
100% sure of our memory allocations we can use standard XMS & EMS functions
|
|
to copy a known string in all our allocated memory blocks. Later we will be
|
|
able to verify that this string is present at the expected physical addres-
|
|
ses. This provides us a way to verify the physical adresses we will use. In
|
|
order to later identify our EMS handle it is best to assign it a name.
|
|
|
|
* we must get the GEMMIS address with an IOCTL read on the "EMMXXXX0" device
|
|
"Memory Managers - GET EMM IMPORT STRUCTURE ADDRESS" we will first verify
|
|
that there is an EMM present, using the standard method defined in lotus/
|
|
intel/microsoft Expanded Memory Specification (EMS) version 4.0 : we must
|
|
open a handle for the EMMXXXX0 device, using the same function as for a file
|
|
open, i.e. a call to int 21h with ax=3d00h and ds:dx pointing to the device
|
|
name. If the handle exists (that should be the case) we issue a 'get device
|
|
information' IOCTL, i.e. a call to int 21h with ax=4400h and the handle to
|
|
the EMMXXXX0 device in bx. This call will return us some information about
|
|
our handle in dx, and we must verify that bits 14 and 7 are set, indicating
|
|
that this handle is associated to a device driver and not a file, and that
|
|
this device driver accepts IOCTL reads. We will then issue a 'get output
|
|
status' IOCTL on this device, i.e. a call to int 21h with ax=4407h and the
|
|
device handle in bx. we expect to have ax=255 upon exit, indicating that the
|
|
device is ready. At this step we know that there is an EMM in memory, but we
|
|
don't know which version of the EMM specification does it support. We close
|
|
our device handle (using standard file close function ah=3eh bx=device
|
|
handle) and we ask for EMM version, using int 67h function ah=46h. The
|
|
version returned in al should be at least 40h i.e. 4.0 now this memory mana-
|
|
ger should be recent enough to support the GEMMIS specification, so we will
|
|
attempt an GEMMIS IOCTL read on it. We must open again the EMMXXXX0 device
|
|
(same process as above) and issue an IOCTL read on it i.e. a call to int 21h
|
|
with ax=4402h bx=the device handle ds:dx pointing on a 6 bytes buffer and
|
|
cx=6=number of bytes to read. The first byte of the buffer must be pre-
|
|
filled with a 01h value. (it's used by the GEMMIS protocol as a sub-function
|
|
value) upon exit we should have AX=6=nb bytes read, and the buffer will be
|
|
filled with a dword value indicating the physical memory address of the
|
|
GEMMIS data table, and a word value indicating the version of the GEMMIS
|
|
specification. (format for the version word : one byte for the major number,
|
|
and one byte for the minor number). We expect the version to be at least 1.0
|
|
(the current version is 1.11 but is not supported by every manager. Version
|
|
1.0 will be enough to know the pages allocated to our EMS block) then we can
|
|
close again the device driver.
|
|
|
|
* we want to switch to real mode. We will use the modeswitch function
|
|
provided at the first step of the GEMMIS procedure. if we call it with ax=0
|
|
it will switch us to real mode. The carry flag should be cleared upon exit
|
|
to indicate a success. This function can (and will) destroy every register
|
|
except CS,IP,SS and SP. the DS,ES,FS,GS segments registers will also be des-
|
|
troyed. The switch to real mode will of course destroy the UMB's and EMS
|
|
frames, so we take care not to enable the interrupts after this step. We
|
|
must clear the interrupt flag before the call, and we can expect it to be
|
|
still cleared after the call. This function will not only switch us to real
|
|
mode, it will also fill up the GEMMIS data block with the V86 memory mapping
|
|
using a data format that I won't detail here, but you can look at the
|
|
GEMMIS.DOC file that I spread you with my code example, it's an extract from
|
|
RB's interrupt list and it's very detailed.
|
|
|
|
* Now we run in real mode (at CPL0) so we can do what we want with the CPU.
|
|
It's time to initialize flat real mode, or protected mode at CPL0 or whate-
|
|
ver else we wanted to have acces to. We must just have access to all the
|
|
physical memory, because the GEMMIS data table will be located in high memo-
|
|
ry and we won't be able to acces it from [non-flat] real mode. Nothing spe-
|
|
cial to explain here, I'll assume that you already know how to enter into
|
|
flat real mode.
|
|
|
|
* Now that we have access to all the physical memory it's time to analyse
|
|
the the GEMMIS data table. Just look at its format in the GEMMIS.DOC file,
|
|
and at my tips in the next section. This analysis will allow us to get the
|
|
physical addresses of the pages of our EMS handle. The interesting part of
|
|
the GEMMIS data block will be the EMS handle info records, however we will
|
|
need to use the rest of the GEMMIS data block to find them. We musn't rely
|
|
on the EMS handle number to identify our EMS handle, it is necessary to use
|
|
the EMS handle name we assigned. (because 386max won't give us the right
|
|
handle numbers).
|
|
|
|
* We have done with the GEMMIS protocol. The only problem now is that we
|
|
can't enable interrupts, because the interrupt handlers could try to access
|
|
to an UMB or to map some EMS pages and they don't exist anymore. So we can
|
|
either keep the interrupts disabled, or create some new interrupt handlers
|
|
that will switch back the CPU to V86 mode (using the modeswitch function),
|
|
call the old interrupt handler, and switch the CPU to flat real mode again.
|
|
(or to protected mode if you prefer, do what you like)
|
|
|
|
* At the end of our program (or when an interrupt occurs) we will want to
|
|
switch back the CPU to V86 mode, in order to restore the UMB and EMS native
|
|
structure of the DOS system. We just have to call the modeswitch function
|
|
with AX=1. Again, this call will destroy every register except CS,IP,SS,SP.
|
|
and we must call it with the interrupts disabled.
|
|
|
|
* At the end of our code we will have to fake a Windows exit. We just have
|
|
to call int 2fh with ax=1606h and dx=0
|
|
|
|
Phew, we are done with this shitty GEMMIS protocol ! Yes, but there is more
|
|
to come.....
|
|
|
|
PART THREE - THE PROBLEMS
|
|
|
|
Here I have to explain that this interface was keept very secret. There was
|
|
no official documentation about it and some of the memory manager makers did
|
|
not had the specification in the hands when they had to support it. For
|
|
example it is known that Novell had to reverse engineer some code to support
|
|
the last version of the specification. This explains why the GEMMIS protocol
|
|
is so badly implemented in several memory managers. Some data records are
|
|
sometimes not filled for example.... :(
|
|
|
|
By the way I think it shows us a very bad attitude from Micro$oft. They
|
|
sometimes are accusated to monopolyse the market but I just have to say that
|
|
this is true, because I cannot admit that someone keeps secret a specifica-
|
|
tion that is necessary for a memory manager to support Windows, and for a
|
|
Windows-like program to support the memory managers. It's a shame.
|
|
|
|
So here is a few tips that you must use if you want your GEMMIS implementa-
|
|
tion to support every memory manager around (you should read this part of
|
|
the article with a printed copy of GEMMIS.DOC near you. This is only imple-
|
|
mentation details, so you won't understand anything if you don't have the
|
|
format of the GEMMIS data table in the hands)
|
|
|
|
* at offset 4 in the GEMMIS data you should find a word containing the
|
|
GEMMIS version number. Don't use it. Several memory managers, for example
|
|
QEMM386 v6.0, won't fill it. Instead you should use the version number
|
|
provided by the GEMMIS IOCTL read, at offset 4 in the 6-bytes import buffer.
|
|
|
|
* don't rely on the informations provided by GEMMIS version 1.10 : several
|
|
memory managers still only implement version 1.0 (for example 386max)
|
|
* in the EMS frame status records, don't use the "flags for non-EMS frames"
|
|
byte (located at offset 5) : several memory managers, including 386max and
|
|
QEMM386, won't fill it. Use the "EMS frame type" at offset 0 instead.
|
|
|
|
* use the "number of UMB frame descriptors following" byte at offset 18Bh in
|
|
the GEMMIS data. Sometimes, for example if there is 386max loaded, the UMB
|
|
frame descriptor table will not be entirely filled but this byte will always
|
|
have the right value.
|
|
|
|
* the "number of EMS handle info records following" appears to be reliable.
|
|
|
|
* in each EMS handle info record, ignore the "handle number" byte. Better
|
|
use the "EMS handle's name" to identify your EMS handle. This is necessary
|
|
with 386max. The "physical address of page table entries forming page map"
|
|
entry points to a table giving us the physical memory adresses used for each
|
|
4K-page of the EMS handle. This table contains 4*"number of 16K pages for
|
|
handle" double words, and each or this double words gives us the physical
|
|
address of one 4K-page used for the EMS handle. The lower 12 bits of this
|
|
double words should be ignored because they will be filled with some
|
|
garbage.
|
|
|
|
* once again : don't use the information provided by GEMMIS version 1.10,
|
|
even Micro$oft's EMM386 v4.49 does not seems to fill it... You can just know
|
|
the structure in order to skip them, thus giving you access to the GEMMIS
|
|
version 1.11 records (the memory manager name)
|
|
|
|
* when you allocate your XMS memory, you will probably want to use the XMS
|
|
function 08h "Query Free Extended Memory". At this step don't rely on
|
|
getting an errorlevel in BL. Every XMS function returns some errorlevels in
|
|
BL, and zero means OK, but obviously Micro$oft just forgot the line saying
|
|
"BL = 00h if the function succeeds" in their original "eXtended Memory
|
|
Specification (XMS), ver 3.0" publication. The programmers at QuarterDeck
|
|
probably are not intelligent enough to understand this evidence, so when
|
|
everything is OK QEMM386 does not modify the BL register. So you should set
|
|
BL to zero before the call, and also test for AX=0 (no available XMS memory)
|
|
instead of just relying on the errorlevel in BL.
|
|
|
|
* QEMM is harder to support than the others. With QEMM you will have to
|
|
restart the whole protocol if you want to temporarily switch back to V86
|
|
mode. Here is the thing you will have to code if you want to switch back to
|
|
V86 mode for a little time (for launching a system interrupt for example) :
|
|
disable flat real mode, switch to native V86 mode using the modeswitch func-
|
|
tion, fake a windows exit, call your system interrupt, fake another windows
|
|
init, switch back to real mode with the modeswitch function, enable flat
|
|
real mode again. You may think it's stupid to fake a windows exit just for
|
|
entering windows again later, yes it's stupid but remember : we don't try to
|
|
be smart, we just try to emulate Windows code. ;-)
|
|
|
|
|
|
PART FOUR - THE IMPLEMENTATION
|
|
|
|
As an example I'll show you a simple flat real mode initializer. This imple-
|
|
mentation supports RAW, XMS and EMS memory allocations. It's a "do-nothing"
|
|
implementation, i.e. it just initializes flat real mode, then restores the
|
|
previous mode and exits to dos. Well this should be enough for demonstration
|
|
purposes.
|
|
|
|
It's written in turbo assembler 2.0, and I compile it with "tasm gemmis" and
|
|
"tlink /m/3 gemmis". I guess it could also be compiled with further versions
|
|
of turbo assembler, but I didn't tested this.
|
|
It should be very easy to use this code : just include what you want between
|
|
the init and end portions of the code, it could be a flat real mode program
|
|
or a protected mode initialization with CPL0, and it will work. You can use
|
|
the 'available_memory_table', it will give you the adresses of every alloca-
|
|
ted memory block. The number of such blocks will be stored in 'nb_available_
|
|
memory'
|
|
|
|
The init of this code may seem slow, the main reason for this is that I fil-
|
|
led every allocated memory block with a known string in order to verify that
|
|
the physical adresses are the right ones after the flat real mode switch.
|
|
|
|
There is two standards provided for interrupt support: The easyest one is to
|
|
just call the interrupt, there will be a handler installed which will trap
|
|
the interrupt and switch back to native V86 mode to execute it. Then the
|
|
system will return to flat real mode and you will not have to worry about
|
|
the modeswitch. However it will be a bit slow. (by the way you should trap
|
|
the timer interrupt yourself if you want to use this code in a demonstration
|
|
in order to avoid all this unnecessary mode switches)
|
|
The other method for the interrupt support is to ask the memory manager to
|
|
disable the flat real mode in a code section delimited by two function
|
|
calls. In this section you won't be in flat real mode, but you will be able
|
|
to call every interrupt without any unnecessary delay. I call that a native
|
|
mode code section.
|
|
|
|
Look at my code example : you just have to call mem32_init to setup every-
|
|
thing. At exit you will be in flat real mode and all the free high memory
|
|
will be allocated to you. You will be able to use the data in
|
|
available_memory_table to get some memory : it contains the starting and
|
|
ending physical adresses of every allocated memory blocks. The number of
|
|
such blocks will be stored in nb_available_memory.
|
|
|
|
When you want to define a native mode code section you'll just have to call
|
|
the mem32_native and mem32_flatreal functions.
|
|
When you want to exit your proggy just jump to the exit label, with DS:DX
|
|
pointing to a message you want to display.
|
|
|
|
As an example proggy I just included a proggy that dumps the GEMMIS v1.0
|
|
data table. It's damn slow because I use DOS interrupts to display the
|
|
information and this causes a lot of mode switches. In a real code you
|
|
probably would like to build your own string displayer and it would be
|
|
pretty much faster.
|
|
|
|
This implementation has been successfully tested with the following memory
|
|
managers : HIMEM.SYS alone, EMM386.EXE v4.49, QEMM386.SYS v6.00 beta,
|
|
386MAX.SYS v6.02
|
|
|
|
PART FIVE - GOING FURTHER
|
|
|
|
In my code example I used flat real mode, but it could be easy to change the
|
|
library to use protected mode if you prefer it. By the way a lot of protec-
|
|
ted mode coders insists on the fact that protected mode is faster because it
|
|
eliminates the need for prefixes before 32-bit instructs, but do they know
|
|
that it's also possible to use some 32-bit flat real mode without this ugly
|
|
prefixes ??? most of them does not seems to.
|
|
|
|
The obvious problem with this little example is that you have a delay each
|
|
time you want to call an interrupt. Because of that you will want to make
|
|
most of your init work in each part in a native mode code section, without
|
|
the benefits of privilege level 0, and then start your part without any call
|
|
to a V86 interrupt.
|
|
The alternate method, a much better one but far beyond the scope of this
|
|
introduction, would be to have your own V86 mode handler which would be able
|
|
to run the native system interrupts. This way you could bypass all the open
|
|
windows/close windows shit. If someone goes this way, I would be very happy
|
|
to get a little words from him. This is the way microsoft windows handles
|
|
the whole thing.
|
|
|
|
PART SIX - GOODBYE
|
|
|
|
Demomakers : the only thing you have to remember is that if this article and
|
|
the associated code example is useful for you I would be happy to get a lit-
|
|
tle greet. It would be nice also to say hello to my friend LCA/Infiny. If
|
|
you encounter some incompatibility problem with some hardware or with some
|
|
memory manager please let us know. Now I'm more or less obliged to include
|
|
the following disclaimer :
|
|
|
|
You can use this code example as you wish if it's for your private use. I
|
|
would prefer to receive a letter just to know that someone uses it, but it's
|
|
not necessary. Do what you want.
|
|
|
|
Here's an adress if you want to join me :
|
|
|
|
Walken / IMPACT Studios [coder]
|
|
Michel LESPINASSE
|
|
18 rue Jean Giono
|
|
80090 Amiens
|
|
France
|