Chapter 3 Memory Management

This chapter covers memory management issues and the LynxOS system calls that support device driver memory management. It describes the LynxOS memory model, supported address types, memory allocation, memory locking, address translation, accessing user space from interrupt handlers and kernel threads, and hardware access.

LynxOS Virtual Memory Model

LynxOS uses a virtual addressing architecture. All memory addresses generated by the CPU are translated by the hardware MMU (Memory Management Unit). Each user task has its own protected virtual address space that prevents tasks from interfering with each other.

The kernel, which includes device drivers, and the currently executing user task exist within the same virtual address space. The user task is mapped into the lower part, the kernel into the upper part. Only the part of the virtual map occupied by the user task is remapped during a context switch. Applications cannot access the part of the address space occupied by the kernel and its data structures. The constant OSBASE defines the upper limit of the user accessible space. Addresses above this limit are accessible only by the kernel.

Kernel code, on the other hand, has access to the entire virtual address space. This facilitates the passing of data between drivers and user tasks. A device driver, as part of the kernel, can read or write a user address as a direct memory reference, without the need to use special functions. Because of this, precautions should be taken to restrict the access of the device driver only to necessary structures.

Please see "Accessing Hardware" for detailed LynxOS virtual memory maps of the currently supported platforms.

DMA Transfers

All addresses generated by the CPU are treated as virtual and are converted to physical addresses by the MMU. This makes the programming of DMA transfers slightly more complicated because memory that is contiguous in virtual space can be mapped to non-contiguous physical pages. DMA devices, however, typically work with physical addresses. Therefore, a driver must convert virtual addresses to their corresponding physical addresses before passing them to a DMA controller.

In addition, user memory may be paged, which can lead to changes in the virtual to physical address mapping and to a physical page being reallocated to another user task. Paging must therefore be suppressed on user memory that is involved in a DMA transfer by locking the memory region.

The sections "Memory Locking" and "Address Translation" covers memory locking and address translation issues and functions relating to DMA transfers.

LynxOS Address Types

Address Types
Type	Description
User Virtual	These addresses are passed to the driver from a user application. Typically they are addresses of buffers or data structures used to transfer data between the application and a device. They are valid only when the user task that passed the address is the current task.
Kernel Virtual	These are addresses of kernel functions and variables that can be used by a driver. The mapping of kernel virtual addresses is permanent so they are valid from anywhere within a device driver. They are not accessible from an application.
Kernel Direct Mapped	A region of the kernel virtual space that is directly mapped to physical memory (that is, contiguous virtual addresses map to contiguous physical addresses) - The base of this region is defined by the constant PHYSBASE, which maps to the start of RAM. The size of the region is platform-dependent. Note: Memory that exists on devices or non-system buses (PCI or VME for example) is not accessible by way of PHYSBASE.
Physical	Physical memory is the non-translated address for memory. Physical addresses are used when a driver needs to set up pointers to physical memory for controllers that bypass the MMU (DMA controllers, for example).
Device	Device addresses include I/O port addresses, PCI addresses, VME addresses, and so on.

Allocating Memory

The the following table summarizes the LynxOS memory allocation and deallocation functions that support device drivers. Note that these system calls cannot be called from within an ISR.

Allocated memory comes from the kernel address space and is not accessible to applications. Refer to section "Address Translation" for address translation considerations. A complete description of these functions is available in their respective man pages.

Memory Allocation Functions
System Call	Summary
sysbrk()	sysbrk() is useful for allocating shared static data structures and queues used by the device driver. It is recommended that sysbrk() be used primarily in the install() and open() entry point routines though it can be used in any of the others. The prototype for sysbrk() is: char * sysbrk(long size) size is the number of bytes required. sysbrk() returns NULL if no memory is available. Note: malloc() cannot be used within device drivers.
sysfree()	sysfree() deallocates memory allocated by sysbrk() and returns it to the free list. The prototype for sysfree() is: void sysfree(char *p, long size) p is the pointer to the memory to free and size is its size.
get1page()	get1page() allocates one page of physical memory and returns a virtual address that can be used to access it. get1page() is useful for obtaining buffers for DMA. The virtual address is above OSBASE, and is thus valid in the context of any task including interrupt service routines. The page of memory is PAGESIZE in length (4096 bytes for example) and is defined in <kernel.h>. The prototype for get1page() is: char * get1page() get1page() returns NULL if no memory is available.
free1page()	free1page() releases a page of memory allocated by get1page(). The prototype for free1page() is: void free1page(char * addr) addr is pointer to the page of memory to free.
alloc_cmem()	alloc_cmem() returns a block of contiguous physical memory. It is useful for obtaining DMA buffers for devices incapable of scatter/gather transfers. It is recommended that alloc_cmem() be used primarily in the install() entry point routine, or if necessary, in the open() entry point, though it can be used any of the others. The prototype for alloc_cmem() is: char * alloc_cmem(int size) size is the number of bytes to allocate. alloc_cmem() returns a pointer to a block of pages contiguous in memory. The pointer it returns is a virtual address. If insufficient memory is available, alloc_cmem() returns NULL.
free_cmem()	free_cmem() releases the memory allocated by alloc_cmem(). The prototype for free_cmem() is: void free_cmem(char * p, int size) p is the pointer to the memory to free and size is its size.

Memory Locking

LynxOS provides mem_lock() and mem_unlock() system calls for locking and unlocking user memory. These functions are provided to support DMA transfers.

User memory may be paged, which can lead to changes in the virtual to physical address mapping and to a physical page being reallocated to another user task. For DMA transfers, paging must be suppressed on user memory.

If paging has not been activated, either by the vmstart command or a system call, these routines have no effect. Because a device driver cannot determine if paging is active, it should always lock memory used with DMA transfers.

mem_lock()

The mem_lock() system call is used to prevent a region of memory from being paged. mem_lock() cannot be used from within an ISR.

The arguments specify the start address (uaddr) and size (size) of a memory region in the process specified by pid. These typically correspond to the arguments passed to the read() or write() entry point functions of the device driver.

The getpid() system call can be used to get the process ID of the current task. For example:

if (mem_lock (pid = getpid(), uaddr, size) == SYSERR)
{
pseterr (ENOMEM);
return (SYSERR);
}

If paging is activated and any virtual addresses in the specified range are not currently mapped to physical memory, mem_lock() attempts to allocate and map the address into physical memory. If there is not enough physical memory to do this, SYSERR is returned. Otherwise, OK is returned. A successful return means that the specified virtual address range is mapped to physical memory and is locked. If paging is not activated, mem_lock() returns OK.

It is permissible to lock the same address multiple times. This may occur, for example, when locking overlapping regions. However, the memory must be unlocked the same number of times.


Note: Kernel memory is not paged. Therefore, memory allocated with sysbrk(), get1page() or alloc_cmem() does not require locking.

mem_unlock()

The mem_unlock() function is used to unlock memory locked by mem_lock(). mem_unlock() cannot be used from within an ISR.

if (mem_unlock (pid, uaddr, size, dirty) == SYSERR)
{
return (SYSERR); /* error set to EINVAL by
mem_unlock */
}

Once memory is unlocked it becomes eligible for paging again. Attempting to unlock memory that was not previously locked causes the routine to return SYSERR. A successful unlock returns OK. If paging is not activated, mem_unlock() returns OK.

If the memory contents are modified by another processor, such as a DMA controller, the dirty flag should be set to 1. This ensures that the contents are written out to the swap file if the memory region is subsequently paged out.

A driver should always ensure that any locked memory is released when a task closes the device or when the device is uninstalled. Pages that remain locked after a task has exited, though usable by other tasks, will not be paged.

Address Translation

Device drivers must convert virtual addresses to their corresponding physical addresses before passing them to a DMA controller. However, the physical address range of virtual memory segments are not guaranteed to be contiguous as depicted in the figure below.

Virtual to Physical Memory Mapping

The mmchain() system call is used to obtain the physical addresses and sizes of all memory pages that make up a contiguous virtual memory segment. mmchain() cannot be used from within an ISR.

mmchain() writes into array the physical addresses and sizes of all memory pages that make up the contiguous virtual memory segment described by vaddr and size, and returns the number of elements written into array. If no physical memory is mapped to a virtual address, mmchain() sets the converted address to 0. To ensure valid mappings, mem_lock() should be used prior to mmchain().

array is an array of dmachain structures (defined in <mem.h>). The size of array must be one more than the number of memory pages contained in size. The size of array can be computed as follows:

The physical addresses returned are offset by PHYSBASE. For maximum portability, the physical address should be offset by the system constant, drambase. The start of RAM is contained in drambase.

mmchain (array, virtaddr, nbytes);
for (i = 0; i < nsegments; i++) {
  physaddr = array[i].address PHYSBASE + drambase;
  length = array[i].count;
  ...
}

The virtual memory segment used with mmchain() refers to the memory space owned by the current process. The system call mmchainjob() can be used to obtain a dmachain array for a specific process. Refer to the man page for mmchainjob() for more information.

Virtual Address Conversion

A single virtual address can be converted to its physical addresses using get_phys(). get_phys() cannot be used from within an ISR.

The address returned is a kernel direct mapped address that is offset by PHYSBASE. To convert this address to its physical address, PHYSBASE must be subtracted and drambase added. For example:

The start of RAM is contained in drambase. On most platforms, this is 0, but for maximum portability, it should be used in the calculation.

Validating Addresses

The addresses passed into the ioctl() entry point function by application code must be validated before they can be used. The functions rbounds() and wbounds() are used for this purpose. These calls may be used in an interrupt routine.

The return value from rbounds() should be compared to the size of the object the device driver expects to be referenced. The error code EFAULT should be returned if addr is found to be erroneous.

rbounds() returns the number of bytes to the next boundary of non-readable memory in the virtual address space of the calling process. In other words, it returns the number of bytes readable starting at addr.

wbounds() returns the number of bytes to the next boundary of non-writable memory in the virtual address space. This works similarly to rbounds(), except that checking for addr in the text segment is not done.

Accessing User Space from Interrupt Handlers and Kernel Threads

Interrupt handlers and kernel threads execute asynchronously with respect to the user task making requests to the driver. An interrupt handler executes in the context of the task that was current when the interrupt occurred. Kernel threads execute in the context of the null process (process 0). Because the null process has no user context associated with it, the switch to a kernel thread is much quicker than to a user thread. It is sometimes necessary for an interrupt handler or a kernel thread to access a location in a user task, even though the target task may not be the currently mapped task. This requires some special considerations. The key to understanding how this can be done is the fact that there is a second virtual address that can be used to access an address in user space.

As the figure above shows, for any address in user space there are in fact two virtual addresses mapped to the physical address. One is the user virtual address (a). However, this mapping is valid only when the user task is the current task. So, it cannot be used from an interrupt handler or kernel thread. This mapping may also be changed if paging is activated. The second mapping is the direct mapped kernel virtual address (b). This mapping is permanent so it can safely be used anywhere in a driver.

Therefore, care must be taken when accessing user space from an interrupt handler or kernel thread. The driver must first convert the user virtual address to its corresponding kernel direct mapped address in the top-half routine (usually the ioctl() entry point) and then pass this address to the interrupt handler or kernel thread by way of the statics structure (see "Statics Data Structure"). The user address must also be locked to prevent the address mapping from being changed by the paging system.

In the following example, u_addr and size specify the virtual address and size of the user memory to be accessed; this is passed to the driver from the application.

dev_ioctl (s, f, cmd, arg)
struct statics *s;
struct file *f;
char *arg;
{
  char *kvaddr;
  ...
  if (mem_lock (getpid (), u_addr, size) == SYSERR)
  {
    pseterr (ENOMEM);
    return (SYSERR);
  }
kvaddr = get_phys (u_addr); /* convert user addr */
s->u_status = kvaddr; /* save for later use */
s->u_size = size;
s->u_addr = u_addr; /* used for unlocking memory */
s->pid = getpid ();
...
}

The pointer can now be used from an interrupt handler or kernel thread to access the user memory.

kernel_thread (s)
struct statics *s;
{
  ...
  if (s->u_status)
  {
    *(s->u_status) = status; /* pass info to user task */
    /* unlock user memory */
    mem_unlock (s->pid, s->u_addr, s->size, 0);
    s->u_status = NULL);
  }
...
}

Accessing Hardware

This section describes how device drivers access the hardware layer and illustrates the virtual address mappings used by LynxOS on different hardware platforms.

The following sections contain platform-specific information about hardware device access from LynxOS. Each section contains memory map figures to illustrate the mapping of LynxOS virtual addresses to the hardware device.

In general, the kernel has permissions to access the full virtual address space while the user processes have restricted access. The table below shows a generalization of this concept.

Virtual Memory Access to User Processes
LynxOS Virtual Memory Area	Permissions
OSBASE and above	Kernel only; no user access
SPECPAGE	Read-only to user
Kernel Stack	Read-only to user
Shared Memory	Depends on mapping
User Stack	Read-write to user
User Data	Read-write to user
User Text	Read-only to user

Using permap()

The function permap() allows a driver to map a memory-mapped device into the kernel virtual address map so that the device can be accessed from the driver. If the memory region in which the device resides is already mapped, then it is not necessary to use permap().

The virtual address PHYSBASE is always mapped to the physical address corresponding to the start of RAM. The size of the mapped region is equal to the system RAM size (up to 512 MB). The permap() function must be used to access devices that are outside of the pre-mapped region (640 KB to 1 MB). For example, in a VMEbus-based PC, the VMEbus is often mapped in the high end of the physical address space, above 512 MB. The code to map this using permap() would be similar to:

/* VMEADDR: physical address where VMEbus appears */
#define VMEADDR 0x40000000

/* vme_addr: virtual address 32 bit VMEbus accesses */
unsigned long *vme_vaddr;

The physical address passed to permap() must be aligned on a page boundary. The size, in bytes, must be a multiple of PAGESIZE.

Device Access on x86 Systems

Reading and Writing Device Registers

The majority of devices for x86 systems exist in the CPU's I/O space, which is accessed with the in and out instructions. The file port_ops_x86.h (located in $ENV_PREFIX/sys/include/family/x86/) contains macros that can be called to read and write device registers.

The memory on I/O devices in the 640 KB to 1 MB range can be directly accessed using the PHYSBASE offset. The constant PHYSBASE is defined in kernel.h (in $ENV_PREFIX/sys/include/kernel/). For example, to access I/O devices using the PHYSBASE offset:

/* RAMBASE: RAMbase address of Ethernet controller */
#define RAMBASE 0xCC000
unsigned long * vaddr;
vaddr = (unsigned long *) (PHYSBASE + RAMBASE);

The following two figures illustrate the LynxOS virtual memory model on the x86 platform.

LynxOS dynamic user applications (applications that use shared libraries) are linked at a default starting address of 0x00400000 (4 MB).

Linux dynamic user applications are linked at a default starting address of 0x080000000 (128 MB).

ld.so is loaded into location 0x0 and the virtual address space between the end of ld.so and the start of the application is used for shared libraries.

Device Access on PowerPC Systems

ISA Bus Access

The PowerPC reference platform contains a primary PCI bus and a secondary ISA bus for system I/O. All access to the ISA bus goes through the PCI bus and the PCI-to-ISA bridge hardware. On Motorola VME systems, the VME bus interfaces to the PCI bus through the VME-to-PCI bridge hardware. The PReP reference memory map defines all physical addresses above 2 GB to be directed to the PCI bus and all physical addresses less than 2 GB as memory access (see following figure). The PCI devices on the PCI bus are configured to claim address ranges. The ISA Bridge hardware claims all unclaimed PCI addresses. This is referred to as subtractive decoding.

The physical address of all ISA devices are mapped to 2GB + default ISA address on x86 in a 64 K address range. For example, the serial ports COM1 and COM2 reside at 0x3F8 and 0x2F8 on x86-based PCs. On the PReP reference hardware, COM1 and COM2 are mapped to 0X800003F8 and 0X800002F8. It is possible to think of ISA devices being shifted by 2 GB address as memory-mapped devices rather than I/O-mapped devices. To ensure that access to the I/O devices occurs as desired, it is necessary to add the PowerPC eieio (enforce in-order execution of I/O) instruction for access to the device. A service function eieio() is provided by the LynxOS kernel. Also, if there is a necessity to ensure completion of all writes, the PowerPC instruction sync is available as a driver service call do_sync().

PCI Support Facilities

Earlier releases of LynxOS provided a set of functions called PCI support facilities, for accessing PCI devices on PCI Bus0. These functions have been superseded by the Device Resource Manager. New device drivers should be written using the DRM, not the PCI support facilities.

Device Resource Manager

Device Resource Manager is a LynxOS module that manages device resources. The DRM assists device drivers in identifying and setting up devices, as well as accessing and managing the device resource space. Developers of new device drivers should use the DRM for managing PCI devices. Chapter 9 describes the DRM in detail.

The following figures illustrate the LynxOS virtual memory model on the PowerPC platform.