diff --git a/en_US.ISO8859-1/books/arch-handbook/driverbasics/chapter.sgml b/en_US.ISO8859-1/books/arch-handbook/driverbasics/chapter.sgml index 8dfde91426..72970d9733 100644 --- a/en_US.ISO8859-1/books/arch-handbook/driverbasics/chapter.sgml +++ b/en_US.ISO8859-1/books/arch-handbook/driverbasics/chapter.sgml @@ -1,390 +1,390 @@ Writing FreeBSD Device Drivers This chapter was written by &a.murray; with selections from a variety of sources including the intro(4) man page by &a.joerg;. Introduction This chapter provides a brief introduction to writing device drivers for FreeBSD. A device in this context is a term used mostly for hardware-related stuff that belongs to the system, like disks, printers, or a graphics display with its keyboard. A device driver is the software component of the operating system that controls a specific device. There are also so-called pseudo-devices where a device driver emulates the behaviour of a device in software without any particular underlying hardware. Device drivers can be compiled into the system statically or loaded on demand through the dynamic kernel linker facility `kld'. Most devices in a Unix-like operating system are accessed through device-nodes, sometimes also called special files. These files are usually located under the directory /dev in the file system hierarchy. Until devfs is fully integrated into FreeBSD, each device node must be created statically and independent of the existence of the associated device driver. Most device nodes on the system are created by running MAKEDEV. Device drivers can roughly be broken down into two categories; character and network device drivers. Dynamic Kernel Linker Facility - KLD The kld interface allows system administrators to dynamically add and remove functionality from a running system. This allows device driver writers to load their new changes into a running kernel without constantly rebooting to test changes. The kld interface is used through the following - administrator commands : + administrator commands: kldload - loads a new kernel module kldunload - unloads a kernel module kldstat - lists the currently loaded modules Skeleton Layout of a kernel module /* * KLD Skeleton * Inspired by Andrew Reiter's Daemonnews article */ #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ /* * Load handler that deals with the loading and unloading of a KLD. */ static int skel_loader(struct module *m, int what, void *arg) { int err = 0; switch (what) { case MOD_LOAD: /* kldload */ uprintf("Skeleton KLD loaded.\n"); break; case MOD_UNLOAD: uprintf("Skeleton KLD unloaded.\n"); break; default: err = EINVAL; break; } return(err); } /* Declare this module to the rest of the kernel */ static moduledata_t skel_mod = { "skel", skel_loader, NULL }; DECLARE_MODULE(skeleton, skel_mod, SI_SUB_KLD, SI_ORDER_ANY); Makefile FreeBSD provides a makefile include that you can use to quickly compile your kernel addition. SRCS=skeleton.c KMOD=skeleton .include <bsd.kmod.mk> Simply running make with this makefile will create a file skeleton.ko that can - be loaded into your system by typing : + be loaded into your system by typing: &prompt.root; kldload -v ./skeleton.ko Accessing a device driver Unix provides a common set of system calls for user applications to use. The upper layers of the kernel dispatch these calls to the corresponding device driver when a user accesses a device node. The /dev/MAKEDEV script makes most of the device nodes for your system but if you are doing your own driver development it may be necessary to create your own device nodes with mknod Creating static device nodes The mknod command requires four arguments to create a device node. You must specify the name of this device node, the type of device, the major number of the device, and the minor number of the device. Dynamic device nodes The device filesystem, or devfs, provides access to the kernel's device namespace in the global filesystem namespace. This eliminates the problems of potentially having a device driver without a static device node, or a device node without an installed device driver. Devfs is still a work in progress, but it is already working quite nice. Character Devices A character device driver is one that transfers data directly to and from a user process. This is the most common type of device driver and there are plenty of simple examples in the source tree. This simple example pseudo-device remembers whatever values you write to it and can then supply them back to you when you read from it. /* * Simple `echo' pseudo-device KLD * * Murray Stokely */ #define MIN(a,b) (((a) < (b)) ? (a) : (b)) #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ #include <sys/conf.h> /* cdevsw struct */ #include <sys/uio.h> /* uio struct */ #include <sys/malloc.h> #define BUFFERSIZE 256 /* Function prototypes */ d_open_t echo_open; d_close_t echo_close; d_read_t echo_read; d_write_t echo_write; /* Character device entry points */ static struct cdevsw echo_cdevsw = { echo_open, echo_close, echo_read, echo_write, noioctl, nopoll, nommap, nostrategy, "echo", 33, /* reserved for lkms - /usr/src/sys/conf/majors */ nodump, nopsize, D_TTY, -1 }; typedef struct s_echo { char msg[BUFFERSIZE]; int len; } t_echo; /* vars */ static dev_t sdev; static int len; static int count; static t_echo *echomsg; MALLOC_DECLARE(M_ECHOBUF); MALLOC_DEFINE(M_ECHOBUF, "echobuffer", "buffer for echo module"); /* * This function acts is called by the kld[un]load(2) system calls to * determine what actions to take when a module is loaded or unloaded. */ static int echo_loader(struct module *m, int what, void *arg) { int err = 0; switch (what) { case MOD_LOAD: /* kldload */ sdev = make_dev(&echo_cdevsw, 0, UID_ROOT, GID_WHEEL, 0600, "echo"); /* kmalloc memory for use by this driver */ /* malloc(256,M_ECHOBUF,M_WAITOK); */ MALLOC(echomsg, t_echo *, sizeof(t_echo), M_ECHOBUF, M_WAITOK); printf("Echo device loaded.\n"); break; case MOD_UNLOAD: destroy_dev(sdev); FREE(echomsg,M_ECHOBUF); printf("Echo device unloaded.\n"); break; default: err = EINVAL; break; } return(err); } int echo_open(dev_t dev, int oflags, int devtype, struct proc *p) { int err = 0; uprintf("Opened device \"echo\" successfully.\n"); return(err); } int echo_close(dev_t dev, int fflag, int devtype, struct proc *p) { uprintf("Closing device \"echo.\"\n"); return(0); } /* * The read function just takes the buf that was saved via * echo_write() and returns it to userland for accessing. * uio(9) */ int echo_read(dev_t dev, struct uio *uio, int ioflag) { int err = 0; int amt; /* How big is this read operation? Either as big as the user wants, or as big as the remaining data */ amt = MIN(uio->uio_resid, (echomsg->len - uio->uio_offset > 0) ? echomsg->len - uio->uio_offset : 0); if ((err = uiomove(echomsg->msg + uio->uio_offset,amt,uio)) != 0) { uprintf("uiomove failed!\n"); } return err; } /* * echo_write takes in a character string and saves it * to buf for later accessing. */ int echo_write(dev_t dev, struct uio *uio, int ioflag) { int err = 0; /* Copy the string in from user memory to kernel memory */ err = copyin(uio->uio_iov->iov_base, echomsg->msg, MIN(uio->uio_iov->iov_len,BUFFERSIZE)); /* Now we need to null terminate */ *(echomsg->msg + MIN(uio->uio_iov->iov_len,BUFFERSIZE)) = 0; /* Record the length */ echomsg->len = MIN(uio->uio_iov->iov_len,BUFFERSIZE); if (err != 0) { uprintf("Write failed: bad address!\n"); } count++; return(err); } DEV_MODULE(echo,echo_loader,NULL); To install this driver you will first need to make a node on - your filesystem with a command such as : + your filesystem with a command such as: &prompt.root; mknod /dev/echo c 33 0 With this driver loaded you should now be able to type - something like : + something like: &prompt.root; echo -n "Test Data" > /dev/echo &prompt.root; cat /dev/echo Test Data Real hardware devices in the next chapter.. Additional Resources Dynamic Kernel Linker (KLD) Facility Programming Tutorial - Daemonnews October 2000 How to Write Kernel Drivers with NEWBUS - Daemonnews July 2000 Network Drivers Drivers for network devices do not use device nodes in order to be accessed. Their selection is based on other decisions made inside the kernel and instead of calling open(), use of a network device is generally introduced by using the system call socket(2). man ifnet(), loopback device, Bill Paul's drivers, etc.. diff --git a/en_US.ISO8859-1/books/arch-handbook/isa/chapter.sgml b/en_US.ISO8859-1/books/arch-handbook/isa/chapter.sgml index dde9cd3968..f90c0067c3 100644 --- a/en_US.ISO8859-1/books/arch-handbook/isa/chapter.sgml +++ b/en_US.ISO8859-1/books/arch-handbook/isa/chapter.sgml @@ -1,2479 +1,2479 @@ ISA device drivers This chapter was written by &a.babkin; Modifications for the handbook made by &a.murray;, &a.wylie;, and &a.logo;. Synopsis This chapter introduces the issues relevant to writing a driver for an ISA device. The pseudo-code presented here is rather detailed and reminiscent of the real code but is still only pseudo-code. It avoids the details irrelevant to the subject of the discussion. The real-life examples can be found in the source code of real drivers. In particular the drivers "ep" and "aha" are good sources of information. Basic information A typical ISA driver would need the following include files: #include <sys/module.h> #include <sys/bus.h> #include <machine/bus.h> #include <machine/resource.h> #include <sys/rman.h> #include <isa/isavar.h> #include <isa/pnpvar.h> They describe the things specific to the ISA and generic bus subsystem. The bus subsystem is implemented in an object-oriented fashion, its main structures are accessed by associated method functions. The list of bus methods implemented by an ISA driver is like one for any other bus. For a hypothetical driver named "xxx" they would be: static void xxx_isa_identify (driver_t *, device_t); Normally used for bus drivers, not device drivers. But for ISA devices this method may have special use: if the device provides some device-specific (non-PnP) way to auto-detect devices this routine may implement it. static int xxx_isa_probe (device_t dev); Probe for a device at a known (or PnP) location. This routine can also accommodate device-specific auto-detection of parameters for partially configured devices. static int xxx_isa_attach (device_t dev); Attach and initialize device. static int xxx_isa_detach (device_t dev); Detach device before unloading the driver module. static int xxx_isa_shutdown (device_t dev); Execute shutdown of the device before system shutdown. static int xxx_isa_suspend (device_t dev); Suspend the device before the system goes to the power-save state. May also abort transition to the power-save state. static int xxx_isa_resume (device_t dev); Resume the device activity after return from power-save state. xxx_isa_probe() and xxx_isa_attach() are mandatory, the rest of the routines are optional, depending on the device's needs. The driver is linked to the system with the following set of descriptions. /* table of supported bus methods */ static device_method_t xxx_isa_methods[] = { /* list all the bus method functions supported by the driver */ /* omit the unsupported methods */ DEVMETHOD(device_identify, xxx_isa_identify), DEVMETHOD(device_probe, xxx_isa_probe), DEVMETHOD(device_attach, xxx_isa_attach), DEVMETHOD(device_detach, xxx_isa_detach), DEVMETHOD(device_shutdown, xxx_isa_shutdown), DEVMETHOD(device_suspend, xxx_isa_suspend), DEVMETHOD(device_resume, xxx_isa_resume), { 0, 0 } }; static driver_t xxx_isa_driver = { "xxx", xxx_isa_methods, sizeof(struct xxx_softc), }; static devclass_t xxx_devclass; DRIVER_MODULE(xxx, isa, xxx_isa_driver, xxx_devclass, load_function, load_argument); Here struct xxx_softc is a device-specific structure that contains private driver data and descriptors for the driver's resources. The bus code automatically allocates one softc descriptor per device as needed. If the driver is implemented as a loadable module then load_function() is called to do driver-specific initialization or clean-up when the driver is loaded or unloaded and load_argument is passed as one of its arguments. If the driver does not support dynamic loading (in other words it must always be linked into kernel) then these values should be set to 0 and the last definition would look like: DRIVER_MODULE(xxx, isa, xxx_isa_driver, xxx_devclass, 0, 0); If the driver is for a device which supports PnP then a table of supported PnP IDs must be defined. The table consists of a list of PnP IDs supported by this driver and human-readable descriptions of the hardware types and models having these IDs. It looks like: static struct isa_pnp_id xxx_pnp_ids[] = { /* a line for each supported PnP ID */ { 0x12345678, "Our device model 1234A" }, { 0x12345679, "Our device model 1234B" }, { 0, NULL }, /* end of table */ }; If the driver does not support PnP devices it still needs an empty PnP ID table, like: static struct isa_pnp_id xxx_pnp_ids[] = { { 0, NULL }, /* end of table */ }; Device_t pointer Device_t is the pointer type for the device structure. Here we consider only the methods interesting from the device driver writer's standpoint. The methods to manipulate values in the device structure are: device_t device_get_parent(dev) Get the parent bus of a device. driver_t device_get_driver(dev) Get pointer to its driver structure. char *device_get_name(dev) Get the driver name, such as "xxx" for our example. int device_get_unit(dev) Get the unit number (units are numbered from 0 for the devices associated with each driver). char *device_get_nameunit(dev) Get the device name - including the unit number, such as "xxx0" , "xxx1" and so + including the unit number, such as "xxx0", "xxx1" and so on. char *device_get_desc(dev) Get the device description. Normally it describes the exact model of device in human-readable form. device_set_desc(dev, desc) Set the description. This makes the device description point to the string desc which may not be deallocated or changed after that. device_set_desc_copy(dev, desc) Set the description. The description is copied into an internal dynamically allocated buffer, so the string desc may be changed afterwards without adverse effects. void *device_get_softc(dev) Get pointer to the device descriptor (struct xxx_softc) associated with this device. u_int32_t device_get_flags(dev) Get the flags specified for the device in the configuration file. A convenience function device_printf(dev, fmt, ...) may be used to print the messages from the device driver. It automatically prepends the unitname and colon to the message. The device_t methods are implemented in the file kern/bus_subr.c. Config file and the order of identifying and probing during auto-configuration The ISA devices are described in the kernel config file like: device xxx0 at isa? port 0x300 irq 10 drq 5 iomem 0xd0000 flags 0x1 sensitive The values of port, IRQ and so on are converted to the resource values associated with the device. They are optional, depending on the device needs and abilities for auto-configuration. For example, some devices do not need DRQ at all and some allow the driver to read the IRQ setting from the device configuration ports. If a machine has multiple ISA buses the exact bus may be specified in the configuration line, like "isa0" or "isa1", otherwise the device would be searched for on all the ISA buses. "sensitive" is a resource requesting that this device must be probed before all non-sensitive devices. It is supported but does not seem to be used in any current driver. For legacy ISA devices in many cases the drivers are still able to detect the configuration parameters. But each device to be configured in the system must have a config line. If two devices of some type are installed in the system but there is only one configuration line for the corresponding driver, ie: device xxx0 at isa? then only one device will be configured. But for the devices supporting automatic identification by the means of Plug-n-Play or some proprietary protocol one configuration line is enough to configure all the devices in the system, like the one above or just simply: device xxx at isa? If a driver supports both auto-identified and legacy devices and both kinds are installed at once in one machine then it is enough to describe in the config file the legacy devices only. The auto-identified devices will be added automatically. When an ISA bus is auto-configured the events happen as follows: All the drivers' identify routines (including the PnP identify routine which identifies all the PnP devices) are called in random order. As they identify the devices they add them to the list on the ISA bus. Normally the drivers' identify routines associate their drivers with the new devices. The PnP identify routine does not know about the other drivers yet so it does not associate any with the new devices it adds. The PnP devices are put to sleep using the PnP protocol to prevent them from being probed as legacy devices. The probe routines of non-PnP devices marked as "sensitive" are called. If probe for a device went successfully, the attach routine is called for it. The probe and attach routines of all non-PNP devices are called likewise. The PnP devices are brought back from the sleep state and assigned the resources they request: I/O and memory address ranges, IRQs and DRQs, all of them not conflicting with the attached legacy devices. Then for each PnP device the probe routines of all the present ISA drivers are called. The first one that claims the device gets attached. It is possible that multiple drivers would claim the device with different priority, the highest-priority driver wins. The probe routines must call ISA_PNP_PROBE() to compare the actual PnP ID with the list of the IDs supported by the driver and if the ID is not in the table return failure. That means that absolutely every driver, even the ones not supporting any PnP devices must call ISA_PNP_PROBE(), at least with an empty PnP ID table to return failure on unknown PnP devices. The probe routine returns a positive value (the error code) on error, zero or negative value on success. The negative return values are used when a PnP device supports multiple interfaces. For example, an older compatibility interface and a newer advanced interface which are supported by different drivers. Then both drivers would detect the device. The driver which returns a higher value in the probe routine takes precedence (in other words, the driver returning 0 has highest precedence, returning -1 is next, returning -2 is after it and so on). In result the devices which support only the old interface will be handled by the old driver (which should return -1 from the probe routine) while the devices supporting the new interface as well will be handled by the new driver (which should return 0 from the probe routine). If multiple drivers return the same value then the one called first wins. So if a driver returns value 0 it may be sure that it won the priority arbitration. The device-specific identify routines can also assign not a driver but a class of drivers to the device. Then all the drivers in the class are probed for this device, like the case with PnP. This feature is not implemented in any existing driver and is not considered further in this document. Because the PnP devices are disabled when probing the legacy devices they will not be attached twice (once as legacy and once as PnP). But in case of device-dependent identify routines it is the responsibility of the driver to make sure that the same device will not be attached by the driver twice: once as legacy user-configured and once as auto-identified. Another practical consequence for the auto-identified devices (both PnP and device-specific) is that the flags can not be passed to them from the kernel configuration file. So they must either not use the flags at all or use the flags from the device unit 0 for all the auto-identified devices or use the sysctl interface instead of flags. Other unusual configurations may be accommodated by accessing the configuration resources directly with functions of families resource_query_*() and resource_*_value(). Their implementations are located in kern/subr_bus.h. The old IDE disk driver i386/isa/wd.c contains examples of such use. But the standard means of configuration must always be preferred. Leave parsing the configuration resources to the bus configuration code. Resources The information that a user enters into the kernel configuration file is processed and passed to the kernel as configuration resources. This information is parsed by the bus configuration code and transformed into a value of structure device_t and the bus resources associated with it. The drivers may access the configuration resources directly using functions resource_* for more complex cases of configuration. But generally it is not needed nor recommended, so this issue is not discussed further. The bus resources are associated with each device. They are identified by type and number within the type. For the ISA bus the following types are defined: SYS_RES_IRQ - interrupt number SYS_RES_DRQ - ISA DMA channel number SYS_RES_MEMORY - range of device memory mapped into the system memory space SYS_RES_IOPORT - range of device I/O registers The enumeration within types starts from 0, so if a device has two memory regions if would have resources of type SYS_RES_MEMORY numbered 0 and 1. The resource type has nothing to do with the C language type, all the resource values have the C language type "unsigned long" and must be cast as necessary. The resource numbers do not have to be contiguous although for ISA they normally would be. The permitted resource numbers for ISA devices are: IRQ: 0-1 DRQ: 0-1 MEMORY: 0-3 IOPORT: 0-7 All the resources are represented as ranges, with a start value and count. For IRQ and DRQ resources the count would be normally equal to 1. The values for memory refer to the physical addresses. Three types of activities can be performed on resources: set/get allocate/release activate/deactivate Setting sets the range used by the resource. Allocation reserves the requested range that no other driver would be able to reserve it (and checking that no other driver reserved this range already). Activation makes the resource accessible to the driver doing whatever is necessary for that (for example, for memory it would be mapping into the kernel virtual address space). The functions to manipulate resources are: int bus_set_resource(device_t dev, int type, int rid, u_long start, u_long count) Set a range for a resource. Returns 0 if successful, error code otherwise. Normally the only reason this function would return an error is value of type, rid, start or count out of permitted range. dev - driver's device type - type of resource, SYS_RES_* rid - resource number (ID) within type start, count - resource range int bus_get_resource(device_t dev, int type, int rid, u_long *startp, u_long *countp) Get the range of resource. Returns 0 if successful, error code if the resource is not defined yet. u_long bus_get_resource_start(device_t dev, int type, int rid) u_long bus_get_resource_count (device_t dev, int type, int rid) Convenience functions to get only the start or count. Return 0 in case of error, so if the resource start has 0 among the legitimate values it would be impossible to tell if the value is 0 or an error occurred. Luckily, no ISA resources for add-on drivers may have a start value equal 0. void bus_delete_resource(device_t dev, int type, int rid) Delete a resource, make it undefined. struct resource * bus_alloc_resource(device_t dev, int type, int *rid, u_long start, u_long end, u_long count, u_int flags) Allocate a resource as a range of count values not allocated by anyone else, somewhere between start and end. Alas, alignment is not supported. If the resource was not set yet it is automatically created. The special values of start 0 and end ~0 (all ones) means that the fixed values previously set by bus_set_resource() must be used instead: start and count as themselves and end=(start+count), in this case if the resource was not defined before then an error is returned. Although rid is passed by reference it is not set anywhere by the resource allocation code of the ISA bus. (The other buses may use a different approach and modify it). Flags are a bitmap, the flags interesting for the caller are: RF_ACTIVE - causes the resource to be automatically activated after allocation. RF_SHAREABLE - resource may be shared at the same time by multiple drivers. RF_TIMESHARE - resource may be time-shared by multiple drivers, i.e. allocated at the same time by many but activated only by one at any given moment of time. Returns 0 on error. The allocated values may be obtained from the returned handle using methods rhand_*(). int bus_release_resource(device_t dev, int type, int rid, struct resource *r) Release the resource, r is the handle returned by bus_alloc_resource(). Returns 0 on success, error code otherwise. int bus_activate_resource(device_t dev, int type, int rid, struct resource *r) int bus_deactivate_resource(device_t dev, int type, int rid, struct resource *r) Activate or deactivate resource. Return 0 on success, error code otherwise. If the resource is time-shared and currently activated by another driver then EBUSY is returned. int bus_setup_intr(device_t dev, struct resource *r, int flags, driver_intr_t *handler, void *arg, void **cookiep) int bus_teardown_intr(device_t dev, struct resource *r, void *cookie) Associate or de-associate the interrupt handler with a device. Return 0 on success, error code otherwise. r - the activated resource handler describing the IRQ flags - the interrupt priority level, one of: INTR_TYPE_TTY - terminals and other likewise character-type devices. To mask them use spltty(). (INTR_TYPE_TTY | INTR_TYPE_FAST) - terminal type devices with small input buffer, critical to the data loss on input (such as the old-fashioned serial ports). To mask them use spltty(). INTR_TYPE_BIO - block-type devices, except those on the CAM controllers. To mask them use splbio(). INTR_TYPE_CAM - CAM (Common Access Method) bus controllers. To mask them use splcam(). INTR_TYPE_NET - network interface controllers. To mask them use splimp(). INTR_TYPE_MISC - miscellaneous devices. There is no other way to mask them than by splhigh() which masks all interrupts. When an interrupt handler executes all the other interrupts matching its priority level will be masked. The only exception is the MISC level for which no other interrupts are masked and which is not masked by any other interrupt. handler - pointer to the handler function, the type driver_intr_t is defined as "void driver_intr_t(void *)" arg - the argument passed to the handler to identify this particular device. It is cast from void* to any real type by the handler. The old convention for the ISA interrupt handlers was to use the unit number as argument, the new (recommended) convention is using a pointer to the device softc structure. cookie[p] - the value received from setup() is used to identify the handler when passed to teardown() A number of methods is defined to operate on the resource handlers (struct resource *). Those of interest to the device driver writers are: u_long rman_get_start(r) u_long rman_get_end(r) Get the start and end of allocated resource range. void *rman_get_virtual(r) Get the virtual address of activated memory resource. Bus memory mapping In many cases data is exchanged between the driver and the device through the memory. Two variants are possible: (a) memory is located on the device card (b) memory is the main memory of computer In the case (a) the driver always copies the data back and forth between the on-card memory and the main memory as necessary. To map the on-card memory into the kernel virtual address space the physical address and length of the on-card memory must be defined as a SYS_RES_MEMORY resource. That resource can then be allocated and activated, and its virtual address obtained using rman_get_virtual(). The older drivers used the function pmap_mapdev() for this purpose, which should not be used directly any more. Now it is one of the internal steps of resource activation. Most of the ISA cards will have their memory configured for physical location somewhere in range 640KB-1MB. Some of the ISA cards require larger memory ranges which should be placed somewhere under 16MB (because of the 24-bit address limitation on the ISA bus). In that case if the machine has more memory than the start address of the device memory (in other words, they overlap) a memory hole must be configured at the address range used by devices. Many BIOSes allow to configure a memory hole of 1MB starting at 14MB or 15MB. FreeBSD can handle the memory holes properly if the BIOS reports them properly (old BIOSes may have this feature broken). In the case (b) just the address of the data is sent to the device, and the device uses DMA to actually access the data in the main memory. Two limitations are present: First, ISA cards can only access memory below 16MB. Second, the contiguous pages in virtual address space may not be contiguous in physical address space, so the device may have to do scatter/gather operations. The bus subsystem provides ready solutions for some of these problems, the rest has to be done by the drivers themselves. Two structures are used for DMA memory allocation, bus_dma_tag_t and bus_dmamap_t. Tag describes the properties required for the DMA memory. Map represents a memory block allocated according to these properties. Multiple maps may be associated with the same tag. Tags are organized into a tree-like hierarchy with inheritance of the properties. A child tag inherits all the requirements of its parent tag or may make them more strict but never more loose. Normally one top-level tag (with no parent) is created for each device unit. If multiple memory areas with different requirements are needed for each device then a tag for each of them may be created as a child of the parent tag. The tags can be used to create a map in two ways. First, a chunk of contiguous memory conformant with the tag requirements may be allocated (and later may be freed). This is normally used to allocate relatively long-living areas of memory for communication with the device. Loading of such memory into a map is trivial: it is always considered as one chunk in the appropriate physical memory range. Second, an arbitrary area of virtual memory may be loaded into a map. Each page of this memory will be checked for conformance to the map requirement. If it conforms then it is left at its original location. If it is not then a fresh conformant "bounce page" is allocated and used as intermediate storage. When writing the data from the non-conformant original pages they will be copied to their bounce pages first and then transferred from the bounce pages to the device. When reading the data would go from the device to the bounce pages and then copied to their non-conformant original pages. The process of copying between the original and bounce pages is called synchronization. This is normally used on per-transfer basis: buffer for each transfer would be loaded, transfer done and buffer unloaded. The functions working on the DMA memory are: int bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment, bus_size_t boundary, bus_addr_t lowaddr, bus_addr_t highaddr, bus_dma_filter_t *filter, void *filterarg, bus_size_t maxsize, int nsegments, bus_size_t maxsegsz, int flags, bus_dma_tag_t *dmat) Create a new tag. Returns 0 on success, the error code otherwise. parent - parent tag, or NULL to create a top-level tag alignment - required physical alignment of the memory area to be allocated for this tag. Use value 1 for "no specific alignment". Applies only to the future bus_dmamem_alloc() but not bus_dmamap_create() calls. boundary - physical address boundary that must not be crossed when allocating the memory. Use value 0 for "no boundary". Applies only to the future bus_dmamem_alloc() but not bus_dmamap_create() calls. Must be power of 2. If the memory is planned to be used in non-cascaded DMA mode (i.e. the DMA addresses will be supplied not by the device itself but by the ISA DMA controller) then the boundary must be no larger than 64KB (64*1024) due to the limitations of the DMA hardware. lowaddr, highaddr - the names are slighlty misleading; these values are used to limit the permitted range of physical addresses used to allocate the memory. The exact meaning varies depending on the planned future use: For bus_dmamem_alloc() all the addresses from 0 to lowaddr-1 are considered permitted, the higher ones are forbidden. For bus_dmamap_create() all the addresses outside the inclusive range [lowaddr; highaddr] are considered accessible. The addresses of pages inside the range are passed to the filter function which decides if they are accessible. If no filter function is supplied then all the range is considered unaccessible. For the ISA devices the normal values (with no filter function) are: lowaddr = BUS_SPACE_MAXADDR_24BIT highaddr = BUS_SPACE_MAXADDR filter, filterarg - the filter function and its argument. If NULL is passed for filter then the whole range [lowaddr, highaddr] is considered unaccessible when doing bus_dmamap_create(). Otherwise the physical address of each attempted page in range [lowaddr; highaddr] is passed to the filter function which decides if it is accessible. The prototype of the filter function is: int filterfunc(void *arg, bus_addr_t paddr) It must return 0 if the page is accessible, non-zero otherwise. maxsize - the maximal size of memory (in bytes) that may be allocated through this tag. In case it is difficult to estimate or could be arbitrarily big, the value for ISA devices would be BUS_SPACE_MAXSIZE_24BIT. nsegments - maximal number of scatter-gather segments supported by the device. If unrestricted then the value BUS_SPACE_UNRESTRICTED should be used. This value is recommended for the parent tags, the actual restrictions would then be specified for the descendant tags. Tags with nsegments equal to BUS_SPACE_UNRESTRICTED may not be used to actually load maps, they may be used only as parent tags. The practical limit for nsegments seems to be about 250-300, higher values will cause kernel stack overflow. But anyway the hardware normally can not support that many scatter-gather buffers. maxsegsz - maximal size of a scatter-gather segment supported by the device. The maximal value for ISA device would be BUS_SPACE_MAXSIZE_24BIT. flags - a bitmap of flags. The only interesting flags are: BUS_DMA_ALLOCNOW - requests to allocate all the potentially needed bounce pages when creating the tag BUS_DMA_ISA - mysterious flag used only on Alpha machines. It is not defined for the i386 machines. Probably it should be used by all the ISA drivers for Alpha machines but it looks like there are no such drivers yet. dmat - pointer to the storage for the new tag to be returned int bus_dma_tag_destroy(bus_dma_tag_t dmat) Destroy a tag. Returns 0 on success, the error code otherwise. dmat - the tag to be destroyed int bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags, bus_dmamap_t *mapp) Allocate an area of contiguous memory described by the tag. The size of memory to be allocated is tag's maxsize. Returns 0 on success, the error code otherwise. The result still has to be loaded by bus_dmamap_load() before used to get the physical address of the memory. dmat - the tag vaddr - pointer to the storage for the kernel virtual address of the allocated area to be returned. flags - a bitmap of flags. The only interesting flag is: BUS_DMA_NOWAIT - if the memory is not immediately available return the error. If this flag is not set then the routine is allowed to sleep waiting until the memory will become available. mapp - pointer to the storage for the new map to be returned void bus_dmamem_free(bus_dma_tag_t dmat, void *vaddr, bus_dmamap_t map) Free the memory allocated by bus_dmamem_alloc(). As of now freeing of the memory allocated with ISA restrictions is not implemented. Because of this the recommended model of use is to keep and re-use the allocated areas for as long as possible. Do not lightly free some area and then shortly allocate it again. That does not mean that bus_dmamem_free() should not be used at all: hopefully it will be properly implemented soon. dmat - the tag vaddr - the kernel virtual address of the memory map - the map of the memory (as returned from bus_dmamem_alloc()) int bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp) Create a map for the tag, to be used in bus_dmamap_load() later. Returns 0 on success, the error code otherwise. dmat - the tag flags - theoretically, a bit map of flags. But no flags are defined yet, so as of now it will be always 0. mapp - pointer to the storage for the new map to be returned int bus_dmamap_destroy(bus_dma_tag_t dmat, bus_dmamap_t map) Destroy a map. Returns 0 on success, the error code otherwise. dmat - the tag to which the map is associated map - the map to be destroyed int bus_dmamap_load(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf, bus_size_t buflen, bus_dmamap_callback_t *callback, void *callback_arg, int flags) Load a buffer into the map (the map must be previously created by bus_dmamap_create() or bus_dmamem_alloc()). All the pages of the buffer are checked for conformance to the tag requirements and for those not conformant the bounce pages are allocated. An array of physical segment descriptors is built and passed to the callback routine. This callback routine is then expected to handle it in some way. The number of bounce buffers in the system is limited, so if the bounce buffers are needed but not immediately available the request will be queued and the callback will be called when the bounce buffers will become available. Returns 0 if the callback was executed immediately or EINPROGRESS if the request was queued for future execution. In the latter case the synchronization with queued callback routine is the responsibility of the driver. dmat - the tag map - the map buf - kernel virtual address of the buffer buflen - length of the buffer callback, callback_arg - the callback function and its argument The prototype of callback function is: void callback(void *arg, bus_dma_segment_t *seg, int nseg, int error) arg - the same as callback_arg passed to bus_dmamap_load() seg - array of the segment descriptors nseg - number of descriptors in array error - indication of the segment number overflow: if it is set to EFBIG then the buffer did not fit into the maximal number of segments permitted by the tag. In this case only the permitted number of descriptors will be in the array. Handling of this situation is up to the driver: depending on the desired semantics it can either consider this an error or split the buffer in two and handle the second part separately Each entry in the segments array contains the fields: ds_addr - physical bus address of the segment ds_len - length of the segment void bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map) unload the map. dmat - tag map - loaded map void bus_dmamap_sync (bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op) Synchronise a loaded buffer with its bounce pages before and after physical transfer to or from device. This is the function that does all the necessary copying of data between the original buffer and its mapped version. The buffers must be synchronized both before and after doing the transfer. dmat - tag map - loaded map op - type of synchronization operation to perform: BUS_DMASYNC_PREREAD - before reading from device into buffer BUS_DMASYNC_POSTREAD - after reading from device into buffer BUS_DMASYNC_PREWRITE - before writing the buffer to device BUS_DMASYNC_POSTWRITE - after writing the buffer to device As of now PREREAD and POSTWRITE are null operations but that may change in the future, so they must not be ignored in the driver. Synchronization is not needed for the memory obtained from bus_dmamem_alloc(). Before calling the callback function from bus_dmamap_load() the segment array is stored in the stack. And it gets pre-allocated for the maximal number of segments allowed by the tag. Because of this the practical limit for the number of segments on i386 architecture is about 250-300 (the kernel stack is 4KB minus the size of the user structure, size of a segment array entry is 8 bytes, and some space must be left). Because the array is allocated based on the maximal number this value must not be set higher than really needed. Fortunately, for most of hardware the maximal supported number of segments is much lower. But if the driver wants to handle buffers with a very large number of scatter-gather segments it should do that in portions: load part of the buffer, transfer it to the device, load next part of the buffer, and so on. Another practical consequence is that the number of segments may limit the size of the buffer. If all the pages in the buffer happen to be physically non-contiguous then the maximal supported buffer size for that fragmented case would be (nsegments * page_size). For example, if a maximal number of 10 segments is supported then on i386 maximal guaranteed supported buffer size would be 40K. If a higher size is desired then special tricks should be used in the driver. If the hardware does not support scatter-gather at all or the driver wants to support some buffer size even if it is heavily fragmented then the solution is to allocate a contiguous buffer in the driver and use it as intermediate storage if the original buffer does not fit. Below are the typical call sequences when using a map depend on the use of the map. The characters -> are used to show the flow of time. For a buffer which stays practically fixed during all the time between attachment and detachment of a device: bus_dmamem_alloc -> bus_dmamap_load -> ...use buffer... -> -> bus_dmamap_unload -> bus_dmamem_free For a buffer that changes frequently and is passed from outside the driver: bus_dmamap_create -> -> bus_dmamap_load -> bus_dmamap_sync(PRE...) -> do transfer -> -> bus_dmamap_sync(POST...) -> bus_dmamap_unload -> ... -> bus_dmamap_load -> bus_dmamap_sync(PRE...) -> do transfer -> -> bus_dmamap_sync(POST...) -> bus_dmamap_unload -> -> bus_dmamap_destroy When loading a map created by bus_dmamem_alloc() the passed address and size of the buffer must be the same as used in bus_dmamem_alloc(). In this case it is guaranteed that the whole buffer will be mapped as one segment (so the callback may be based on this assumption) and the request will be executed immediately (EINPROGRESS will never be returned). All the callback needs to do in this case is to save the physical address. A typical example would be: static void alloc_callback(void *arg, bus_dma_segment_t *seg, int nseg, int error) { *(bus_addr_t *)arg = seg[0].ds_addr; } ... int error; struct somedata { .... }; struct somedata *vsomedata; /* virtual address */ bus_addr_t psomedata; /* physical bus-relative address */ bus_dma_tag_t tag_somedata; bus_dmamap_t map_somedata; ... error=bus_dma_tag_create(parent_tag, alignment, boundary, lowaddr, highaddr, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ sizeof(struct somedata), /*nsegments*/ 1, /*maxsegsz*/ sizeof(struct somedata), /*flags*/ 0, &tag_somedata); if(error) return error; error = bus_dmamem_alloc(tag_somedata, &vsomedata, /* flags*/ 0, &map_somedata); if(error) return error; bus_dmamap_load(tag_somedata, map_somedata, (void *)vsomedata, sizeof (struct somedata), alloc_callback, (void *) &psomedata, /*flags*/0); Looks a bit long and complicated but that is the way to do it. The practical consequence is: if multiple memory areas are allocated always together it would be a really good idea to combine them all into one structure and allocate as one (if the alignment and boundary limitations permit). When loading an arbitrary buffer into the map created by bus_dmamap_create() special measures must be taken to synchronize with the callback in case it would be delayed. The code would look like: { int s; int error; s = splsoftvm(); error = bus_dmamap_load( dmat, dmamap, buffer_ptr, buffer_len, callback, /*callback_arg*/ buffer_descriptor, /*flags*/0); if (error == EINPROGRESS) { /* * Do whatever is needed to ensure synchronization * with callback. Callback is guaranteed not to be started * until we do splx() or tsleep(). */ } splx(s); } Two possible approaches for the processing of requests are: 1. If requests are completed by marking them explicitly as done (such as the CAM requests) then it would be simpler to put all the further processing into the callback driver which would mark the request when it is done. Then not much extra synchronization is needed. For the flow control reasons it may be a good idea to freeze the request queue until this request gets completed. 2. If requests are completed when the function returns (such as classic read or write requests on character devices) then a synchronization flag should be set in the buffer descriptor and tsleep() called. Later when the callback gets called it will do its processing and check this synchronization flag. If it is set then the callback should issue a wakeup. In this approach the callback function could either do all the needed processing (just like the previous case) or simply save the segments array in the buffer descriptor. Then after callback completes the calling function could use this saved segments array and do all the processing. DMA The Direct Memory Access (DMA) is implemented in the ISA bus through the DMA controller (actually, two of them but that is an irrelevant detail). To make the early ISA devices simple and cheap the logic of the bus control and address generation was concentrated in the DMA controller. Fortunately, FreeBSD provides a set of functions that mostly hide the annoying details of the DMA controller from the device drivers. The simplest case is for the fairly intelligent devices. Like the bus master devices on PCI they can generate the bus cycles and memory addresses all by themselves. The only thing they really need from the DMA controller is bus arbitration. So for this purpose they pretend to be cascaded slave DMA controllers. And the only thing needed from the system DMA controller is to enable the cascaded mode on a DMA channel by calling the following function when attaching the driver: void isa_dmacascade(int channel_number) All the further activity is done by programming the device. When detaching the driver no DMA-related functions need to be called. For the simpler devices things get more complicated. The functions used are: int isa_dma_acquire(int chanel_number) Reserve a DMA channel. Returns 0 on success or EBUSY if the channel was already reserved by this or a different driver. Most of the ISA devices are not able to share DMA channels anyway, so normally this function is called when attaching a device. This reservation was made redundant by the modern interface of bus resources but still must be used in addition to the latter. If not used then later, other DMA routines will panic. int isa_dma_release(int chanel_number) Release a previously reserved DMA channel. No transfers must be in progress when the channel is released (as well as the device must not try to initiate transfer after the channel is released). void isa_dmainit(int chan, u_int bouncebufsize) Allocate a bounce buffer for use with the specified channel. The requested size of the buffer can not exceed 64KB. This bounce buffer will be automatically used later if a transfer buffer happens to be not physically contiguous or outside of the memory accessible by the ISA bus or crossing the 64KB boundary. If the transfers will be always done from buffers which conform to these conditions (such as those allocated by bus_dmamem_alloc() with proper limitations) then isa_dmainit() does not have to be called. But it is quite convenient to transfer arbitrary data using the DMA controller. The bounce buffer will automatically care of the scatter-gather issues. chan - channel number bouncebufsize - size of the bounce buffer in bytes void isa_dmastart(int flags, caddr_t addr, u_int nbytes, int chan) Prepare to start a DMA transfer. This function must be called to set up the DMA controller before actually starting transfer on the device. It checks that the buffer is contiguous and falls into the ISA memory range, if not then the bounce buffer is automatically used. If bounce buffer is required but not set up by isa_dmainit() or too small for the requested transfer size then the system will panic. In case of a write request with bounce buffer the data will be automatically copied to the bounce buffer. flags - a bitmask determining the type of operation to be done. The direction bits B_READ and B_WRITE are mutually exclusive. B_READ - read from the ISA bus into memory B_WRITE - write from the memory to the ISA bus B_RAW - if set then the DMA controller will remember the buffer and after the end of transfer will automatically re-initialize itself to repeat transfer of the same buffer again (of course, the driver may change the data in the buffer before initiating another transfer in the device). If not set then the parameters will work only for one transfer, and isa_dmastart() will have to be called again before initiating the next transfer. Using B_RAW makes sense only if the bounce buffer is not used. addr - virtual address of the buffer nbytes - length of the buffer. Must be less or equal to 64KB. Length of 0 is not allowed: the DMA controller will understand it as 64KB while the kernel code will understand it as 0 and that would cause unpredictable effects. For channels number 4 and higher the length must be even because these channels transfer 2 bytes at a time. In case of an odd length the last byte will not be transferred. chan - channel number void isa_dmadone(int flags, caddr_t addr, int nbytes, int chan) Synchronize the memory after device reports that transfer is done. If that was a read operation with a bounce buffer then the data will be copied from the bounce buffer to the original buffer. Arguments are the same as for isa_dmastart(). Flag B_RAW is permitted but it does not affect isa_dmadone() in any way. int isa_dmastatus(int channel_number) Returns the number of bytes left in the current transfer to be transferred. In case the flag B_READ was set in isa_dmastart() the number returned will never be equal to zero. At the end of transfer it will be automatically reset back to the length of buffer. The normal use is to check the number of bytes left after the device signals that the transfer is completed. If the number of bytes is not 0 then probably something went wrong with that transfer. int isa_dmastop(int channel_number) Aborts the current transfer and returns the number of bytes left untransferred. xxx_isa_probe This function probes if a device is present. If the driver supports auto-detection of some part of device configuration (such as interrupt vector or memory address) this auto-detection must be done in this routine. As for any other bus, if the device cannot be detected or is detected but failed the self-test or some other problem happened then it returns a positive value of error. The value ENXIO must be returned if the device is not present. Other error values may mean other conditions. Zero or negative values mean success. Most of the drivers return zero as success. The negative return values are used when a PnP device supports multiple interfaces. For example, an older compatibility interface and a newer advanced interface which are supported by different drivers. Then both drivers would detect the device. The driver which returns a higher value in the probe routine takes precedence (in other words, the driver returning 0 has highest precedence, one returning -1 is next, one returning -2 is after it and so on). In result the devices which support only the old interface will be handled by the old driver (which should return -1 from the probe routine) while the devices supporting the new interface as well will be handled by the new driver (which should return 0 from the probe routine). The device descriptor struct xxx_softc is allocated by the system before calling the probe routine. If the probe routine returns an error the descriptor will be automatically deallocated by the system. So if a probing error occurs the driver must make sure that all the resources it used during probe are deallocated and that nothing keeps the descriptor from being safely deallocated. If the probe completes successfully the descriptor will be preserved by the system and later passed to the routine xxx_isa_attach(). If a driver returns a negative value it can not be sure that it will have the highest priority and its attach routine will be called. So in this case it also must release all the resources before returning and if necessary allocate them again in the attach routine. When xxx_isa_probe() returns 0 releasing the resources before returning is also a good idea, a well-behaved driver should do so. But in case if there is some problem with releasing the resources the driver is allowed to keep resources between returning 0 from the probe routine and execution of the attach routine. A typical probe routine starts with getting the device descriptor and unit: struct xxx_softc *sc = device_get_softc(dev); int unit = device_get_unit(dev); int pnperror; int error = 0; sc->dev = dev; /* link it back */ sc->unit = unit; Then check for the PnP devices. The check is carried out by a table containing the list of PnP IDs supported by this driver and human-readable descriptions of the device models corresponding to these IDs. pnperror=ISA_PNP_PROBE(device_get_parent(dev), dev, xxx_pnp_ids); if(pnperror == ENXIO) return ENXIO; The logic of ISA_PNP_PROBE is the following: If this card (device unit) was not detected as PnP then ENOENT will be returned. If it was detected as PnP but its detected ID does not match any of the IDs in the table then ENXIO is returned. Finally, if it has PnP support and it matches on of the IDs in the table, 0 is returned and the appropriate description from the table is set by device_set_desc(). If a driver supports only PnP devices then the condition would look like: if(pnperror != 0) return pnperror; No special treatment is required for the drivers which do not support PnP because they pass an empty PnP ID table and will always get ENXIO if called on a PnP card. The probe routine normally needs at least some minimal set of resources, such as I/O port number to find the card and probe it. Depending on the hardware the driver may be able to discover the other necessary resources automatically. The PnP devices have all the resources pre-set by the PnP subsystem, so the driver does not need to discover them by itself. Typically the minimal information required to get access to the device is the I/O port number. Then some devices allow to get the rest of information from the device configuration registers (though not all devices do that). So first we try to get the port start value: sc->port0 = bus_get_resource_start(dev, SYS_RES_IOPORT, 0 /*rid*/); if(sc->port0 == 0) return ENXIO; The base port address is saved in the structure softc for future use. If it will be used very often then calling the resource function each time would be prohibitively slow. If we do not get a port we just return an error. Some device drivers can instead be clever and try to probe all the possible ports, like this: /* table of all possible base I/O port addresses for this device */ static struct xxx_allports { u_short port; /* port address */ short used; /* flag: if this port is already used by some unit */ } xxx_allports = { { 0x300, 0 }, { 0x320, 0 }, { 0x340, 0 }, { 0, 0 } /* end of table */ }; ... int port, i; ... port = bus_get_resource_start(dev, SYS_RES_IOPORT, 0 /*rid*/); if(port !=0 ) { for(i=0; xxx_allports[i].port!=0; i++) { if(xxx_allports[i].used || xxx_allports[i].port != port) continue; /* found it */ xxx_allports[i].used = 1; /* do probe on a known port */ return xxx_really_probe(dev, port); } return ENXIO; /* port is unknown or already used */ } /* we get here only if we need to guess the port */ for(i=0; xxx_allports[i].port!=0; i++) { if(xxx_allports[i].used) continue; /* mark as used - even if we find nothing at this port * at least we won't probe it in future */ xxx_allports[i].used = 1; error = xxx_really_probe(dev, xxx_allports[i].port); if(error == 0) /* found a device at that port */ return 0; } /* probed all possible addresses, none worked */ return ENXIO; Of course, normally the driver's identify() routine should be used for such things. But there may be one valid reason why it may be better to be done in probe(): if this probe would drive some other sensitive device crazy. The probe routines are ordered with consideration of the "sensitive" flag: the sensitive devices get probed first and the rest of devices later. But the identify() routines are called before any probes, so they show no respect to the sensitive devices and may upset them. Now, after we got the starting port we need to set the port count (except for PnP devices) because the kernel does not have this information in the configuration file. if(pnperror /* only for non-PnP devices */ && bus_set_resource(dev, SYS_RES_IOPORT, 0, sc->port0, XXX_PORT_COUNT)<0) return ENXIO; Finally allocate and activate a piece of port address space (special values of start and end mean "use those we set by bus_set_resource()"): sc->port0_rid = 0; sc->port0_r = bus_alloc_resource(dev, SYS_RES_IOPORT, &sc->port0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->port0_r == NULL) return ENXIO; Now having access to the port-mapped registers we can poke the device in some way and check if it reacts like it is expected to. If it does not then there is probably some other device or no device at all at this address. Normally drivers do not set up the interrupt handlers until the attach routine. Instead they do probes in the polling mode using the DELAY() function for timeout. The probe routine must never hang forever, all the waits for the device must be done with timeouts. If the device does not respond within the time it is probably broken or misconfigured and the driver must return error. When determining the timeout interval give the device some extra time to be on the safe side: although DELAY() is supposed to delay for the same amount of time on any machine it has some margin of error, depending on the exact CPU. If the probe routine really wants to check that the interrupts really work it may configure and probe the interrupts too. But that is not recommended. /* implemented in some very device-specific way */ if(error = xxx_probe_ports(sc)) goto bad; /* will deallocate the resources before returning */ The function xxx_probe_ports() may also set the device description depending on the exact model of device it discovers. But if there is only one supported device model this can be as well done in a hardcoded way. Of course, for the PnP devices the PnP support sets the description from the table automatically. if(pnperror) device_set_desc(dev, "Our device model 1234"); Then the probe routine should either discover the ranges of all the resources by reading the device configuration registers or make sure that they were set explicitly by the user. We will consider it with an example of on-board memory. The probe routine should be as non-intrusive as possible, so allocation and check of functionality of the rest of resources (besides the ports) would be better left to the attach routine. The memory address may be specified in the kernel configuration file or on some devices it may be pre-configured in non-volatile configuration registers. If both sources are available and different, which one should be used? Probably if the user bothered to set the address explicitly in the kernel configuration file they know what they are doing and this one should take precedence. An example of implementation could be: /* try to find out the config address first */ sc->mem0_p = bus_get_resource_start(dev, SYS_RES_MEMORY, 0 /*rid*/); if(sc->mem0_p == 0) { /* nope, not specified by user */ sc->mem0_p = xxx_read_mem0_from_device_config(sc); if(sc->mem0_p == 0) /* can't get it from device config registers either */ goto bad; } else { if(xxx_set_mem0_address_on_device(sc) < 0) goto bad; /* device does not support that address */ } /* just like the port, set the memory size, * for some devices the memory size would not be constant * but should be read from the device configuration registers instead * to accommodate different models of devices. Another option would * be to let the user set the memory size as "msize" configuration * resource which will be automatically handled by the ISA bus. */ if(pnperror) { /* only for non-PnP devices */ sc->mem0_size = bus_get_resource_count(dev, SYS_RES_MEMORY, 0 /*rid*/); if(sc->mem0_size == 0) /* not specified by user */ sc->mem0_size = xxx_read_mem0_size_from_device_config(sc); if(sc->mem0_size == 0) { /* suppose this is a very old model of device without * auto-configuration features and the user gave no preference, * so assume the minimalistic case * (of course, the real value will vary with the driver) */ sc->mem0_size = 8*1024; } if(xxx_set_mem0_size_on_device(sc) < 0) goto bad; /* device does not support that size */ if(bus_set_resource(dev, SYS_RES_MEMORY, /*rid*/0, sc->mem0_p, sc->mem0_size)<0) goto bad; } else { sc->mem0_size = bus_get_resource_count(dev, SYS_RES_MEMORY, 0 /*rid*/); } Resources for IRQ and DRQ are easy to check by analogy. If all went well then release all the resources and return success. xxx_free_resources(sc); return 0; Finally, handle the troublesome situations. All the resources should be deallocated before returning. We make use of the fact that before the structure softc is passed to us it gets zeroed out, so we can find out if some resource was allocated: then its descriptor is non-zero. bad: xxx_free_resources(sc); if(error) return error; else /* exact error is unknown */ return ENXIO; That would be all for the probe routine. Freeing of resources is done from multiple places, so it is moved to a function which may look like: static void xxx_free_resources(sc) struct xxx_softc *sc; { /* check every resource and free if not zero */ /* interrupt handler */ if(sc->intr_r) { bus_teardown_intr(sc->dev, sc->intr_r, sc->intr_cookie); bus_release_resource(sc->dev, SYS_RES_IRQ, sc->intr_rid, sc->intr_r); sc->intr_r = 0; } /* all kinds of memory maps we could have allocated */ if(sc->data_p) { bus_dmamap_unload(sc->data_tag, sc->data_map); sc->data_p = 0; } if(sc->data) { /* sc->data_map may be legitimately equal to 0 */ /* the map will also be freed */ bus_dmamem_free(sc->data_tag, sc->data, sc->data_map); sc->data = 0; } if(sc->data_tag) { bus_dma_tag_destroy(sc->data_tag); sc->data_tag = 0; } ... free other maps and tags if we have them ... if(sc->parent_tag) { bus_dma_tag_destroy(sc->parent_tag); sc->parent_tag = 0; } /* release all the bus resources */ if(sc->mem0_r) { bus_release_resource(sc->dev, SYS_RES_MEMORY, sc->mem0_rid, sc->mem0_r); sc->mem0_r = 0; } ... if(sc->port0_r) { bus_release_resource(sc->dev, SYS_RES_IOPORT, sc->port0_rid, sc->port0_r); sc->port0_r = 0; } } xxx_isa_attach The attach routine actually connects the driver to the system if the probe routine returned success and the system had chosen to attach that driver. If the probe routine returned 0 then the attach routine may expect to receive the device structure softc intact, as it was set by the probe routine. Also if the probe routine returns 0 it may expect that the attach routine for this device shall be called at some point in the future. If the probe routine returns a negative value then the driver may make none of these assumptions. The attach routine returns 0 if it completed successfully or error code otherwise. The attach routine starts just like the probe routine, with getting some frequently used data into more accessible variables. struct xxx_softc *sc = device_get_softc(dev); int unit = device_get_unit(dev); int error = 0; Then allocate and activate all the necessary resources. Because normally the port range will be released before returning from probe, it has to be allocated again. We expect that the probe routine had properly set all the resource ranges, as well as saved them in the structure softc. If the probe routine had left some resource allocated then it does not need to be allocated again (which would be considered an error). sc->port0_rid = 0; sc->port0_r = bus_alloc_resource(dev, SYS_RES_IOPORT, &sc->port0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->port0_r == NULL) return ENXIO; /* on-board memory */ sc->mem0_rid = 0; sc->mem0_r = bus_alloc_resource(dev, SYS_RES_MEMORY, &sc->mem0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->mem0_r == NULL) goto bad; /* get its virtual address */ sc->mem0_v = rman_get_virtual(sc->mem0_r); The DMA request channel (DRQ) is allocated likewise. To initialize it use functions of the isa_dma*() family. For example: isa_dmacascade(sc->drq0); The interrupt request line (IRQ) is a bit special. Besides allocation the driver's interrupt handler should be associated with it. Historically in the old ISA drivers the argument passed by the system to the interrupt handler was the device unit number. But in modern drivers the convention suggests passing the pointer to structure softc. The important reason is that when the structures softc are allocated dynamically then getting the unit number from softc is easy while getting softc from unit number is difficult. Also this convention makes the drivers for different buses look more uniform and allows them to share the code: each bus gets its own probe, attach, detach and other bus-specific routines while the bulk of the driver code may be shared among them. sc->intr_rid = 0; sc->intr_r = bus_alloc_resource(dev, SYS_RES_MEMORY, &sc->intr_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->intr_r == NULL) goto bad; /* * XXX_INTR_TYPE is supposed to be defined depending on the type of * the driver, for example as INTR_TYPE_CAM for a CAM driver */ error = bus_setup_intr(dev, sc->intr_r, XXX_INTR_TYPE, (driver_intr_t *) xxx_intr, (void *) sc, &sc->intr_cookie); if(error) goto bad; If the device needs to make DMA to the main memory then this memory should be allocated like described before: error=bus_dma_tag_create(NULL, /*alignment*/ 4, /*boundary*/ 0, /*lowaddr*/ BUS_SPACE_MAXADDR_24BIT, /*highaddr*/ BUS_SPACE_MAXADDR, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ BUS_SPACE_MAXSIZE_24BIT, /*nsegments*/ BUS_SPACE_UNRESTRICTED, /*maxsegsz*/ BUS_SPACE_MAXSIZE_24BIT, /*flags*/ 0, &sc->parent_tag); if(error) goto bad; /* many things get inherited from the parent tag * sc->data is supposed to point to the structure with the shared data, * for example for a ring buffer it could be: * struct { * u_short rd_pos; * u_short wr_pos; * char bf[XXX_RING_BUFFER_SIZE] * } *data; */ error=bus_dma_tag_create(sc->parent_tag, 1, 0, BUS_SPACE_MAXADDR, 0, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ sizeof(* sc->data), /*nsegments*/ 1, /*maxsegsz*/ sizeof(* sc->data), /*flags*/ 0, &sc->data_tag); if(error) goto bad; error = bus_dmamem_alloc(sc->data_tag, &sc->data, /* flags*/ 0, &sc->data_map); if(error) goto bad; /* xxx_alloc_callback() just saves the physical address at * the pointer passed as its argument, in this case &sc->data_p. * See details in the section on bus memory mapping. * It can be implemented like: * * static void * xxx_alloc_callback(void *arg, bus_dma_segment_t *seg, * int nseg, int error) * { * *(bus_addr_t *)arg = seg[0].ds_addr; * } */ bus_dmamap_load(sc->data_tag, sc->data_map, (void *)sc->data, sizeof (* sc->data), xxx_alloc_callback, (void *) &sc->data_p, /*flags*/0); After all the necessary resources are allocated the device should be initialized. The initialization may include testing that all the expected features are functional. if(xxx_initialize(sc) < 0) goto bad; The bus subsystem will automatically print on the console the device description set by probe. But if the driver wants to print some extra information about the device it may do so, for example: device_printf(dev, "has on-card FIFO buffer of %d bytes\n", sc->fifosize); If the initialization routine experiences any problems then printing messages about them before returning error is also recommended. The final step of the attach routine is attaching the device to its functional subsystem in the kernel. The exact way to do it depends on the type of the driver: a character device, a block device, a network device, a CAM SCSI bus device and so on. If all went well then return success. error = xxx_attach_subsystem(sc); if(error) goto bad; return 0; Finally, handle the troublesome situations. All the resources should be deallocated before returning an error. We make use of the fact that before the structure softc is passed to us it gets zeroed out, so we can find out if some resource was allocated: then its descriptor is non-zero. bad: xxx_free_resources(sc); if(error) return error; else /* exact error is unknown */ return ENXIO; That would be all for the attach routine. xxx_isa_detach If this function is present in the driver and the driver is compiled as a loadable module then the driver gets the ability to be unloaded. This is an important feature if the hardware supports hot plug. But the ISA bus does not support hot plug, so this feature is not particularly important for the ISA devices. The ability to unload a driver may be useful when debugging it, but in many cases installation of the new version of the driver would be required only after the old version somehow wedges the system and reboot will be needed anyway, so the efforts spent on writing the detach routine may not be worth it. Another argument is that unloading would allow upgrading the drivers on a production machine seems to be mostly theoretical. Installing a new version of a driver is a dangerous operation which should never be performed on a production machine (and which is not permitted when the system is running in secure mode). Still the detach routine may be provided for the sake of completeness. The detach routine returns 0 if the driver was successfully detached or the error code otherwise. The logic of detach is a mirror of the attach. The first thing to do is to detach the driver from its kernel subsystem. If the device is currently open then the driver has two choices: refuse to be detached or forcibly close and proceed with detach. The choice used depends on the ability of the particular kernel subsystem to do a forced close and on the preferences of the driver's author. Generally the forced close seems to be the preferred alternative. struct xxx_softc *sc = device_get_softc(dev); int error; error = xxx_detach_subsystem(sc); if(error) return error; Next the driver may want to reset the hardware to some consistent state. That includes stopping any ongoing transfers, disabling the DMA channels and interrupts to avoid memory corruption by the device. For most of the drivers this is exactly what the shutdown routine does, so if it is included in the driver we can as well just call it. xxx_isa_shutdown(dev); And finally release all the resources and return success. xxx_free_resources(sc); return 0; xxx_isa_shutdown This routine is called when the system is about to be shut down. It is expected to bring the hardware to some consistent state. For most of the ISA devices no special action is required, so the function is not really necessary because the device will be re-initialized on reboot anyway. But some devices have to be shut down with a special procedure, to make sure that they will be properly detected after soft reboot (this is especially true for many devices with proprietary identification protocols). In any case disabling DMA and interrupts in the device registers and stopping any ongoing transfers is a good idea. The exact action depends on the hardware, so we do not consider it here in any details. xxx_intr The interrupt handler is called when an interrupt is received which may be from this particular device. The ISA bus does not support interrupt sharing (except some special cases) so in practice if the interrupt handler is called then the interrupt almost for sure came from its device. Still the interrupt handler must poll the device registers and make sure that the interrupt was generated by its device. If not it should just return. The old convention for the ISA drivers was getting the device unit number as an argument. It is obsolete, and the new drivers receive whatever argument was specified for them in the attach routine when calling bus_setup_intr(). By the new convention it should be the pointer to the structure softc. So the interrupt handler commonly starts as: static void xxx_intr(struct xxx_softc *sc) { It runs at the interrupt priority level specified by the interrupt type parameter of bus_setup_intr(). That means that all the other interrupts of the same type as well as all the software interrupts are disabled. To avoid races it is commonly written as a loop: while(xxx_interrupt_pending(sc)) { xxx_process_interrupt(sc); xxx_acknowledge_interrupt(sc); } The interrupt handler has to acknowledge interrupt to the device only but not to the interrupt controller, the system takes care of the latter. diff --git a/en_US.ISO8859-1/books/arch-handbook/jail/chapter.sgml b/en_US.ISO8859-1/books/arch-handbook/jail/chapter.sgml index df99e3e2a4..47f6523a78 100644 --- a/en_US.ISO8859-1/books/arch-handbook/jail/chapter.sgml +++ b/en_US.ISO8859-1/books/arch-handbook/jail/chapter.sgml @@ -1,611 +1,611 @@ Evan Sarmiento
evms@cs.bu.edu
 2001 Evan Sarmiento The Jail Subsystem On most UNIX systems, root has omnipotent power. This promotes insecurity. If an attacker were to gain root on a system, he would have every function at his fingertips. In FreeBSD there are sysctls which dilute the power of root, in order to minimize the damage caused by an attacker. Specifically, one of these functions is called secure levels. Similarly, another function which is present from FreeBSD 4.0 and onward, is a utility called &man.jail.8;. Jail chroots an environment and sets certain restrictions on processes which are forked from within. For example, a jailed process cannot affect processes outside of the jail, utilize certain system calls, or inflict any damage on the main computer. Jail is becoming the new security model. People are running potentially vulnerable servers such as Apache, BIND, and sendmail within jails, so that if an attacker gains root within the Jail, it is only an annoyance, and not a devastation. This article focuses on the internals (source code) of Jail and Jail NG. It will also suggest improvements upon the jail code base which are already being worked on. If you are looking for a how-to on setting up a Jail, I suggest you look at my other article in Sys Admin Magazine, May 2001, entitled "Securing FreeBSD using Jail." Architecture Jail consists of two realms: the user-space program, jail, and the code implemented within the kernel: the jail() system call and associated restrictions. I will be discussing the user-space program and then how jail is implemented within the kernel. Userland code The source for the user-land jail is located in - /usr/src/usr.sbin/jail , consisting of + /usr/src/usr.sbin/jail, consisting of one file, jail.c. The program takes these arguments: the path of the jail, hostname, ip address, and the command to be executed. Data Structures In jail.c, the first thing I would note is the declaration of an important structure struct jail j; which was included from /usr/include/sys/jail.h . The definition of the jail structure is: /usr/include/sys/jail.h: struct jail { u_int32_t version; char *path; char *hostname; u_int32_t ip_number; }; As you can see, there is an entry for each of the arguments passed to the jail program, and indeed, they are set during it's execution. /usr/src/usr.sbin/jail.c j.version = 0; j.path = argv[1]; j.hostname = argv[2]; Networking One of the arguments passed to the Jail program is an IP address with which the jail can be accessed over the network. Jail translates the ip address given into network byte order and then stores it in j (the jail structure). /usr/src/usr.sbin/jail/jail.c: struct in.addr in; ... i = inet.aton(argv[3], ); ... j.ip.number = ntohl(in.s.addr); The inet_aton3 function "interprets the specified character string as an Internet address, placing the address into the structure provided." The ip number node in the jail structure is set only when the ip address placed onto the in structure by inet aton is translated into network byte order by ntohl(). Jailing The Process Finally, the userland program jails the process, and executes the command specified. Jail now becomes an imprisoned process itself and forks a child process which then executes the command given using &man.execv.3; /usr/src/sys/usr.sbin/jail/jail.c i = jail(); ... i = execv(argv[4], argv + 4); As you can see, the jail function is being called, and its argument is the jail structure which has been filled with the arguments given to the program. Finally, the program you specify is executed. I will now discuss how Jail is implemented within the kernel. Kernel Space We will now be looking at the file /usr/src/sys/kern/kern_jail.c. This is the file where the jail system call, appropriate sysctls, and networking functions are defined. sysctls In kern_jail.c, the following sysctls are defined: /usr/src/sys/kern/kern_jail.c: int jail_set_hostname_allowed = 1; SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW, _set_hostname_allowed, 0, "Processes in jail can set their hostnames"); int jail_socket_unixiproute_only = 1; SYSCTL_INT(_jail, OID_AUTO, socket_unixiproute_only, CTLFLAG_RW, _socket_unixiproute_only, 0, "Processes in jail are limited to creating UNIX/IPv4/route sockets only "); int jail_sysvipc_allowed = 0; SYSCTL_INT(_jail, OID_AUTO, sysvipc_allowed, CTLFLAG_RW, _sysvipc_allowed, 0, "Processes in jail can use System V IPC primitives"); Each of these sysctls can be accessed by the user through the sysctl program. Throughout the kernel, these specific sysctls are recognized by their name. For example, the name of the first sysctl is jail.set.hostname.allowed. &man.jail.2; system call Like all system calls, the &man.jail.2; system call takes two arguments, struct proc *p and struct jail_args *uap. p is a pointer to a proc structure which describes the calling process. In this context, uap is a pointer to a structure which specifies the arguments given to &man.jail.2; from the userland program jail.c. When I described the userland program before, you saw that the &man.jail.2; system call was given a jail structure as its own argument. /usr/src/sys/kern/kern_jail.c: int jail(p, uap) struct proc *p; struct jail_args /* { syscallarg(struct jail *) jail; } */ *uap; Therefore, uap->jail would access the jail structure which was passed to the system call. Next, the system call copies the jail structure into kernel space using the copyin() function. copyin() takes three arguments: the data which is to be copied into kernel space, uap->jail, where to store it, j and the size of the storage. The jail structure uap->jail is copied into kernel space and stored in another jail structure, j. /usr/src/sys/kern/kern_jail.c: error = copyin(uap->jail, , sizeof j); There is another important structure defined in jail.h. It is the prison structure (pr). The prison structure is used exclusively within kernel space. The &man.jail.2; system call copies everything from the jail structure onto the prison structure. Here is the definition of the prison structure. /usr/include/sys/jail.h: struct prison { int pr_ref; char pr_host[MAXHOSTNAMELEN]; u_int32_t pr_ip; void *pr_linux; }; The jail() system call then allocates memory for a pointer to a prison structure and copies data between the two structures. /usr/src/sys/kern/kern_jail.c: MALLOC(pr, struct prison *, sizeof *pr , M_PRISON, M_WAITOK); bzero((caddr_t)pr, sizeof *pr); error = copyinstr(j.hostname, pr_host]]>, sizeof pr->pr_host, 0); if (error) goto bail; Finally, the jail system call chroots the path specified. The chroot function is given two arguments. The first is p, which represents the calling process, the second is a pointer to the structure chroot args. The structure chroot args contains the path which is to be chrooted. As you can see, the path specified in the jail structure is copied to the chroot args structure and used. /usr/src/sys/kern/kern_jail.c: ca.path = j.path; error = chroot(p, ); These next three lines in the source are very important, as they specify how the kernel recognizes a process as jailed. Each process on a Unix system is described by its own proc structure. You can see the whole proc structure in /usr/include/sys/proc.h. For example, the p argument in any system call is actually a pointer to that process' proc structure, as stated before. The proc structure contains nodes which can describe the owner's identity (p_cred), the process resource limits (p_limit), and so on. In the definition of the process structure, there is a pointer to a prison structure. (p_prison). /usr/include/sys/proc.h: struct proc { ... struct prison *p_prison; ... }; In kern_jail.c, the function then copies the pr structure, which is filled with all the information from the original jail structure, over to the p->p_prison structure. It then does a bitwise OR of p->p_flag with the constant P_JAILED, meaning that the calling process is now recognized as jailed. The parent process of each process, forked within the jail, is the program jail itself, as it calls the &man.jail.2; system call. When the program is executed through execve, it inherits the properties of its parents proc structure, therefore it has the p->p_flag set, and the p->p_prison structure is filled. /usr/src/sys/kern/kern_jail.c p->p.prison = pr; p->p.flag --= P.JAILED; When a process is forked from a parent process, the &man.fork.2; system call deals differently with imprisoned processes. In the fork system call, there are two pointers to a proc structure p1 and p2. p1 points to the parent's proc structure and p2 points to the child's unfilled proc structure. After copying all relevant data between the structures, &man.fork.2; checks if the structure p->p_prison is filled on p2. If it is, it increments the pr.ref by one, and sets the p_flag to one on the child process. /usr/src/sys/kern/kern_fork.c: if (p2->p_prison) { p2->p_prison->pr_ref++; p2->p_flag |= P_JAILED; } Restrictions Throughout the kernel there are access restrictions relating to jailed processes. Usually, these restrictions only check if the process is jailed, and if so, returns an error. For example: if (p->p_prison) return EPERM; SysV IPC System V IPC is based on messages. Processes can send each other these messages which tell them how to act. The functions which deal with messages are: msgsys, msgctl, msgget, msgsend and msgrcv. Earlier, I mentioned that there were certain sysctls you could turn on or off in order to affect the behavior of Jail. One of these sysctls was jail_sysvipc_allowed. On most systems, this sysctl is set to 0. If it were set to 1, it would defeat the whole purpose of having a jail; privleged users from within the jail would be able to affect processes outside of the environment. The difference between a message and a signal is that the message only consists of the signal number. /usr/src/sys/kern/sysv_msg.c: &man.msgget.3;: msgget returns (and possibly creates) a message descriptor that designates a message queue for use in other system calls. &man.msgctl.3;: Using this function, a process can query the status of a message descriptor. &man.msgsnd.3;: msgsnd sends a message to a process. &man.msgrcv.3;: a process receives messages using this function In each of these system calls, there is this conditional: /usr/src/sys/kern/sysv msg.c: if (!jail.sysvipc.allowed && p->p_prison != NULL) return (ENOSYS); Semaphore system calls allow processes to synchronize execution by doing a set of operations atomically on a set of semaphores. Basically semaphores provide another way for processes lock resources. However, process waiting on a semaphore, that is being used, will sleep until the resources are relinquished. The following semaphore system calls are blocked inside a jail: semsys, semget, semctl and semop. /usr/src/sys/kern/sysv_sem.c: &man.semctl.2;(id, num, cmd, arg): Semctl does the specified cmd on the semaphore queue indicated by id. &man.semget.2;(key, nsems, flag): Semget creates an array of semaphores, corresponding to key. Key and flag take on the same meaning as they do in msgget. &man.semop.2;(id, ops, num): Semop does the set of semaphore operations in the array of structures ops, to the set of semaphores identified by id. System V IPC allows for processes to share memory. Processes can communicate directly with each other by sharing parts of their virtual address space and then reading and writing data stored in the shared memory. These system calls are blocked within a jailed environment: shmdt, shmat, oshmctl, shmctl, shmget, and shmsys. /usr/src/sys/kern/sysv shm.c: &man.shmctl.2;(id, cmd, buf): shmctl does various control operations on the shared memory region identified by id. &man.shmget.2;(key, size, flag): shmget accesses or creates a shared memory region of size bytes. &man.shmat.2;(id, addr, flag): shmat attaches a shared memory region identified by id to the address space of a process. &man.shmdt.2;(addr): shmdt detaches the shared memory region previously attached at addr. Sockets Jail treats the &man.socket.2; system call and related lower-level socket functions in a special manner. In order to determine whether a certain socket is allowed to be created, it first checks to see if the sysctl jail.socket.unixiproute.only is set. If set, sockets are only allowed to be created if the family specified is either PF_LOCAL, PF_INET or PF_ROUTE. Otherwise, it returns an error. /usr/src/sys/kern/uipc_socket.c: int socreate(dom, aso, type, proto, p) ... register struct protosw *prp; ... { if (p->p_prison && jail_socket_unixiproute_only && prp->pr_domain->dom_family != PR_LOCAL && prp->pr_domain->dom_family != PF_INET && prp->pr_domain->dom_family != PF_ROUTE) return (EPROTONOSUPPORT); ... } Berkeley Packet Filter The Berkeley Packet Filter provides a raw interface to data link layers in a protocol independent fashion. The function bpfopen() opens an Ethernet device. There is a conditional which disallows any jailed processes from accessing this function. /usr/src/sys/net/bpf.c: static int bpfopen(dev, flags, fmt, p) ... { if (p->p_prison) return (EPERM); ... } Protocols There are certain protocols which are very common, such as TCP, UDP, IP and ICMP. IP and ICMP are on the same level: the network layer 2 . There are certain precautions which are taken in order to prevent a jailed process from binding a protocol to a certain port only if the nam parameter is set. nam is a pointer to a sockaddr structure, which describes the address on which to bind the service. A more exact definition is that sockaddr "may be used as a template for reffering to the identifying tag and length of each address"[2] . In the function in pcbbind, sin is a pointer to a sockaddr.in structure, which contains the port, address, length and domain family of the socket which is to be bound. Basically, this disallows any processes from jail to be able to specify the domain family. /usr/src/sys/kern/netinet/in_pcb.c: int in.pcbbind(int, nam, p) ... struct sockaddr *nam; struct proc *p; { ... struct sockaddr.in *sin; ... if (nam) { sin = (struct sockaddr.in *)nam; ... if (sin->sin_addr.s_addr != INADDR_ANY) if (prison.ip(p, 0, ->sin.addr.s_addr)) return (EINVAL); .... } ... } You might be wondering what function prison_ip() does. prison.ip is given three arguments, the current process (represented by p), any flags, and an ip address. It returns 1 if the ip address belongs to a jail or 0 if it does not. As you can see from the code, if it is indeed an ip address belonging to a jail, the protcol is not allowed to bind to a certain port. /usr/src/sys/kern/kern_jail.c: int prison_ip(struct proc *p, int flag, u_int32_t *ip) { u_int32_t tmp; if (!p->p_prison) return (0); if (flag) tmp = *ip; else tmp = ntohl (*ip); if (tmp == INADDR_ANY) { if (flag) *ip = p->p_prison->pr_ip; else *ip = htonl(p->p_prison->pr_ip); return (0); } if (p->p_prison->pr_ip != tmp) return (1); return (0); } Jailed users are not allowed to bind services to an ip which does not belong to the jail. The restriction is also written within the function in_pcbbind: /usr/src/sys/net inet/in_pcb.c if (nam) { ... lport = sin->sin.port; ... if (lport) { ... if (p && p->p_prison) prison = 1; if (prison && prison_ip(p, 0, ->sin_addr.s_addr)) return (EADDRNOTAVAIL); Filesystem Even root users within the jail are not allowed to set any file flags, such as immutable, append, and no unlink flags, if the securelevel is greater than 0. /usr/src/sys/ufs/ufs/ufs_vnops.c: int ufs.setattr(ap) ... { if ((cred->cr.uid == 0) && (p->prison == NULL)) { if ((ip->i_flags & (SF_NOUNLINK | SF_IMMUTABLE | SF_APPEND)) && securelevel > 0) return (EPERM); } Jail NG Jail NG is a "from-scratch re-implementation of Jail" by Robert Watson, a FreeBSD committer. Some of the new features include the ability to add processes to a jail, an improved management tool, and per-jail sysctls. For example, you could have sysvipc_permitted set on one jail while another jail may be allowed to use System V IPC. You can download the kernel patches and utilities for Jail NG from his website at: . diff --git a/en_US.ISO8859-1/books/arch-handbook/pci/chapter.sgml b/en_US.ISO8859-1/books/arch-handbook/pci/chapter.sgml index d1085cce39..15146dc9e9 100644 --- a/en_US.ISO8859-1/books/arch-handbook/pci/chapter.sgml +++ b/en_US.ISO8859-1/books/arch-handbook/pci/chapter.sgml @@ -1,370 +1,369 @@ PCI Devices This chapter will talk about the FreeBSD mechanisms for writing a device driver for a device on a PCI bus. Probe and Attach Information here about how the PCI bus code iterates through the unattached devices and see if a newly loaded kld will attach to any of them. /* * Simple KLD to play with the PCI functions. * * Murray Stokely */ #define MIN(a,b) (((a) < (b)) ? (a) : (b)) #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ #include <sys/conf.h> /* cdevsw struct */ #include <sys/uio.h> /* uio struct */ #include <sys/malloc.h> #include <sys/bus.h> /* structs, prototypes for pci bus stuff */ #include <pci/pcivar.h> /* For get_pci macros! */ /* Function prototypes */ d_open_t mypci_open; d_close_t mypci_close; d_read_t mypci_read; d_write_t mypci_write; /* Character device entry points */ static struct cdevsw mypci_cdevsw = { mypci_open, mypci_close, mypci_read, mypci_write, noioctl, nopoll, nommap, nostrategy, "mypci", 36, /* reserved for lkms - /usr/src/sys/conf/majors */ nodump, nopsize, D_TTY, -1 }; /* vars */ static dev_t sdev; /* We're more interested in probe/attach than with open/close/read/write at this point */ int mypci_open(dev_t dev, int oflags, int devtype, struct proc *p) { int err = 0; uprintf("Opened device \"mypci\" successfully.\n"); return(err); } int mypci_close(dev_t dev, int fflag, int devtype, struct proc *p) { int err=0; uprintf("Closing device \"mypci.\"\n"); return(err); } int mypci_read(dev_t dev, struct uio *uio, int ioflag) { int err = 0; uprintf("mypci read!\n"); return err; } int mypci_write(dev_t dev, struct uio *uio, int ioflag) { int err = 0; uprintf("mypci write!\n"); return(err); } /* PCI Support Functions */ /* * Return identification string if this is device is ours. */ static int mypci_probe(device_t dev) { uprintf("MyPCI Probe\n" "Vendor ID : 0x%x\n" "Device ID : 0x%x\n",pci_get_vendor(dev),pci_get_device(dev)); if (pci_get_vendor(dev) == 0x11c1) { uprintf("We've got the Winmodem, probe successful!\n"); return 0; } return ENXIO; } /* Attach function is only called if the probe is successful */ static int mypci_attach(device_t dev) { uprintf("MyPCI Attach for : deviceID : 0x%x\n",pci_get_vendor(dev)); sdev = make_dev(&mypci_cdevsw, 0, UID_ROOT, GID_WHEEL, 0600, "mypci"); uprintf("Mypci device loaded.\n"); return ENXIO; } /* Detach device. */ static int mypci_detach(device_t dev) { uprintf("Mypci detach!\n"); return 0; } /* Called during system shutdown after sync. */ static int mypci_shutdown(device_t dev) { uprintf("Mypci shutdown!\n"); return 0; } /* * Device suspend routine. */ static int mypci_suspend(device_t dev) { uprintf("Mypci suspend!\n"); return 0; } /* * Device resume routine. */ static int mypci_resume(device_t dev) { uprintf("Mypci resume!\n"); return 0; } static device_method_t mypci_methods[] = { /* Device interface */ DEVMETHOD(device_probe, mypci_probe), DEVMETHOD(device_attach, mypci_attach), DEVMETHOD(device_detach, mypci_detach), DEVMETHOD(device_shutdown, mypci_shutdown), DEVMETHOD(device_suspend, mypci_suspend), DEVMETHOD(device_resume, mypci_resume), { 0, 0 } }; static driver_t mypci_driver = { "mypci", mypci_methods, 0, /* sizeof(struct mypci_softc), */ }; static devclass_t mypci_devclass; DRIVER_MODULE(mypci, pci, mypci_driver, mypci_devclass, 0, 0); Additional Resources PCI Special Interest Group PCI System Architecture, Fourth Edition by Tom Shanley, et al. Bus Resources FreeBSD provides an object-oriented mechanism for requesting resources from a parent bus. Almost all devices will be a child member of some sort of bus (PCI, ISA, USB, SCSI, etc) and these devices need to acquire resources from their parent bus (such as memory segments, interrupt lines, or DMA channels). Base Address Registers To do anything particularly useful with a PCI device you will need to obtain the Base Address Registers (BARs) from the PCI Configuration space. The PCI-specific details of obtaining the BAR is abstracted in the bus_alloc_resource() function. For example, a typical driver might have something similar - to this in the attach() function. : + to this in the attach() function: sc->bar0id = 0x10; sc->bar0res = bus_alloc_resource(dev, SYS_RES_MEMORY, &(sc->bar0id), 0, ~0, 1, RF_ACTIVE); if (sc->bar0res == NULL) { uprintf("Memory allocation of PCI base register 0 failed!\n"); error = ENXIO; goto fail1; } sc->bar1id = 0x14; sc->bar1res = bus_alloc_resource(dev, SYS_RES_MEMORY, &(sc->bar1id), 0, ~0, 1, RF_ACTIVE); if (sc->bar1res == NULL) { uprintf("Memory allocation of PCI base register 1 failed!\n"); error = ENXIO; goto fail2; } sc->bar0_bt = rman_get_bustag(sc->bar0res); sc->bar0_bh = rman_get_bushandle(sc->bar0res); sc->bar1_bt = rman_get_bustag(sc->bar1res); sc->bar1_bh = rman_get_bushandle(sc->bar1res); Handles for each base address register are kept in the softc structure so that they can be used to write to the device later. These handles can then be used to read or write from the device registers with the bus_space_* functions. For example, a driver might contain a shorthand - function to read from a board specific register like this : - + function to read from a board specific register like this: uint16_t board_read(struct ni_softc *sc, uint16_t address) { return bus_space_read_2(sc->bar1_bt, sc->bar1_bh, address); } - Similarly, one could write to the registers with : + Similarly, one could write to the registers with: void board_write(struct ni_softc *sc, uint16_t address, uint16_t value) { bus_space_write_2(sc->bar1_bt, sc->bar1_bh, address, value); } These functions exist in 8bit, 16bit, and 32bit versions and you should use bus_space_{read|write}_{1|2|4} accordingly. Interrupts Interrupts are allocated from the object-oriented bus code in a way similar to the memory resources. First an IRQ resource must be allocated from the parent bus, and then the interrupt handler must be setup to deal with this IRQ. Again, a sample from a device attach() function says more than words. /* Get the IRQ resource */ sc->irqid = 0x0; sc->irqres = bus_alloc_resource(dev, SYS_RES_IRQ, &(sc->irqid), 0, ~0, 1, RF_SHAREABLE | RF_ACTIVE); if (sc->irqres == NULL) { uprintf("IRQ allocation failed!\n"); error = ENXIO; goto fail3; } /* Now we should setup the interrupt handler */ error = bus_setup_intr(dev, sc->irqres, INTR_TYPE_MISC, my_handler, sc, &(sc->handler)); if (error) { printf("Couldn't set up irq\n"); goto fail4; } sc->irq_bt = rman_get_bustag(sc->irqres); sc->irq_bh = rman_get_bushandle(sc->irqres); DMA On the PC, peripherals that want to do bus-mastering DMA must deal with physical addresses. This is a problem since FreeBSD uses virtual memory and deals almost exclusively with virtual addresses. Fortunately, there is a function, vtophys() to help. #include <vm/vm.h> #include <vm/pmap.h> #define vtophys(virtual_address) (...) The solution is a bit different on the alpha however, and what we really want is a function called vtobus(). #if defined(__alpha__) #define vtobus(va) alpha_XXX_dmamap((vm_offset_t)va) #else #define vtobus(va) vtophys(va) #endif Deallocating Resources It is very important to deallocate all of the resources that were allocated during attach(). Care must be taken to deallocate the correct stuff even on a failure condition so that the system will remain usable while your driver dies. diff --git a/en_US.ISO8859-1/books/arch-handbook/scsi/chapter.sgml b/en_US.ISO8859-1/books/arch-handbook/scsi/chapter.sgml index a3881b407e..4e9c6e5332 100644 --- a/en_US.ISO8859-1/books/arch-handbook/scsi/chapter.sgml +++ b/en_US.ISO8859-1/books/arch-handbook/scsi/chapter.sgml @@ -1,1983 +1,1983 @@ Common Access Method SCSI Controllers This chapter was written by &a.babkin; Modifications for the handbook made by &a.murray;. Synopsis This document assumes that the reader has a general understanding of device drivers in FreeBSD and of the SCSI protocol. Much of the information in this document was - extracted from the drivers : + extracted from the drivers: ncr (/sys/pci/ncr.c) by Wolfgang Stanglmeier and Stefan Esser sym (/sys/pci/sym.c) by Gerard Roudier aic7xxx (/sys/dev/aic7xxx/aic7xxx.c) by Justin T. Gibbs and from the CAM code itself (by Justing T. Gibbs, see /sys/cam/*). When some solution looked the most logical and was essentially verbatim extracted from the code by Justin Gibbs, I marked it as "recommended". The document is illustrated with examples in pseudo-code. Although sometimes the examples have many details and look like real code, it is still pseudo-code. It was written to demonstrate the concepts in an understandable way. For a real driver other approaches may be more modular and efficient. It also abstracts from the hardware details, as well as issues that would cloud the demonstrated concepts or that are supposed to be described in the other chapters of the developers handbook. Such details are commonly shown as calls to functions with descriptive names, comments or pseudo-statements. Fortunately real life full-size examples with all the details can be found in the real drivers. General architecture CAM stands for Common Access Method. It is a generic way to address the I/O buses in a SCSI-like way. This allows a separation of the generic device drivers from the drivers controlling the I/O bus: for example the disk driver becomes able to control disks on both SCSI, IDE, and/or any other bus so the disk driver portion does not have to be rewritten (or copied and modified) for every new I/O bus. Thus the two most important active entities are: Peripheral Modules - a driver for peripheral devices (disk, tape, CDROM, etc.) SCSI Interface Modules (SIM) - a Host Bus Adapter drivers for connecting to an I/O bus such as SCSI or IDE. A peripheral driver receives requests from the OS, converts them to a sequence of SCSI commands and passes these SCSI commands to a SCSI Interface Module. The SCSI Interface Module is responsible for passing these commands to the actual hardware (or if the actual hardware is not SCSI but, for example, IDE then also converting the SCSI commands to the native commands of the hardware). Because we are interested in writing a SCSI adapter driver here, from this point on we will consider everything from the SIM standpoint. A typical SIM driver needs to include the following CAM-related header files: #include <cam/cam.h> #include <cam/cam_ccb.h> #include <cam/cam_sim.h> #include <cam/cam_xpt_sim.h> #include <cam/cam_debug.h> #include <cam/scsi/scsi_all.h> The first thing each SIM driver must do is register itself with the CAM subsystem. This is done during the driver's xxx_attach() function (here and further xxx_ is used to denote the unique driver name prefix). The xxx_attach() function itself is called by the system bus auto-configuration code which we do not describe here. This is achieved in multiple steps: first it is necessary to allocate the queue of requests associated with this SIM: struct cam_devq *devq; if(( devq = cam_simq_alloc(SIZE) )==NULL) { error; /* some code to handle the error */ } Here SIZE is the size of the queue to be allocated, maximal number of requests it could contain. It is the number of requests that the SIM driver can handle in parallel on one SCSI card. Commonly it can be calculated as: SIZE = NUMBER_OF_SUPPORTED_TARGETS * MAX_SIMULTANEOUS_COMMANDS_PER_TARGET Next we create a descriptor of our SIM: struct cam_sim *sim; if(( sim = cam_sim_alloc(action_func, poll_func, driver_name, softc, unit, max_dev_transactions, max_tagged_dev_transactions, devq) )==NULL) { cam_simq_free(devq); error; /* some code to handle the error */ } Note that if we are not able to create a SIM descriptor we free the devq also because we can do nothing else with it and we want to conserve memory. If a SCSI card has multiple SCSI buses on it then each bus requires its own cam_sim structure. An interesting question is what to do if a SCSI card has more than one SCSI bus, do we need one devq structure per card or per SCSI bus? The answer given in the comments to the CAM code is: either way, as the driver's author prefers. - The arguments are : + The arguments are: action_func - pointer to the driver's xxx_action function. static void xxx_action struct cam_sim *sim, union ccb *ccb poll_func - pointer to the driver's xxx_poll() static void xxx_poll struct cam_sim *sim driver_name - the name of the actual driver, such as "ncr" or "wds" softc - pointer to the driver's internal descriptor for this SCSI card. This pointer will be used by the driver in future to get private data. unit - the controller unit number, for example for controller "wds0" this number will be 0 max_dev_transactions - maximal number of simultaneous transactions per SCSI target in the non-tagged mode. This value will be almost universally equal to 1, with possible exceptions only for the non-SCSI cards. Also the drivers that hope to take advantage by preparing one transaction while another one is executed may set it to 2 but this does not seem to be worth the complexity. max_tagged_dev_transactions - the same thing, but in the tagged mode. Tags are the SCSI way to initiate multiple transactions on a device: each transaction is assigned a unique tag and the transaction is sent to the device. When the device completes some transaction it sends back the result together with the tag so that the SCSI adapter (and the driver) can tell which transaction was completed. This argument is also known as the maximal tag depth. It depends on the abilities of the SCSI adapter. Finally we register the SCSI buses associated with our SCSI adapter: if(xpt_bus_register(sim, bus_number) != CAM_SUCCESS) { cam_sim_free(sim, /*free_devq*/ TRUE); error; /* some code to handle the error */ } If there is one devq structure per SCSI bus (i.e. we consider a card with multiple buses as multiple cards with one bus each) then the bus number will always be 0, otherwise each bus on the SCSI card should be get a distinct number. Each bus needs its own separate structure cam_sim. After that our controller is completely hooked to the CAM system. The value of devq can be discarded now: sim will be passed as an argument in all further calls from CAM and devq can be derived from it. CAM provides the framework for such asynchronous events. Some events originate from the lower levels (the SIM drivers), some events originate from the peripheral drivers, some events originate from the CAM subsystem itself. Any driver can register callbacks for some types of the asynchronous events, so that it would be notified if these events occur. A typical example of such an event is a device reset. Each transaction and event identifies the devices to which it applies by the means of "path". The target-specific events normally occur during a transaction with this device. So the path from that transaction may be re-used to report this event (this is safe because the event path is copied in the event reporting routine but not deallocated nor passed anywhere further). Also it is safe to allocate paths dynamically at any time including the interrupt routines, although that incurs certain overhead, and a possible problem with this approach is that there may be no free memory at that time. For a bus reset event we need to define a wildcard path including all devices on the bus. So we can create the path for the future bus reset events in advance and avoid problems with the future memory shortage: struct cam_path *path; if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), CAM_TARGET_WILDCARD, CAM_LUN_WILDCARD) != CAM_REQ_CMP) { xpt_bus_deregister(cam_sim_path(sim)); cam_sim_free(sim, /*free_devq*/TRUE); error; /* some code to handle the error */ } softc->wpath = path; softc->sim = sim; As you can see the path includes: ID of the peripheral driver (NULL here because we have none) ID of the SIM driver (cam_sim_path(sim)) SCSI target number of the device (CAM_TARGET_WILDCARD means "all devices") SCSI LUN number of the subdevice (CAM_LUN_WILDCARD means "all LUNs") If the driver can not allocate this path it will not be able to work normally, so in that case we dismantle that SCSI bus. And we save the path pointer in the softc structure for future use. After that we save the value of sim (or we can also discard it on the exit from xxx_probe() if we wish). That is all for a minimalistic initialization. To do things right there is one more issue left. For a SIM driver there is one particularly interesting event: when a target device is considered lost. In this case resetting the SCSI negotiations with this device may be a good idea. So we register a callback for this event with CAM. The request is passed to CAM by requesting CAM action on a CAM control block for this type of request: struct ccb_setasync csa; xpt_setup_ccb(&csa.ccb_h, path, /*priority*/5); csa.ccb_h.func_code = XPT_SASYNC_CB; csa.event_enable = AC_LOST_DEVICE; csa.callback = xxx_async; csa.callback_arg = sim; xpt_action((union ccb *)&csa); Now we take a look at the xxx_action() and xxx_poll() driver entry points. static void xxx_action struct cam_sim *sim, union ccb *ccb Do some action on request of the CAM subsystem. Sim describes the SIM for the request, CCB is the request itself. CCB stands for "CAM Control Block". It is a union of many specific instances, each describing arguments for some type of transactions. All of these instances share the CCB header where the common part of arguments is stored. CAM supports the SCSI controllers working in both initiator ("normal") mode and target (simulating a SCSI device) mode. Here we only consider the part relevant to the initiator mode. There are a few function and macros (in other words, methods) defined to access the public data in the struct sim: cam_sim_path(sim) - the path ID (see above) cam_sim_name(sim) - the name of the sim cam_sim_softc(sim) - the pointer to the softc (driver private data) structure cam_sim_unit(sim) - the unit number cam_sim_bus(sim) - the bus ID To identify the device, xxx_action() can get the unit number and pointer to its structure softc using these functions. The type of request is stored in ccb->ccb_h.func_code. So generally xxx_action() consists of a big switch: struct xxx_softc *softc = (struct xxx_softc *) cam_sim_softc(sim); struct ccb_hdr *ccb_h = &ccb->ccb_h; int unit = cam_sim_unit(sim); int bus = cam_sim_bus(sim); switch(ccb_h->func_code) { case ...: ... default: ccb_h->status = CAM_REQ_INVALID; xpt_done(ccb); break; } As can be seen from the default case (if an unknown command was received) the return code of the command is set into ccb->ccb_h.status and the completed CCB is returned back to CAM by calling xpt_done(ccb). xpt_done() does not have to be called from xxx_action(): For example an I/O request may be enqueued inside the SIM driver and/or its SCSI controller. Then when the device would post an interrupt signaling that the processing of this request is complete xpt_done() may be called from the interrupt handling routine. Actually, the CCB status is not only assigned as a return code but a CCB has some status all the time. Before CCB is passed to the xxx_action() routine it gets the status CCB_REQ_INPROG meaning that it is in progress. There are a surprising number of status values defined in /sys/cam/cam.h which should be able to represent the status of a request in great detail. More interesting yet, the status is in fact a "bitwise or" of an enumerated status value (the lower 6 bits) and possible additional flag-like bits (the upper bits). The enumerated values will be discussed later in more detail. The summary of them can be found in the Errors Summary section. The possible status flags are: CAM_DEV_QFRZN - if the SIM driver gets a serious error (for example, the device does not respond to the selection or breaks the SCSI protocol) when processing a CCB it should freeze the request queue by calling xpt_freeze_simq(), return the other enqueued but not processed yet CCBs for this device back to the CAM queue, then set this flag for the troublesome CCB and call xpt_done(). This flag causes the CAM subsystem to unfreeze the queue after it handles the error. CAM_AUTOSNS_VALID - if the device returned an error condition and the flag CAM_DIS_AUTOSENSE is not set in CCB the SIM driver must execute the REQUEST SENSE command automatically to extract the sense (extended error information) data from the device. If this attempt was successful the sense data should be saved in the CCB and this flag set. CAM_RELEASE_SIMQ - like CAM_DEV_QFRZN but used in case there is some problem (or resource shortage) with the SCSI controller itself. Then all the future requests to the controller should be stopped by xpt_freeze_simq(). The controller queue will be restarted after the SIM driver overcomes the shortage and informs CAM by returning some CCB with this flag set. CAM_SIM_QUEUED - when SIM puts a CCB into its request queue this flag should be set (and removed when this CCB gets dequeued before being returned back to CAM). This flag is not used anywhere in the CAM code now, so its purpose is purely diagnostic. The function xxx_action() is not allowed to sleep, so all the synchronization for resource access must be done using SIM or device queue freezing. Besides the aforementioned flags the CAM subsystem provides functions xpt_selease_simq() and xpt_release_devq() to unfreeze the queues directly, without passing a CCB to CAM. The CCB header contains the following fields: path - path ID for the request target_id - target device ID for the request target_lun - LUN ID of the target device timeout - timeout interval for this command, in milliseconds timeout_ch - a convenience place for the SIM driver to store the timeout handle (the CAM subsystem itself does not make any assumptions about it) flags - various bits of information about the request spriv_ptr0, spriv_ptr1 - fields reserved for private use by the SIM driver (such as linking to the SIM queues or SIM private control blocks); actually, they exist as unions: spriv_ptr0 and spriv_ptr1 have the type (void *), spriv_field0 and spriv_field1 have the type unsigned long, sim_priv.entries[0].bytes and sim_priv.entries[1].bytes are byte arrays of the size consistent with the other incarnations of the union and sim_priv.bytes is one array, twice bigger. The recommended way of using the SIM private fields of CCB is to define some meaningful names for them and use these meaningful names in the driver, like: #define ccb_some_meaningful_name sim_priv.entries[0].bytes #define ccb_hcb spriv_ptr1 /* for hardware control block */ The most common initiator mode requests are: XPT_SCSI_IO - execute an I/O transaction The instance "struct ccb_scsiio csio" of the union ccb is used to transfer the arguments. They are: cdb_io - pointer to the SCSI command buffer or the buffer itself cdb_len - SCSI command length data_ptr - pointer to the data buffer (gets a bit complicated if scatter/gather is used) dxfer_len - length of the data to transfer sglist_cnt - counter of the scatter/gather segments scsi_status - place to return the SCSI status sense_data - buffer for the SCSI sense information if the command returns an error (the SIM driver is supposed to run the REQUEST SENSE command automatically in this case if the CCB flag CAM_DIS_AUTOSENSE is not set) sense_len - the length of that buffer (if it happens to be higher than size of sense_data the SIM driver must silently assume the smaller value) resid, sense_resid - if the transfer of data or SCSI sense returned an error these are the returned counters of the residual (not transferred) data. They do not seem to be especially meaningful, so in a case when they are difficult to compute (say, counting bytes in the SCSI controller's FIFO buffer) an approximate value will do as well. For a successfully completed transfer they must be set to zero. tag_action - the kind of tag to use: CAM_TAG_ACTION_NONE - do not use tags for this transaction MSG_SIMPLE_Q_TAG, MSG_HEAD_OF_Q_TAG, MSG_ORDERED_Q_TAG - value equal to the appropriate tag message (see /sys/cam/scsi/scsi_message.h); this gives only the tag type, the SIM driver must assign the tag value itself The general logic of handling this request is the following: The first thing to do is to check for possible races, to make sure that the command did not get aborted when it was sitting in the queue: struct ccb_scsiio *csio = &ccb->csio; if ((ccb_h->status & CAM_STATUS_MASK) != CAM_REQ_INPROG) { xpt_done(ccb); return; } Also we check that the device is supported at all by our controller: if(ccb_h->target_id > OUR_MAX_SUPPORTED_TARGET_ID || cch_h->target_id == OUR_SCSI_CONTROLLERS_OWN_ID) { ccb_h->status = CAM_TID_INVALID; xpt_done(ccb); return; } if(ccb_h->target_lun > OUR_MAX_SUPPORTED_LUN) { ccb_h->status = CAM_LUN_INVALID; xpt_done(ccb); return; } Then allocate whatever data structures (such as card-dependent hardware control block) we need to process this request. If we ca not then freeze the SIM queue and remember that we have a pending operation, return the CCB back and ask CAM to re-queue it. Later when the resources become available the SIM queue must be unfrozen by returning a ccb with the CAM_SIMQ_RELEASE bit set in its status. Otherwise, if all went well, link the CCB with the hardware control block (HCB) and mark it as queued. struct xxx_hcb *hcb = allocate_hcb(softc, unit, bus); if(hcb == NULL) { softc->flags |= RESOURCE_SHORTAGE; xpt_freeze_simq(sim, /*count*/1); ccb_h->status = CAM_REQUEUE_REQ; xpt_done(ccb); return; } hcb->ccb = ccb; ccb_h->ccb_hcb = (void *)hcb; ccb_h->status |= CAM_SIM_QUEUED; Extract the target data from CCB into the hardware control block. Check if we are asked to assign a tag and if yes then generate an unique tag and build the SCSI tag messages. The SIM driver is also responsible for negotiations with the devices to set the maximal mutually supported bus width, synchronous rate and offset. hcb->target = ccb_h->target_id; hcb->lun = ccb_h->target_lun; generate_identify_message(hcb); if( ccb_h->tag_action != CAM_TAG_ACTION_NONE ) generate_unique_tag_message(hcb, ccb_h->tag_action); if( !target_negotiated(hcb) ) generate_negotiation_messages(hcb); Then set up the SCSI command. The command storage may be specified in the CCB in many interesting ways, specified by the CCB flags. The command buffer can be contained in CCB or pointed to, in the latter case the pointer may be physical or virtual. Since the hardware commonly needs physical address we always convert the address to the physical one. A NOT-QUITE RELATED NOTE: Normally this is done by a call to vtophys(), but for the PCI device (which account for most of the SCSI controllers now) drivers' portability to the Alpha architecture the conversion must be done by vtobus() instead due to special Alpha quirks. [IMHO it would be much better to have two separate functions, vtop() and ptobus() then vtobus() would be a simple superposition of them.] In case if a physical address is requested it is OK to return the CCB with the status CAM_REQ_INVALID, the current drivers do that. But it is also possible to compile the Alpha-specific piece of code, as in this example (there should be a more direct way to do that, without conditional compilation in the drivers). If necessary a physical address can be also converted or mapped back to a virtual address but with big pain, so we do not do that. if(ccb_h->flags & CAM_CDB_POINTER) { /* CDB is a pointer */ if(!(ccb_h->flags & CAM_CDB_PHYS)) { /* CDB pointer is virtual */ hcb->cmd = vtobus(csio->cdb_io.cdb_ptr); } else { /* CDB pointer is physical */ #if defined(__alpha__) hcb->cmd = csio->cdb_io.cdb_ptr | alpha_XXX_dmamap_or ; #else hcb->cmd = csio->cdb_io.cdb_ptr ; #endif } } else { /* CDB is in the ccb (buffer) */ hcb->cmd = vtobus(csio->cdb_io.cdb_bytes); } hcb->cmdlen = csio->cdb_len; Now it is time to set up the data. Again, the data storage may be specified in the CCB in many interesting ways, specified by the CCB flags. First we get the direction of the data transfer. The simplest case is if there is no data to transfer: int dir = (ccb_h->flags & CAM_DIR_MASK); if (dir == CAM_DIR_NONE) goto end_data; Then we check if the data is in one chunk or in a scatter-gather list, and the addresses are physical or virtual. The SCSI controller may be able to handle only a limited number of chunks of limited length. If the request hits this limitation we return an error. We use a special function to return the CCB to handle in one place the HCB resource shortages. The functions to add chunks are driver-dependent, and here we leave them without detailed implementation. See description of the SCSI command (CDB) handling for the details on the address-translation issues. If some variation is too difficult or impossible to implement with a particular card it is OK to return the status CAM_REQ_INVALID. Actually, it seems like the scatter-gather ability is not used anywhere in the CAM code now. But at least the case for a single non-scattered virtual buffer must be implemented, it is actively used by CAM. int rv; initialize_hcb_for_data(hcb); if((!(ccb_h->flags & CAM_SCATTER_VALID)) { /* single buffer */ if(!(ccb_h->flags & CAM_DATA_PHYS)) { rv = add_virtual_chunk(hcb, csio->data_ptr, csio->dxfer_len, dir); } } else { rv = add_physical_chunk(hcb, csio->data_ptr, csio->dxfer_len, dir); } } else { int i; struct bus_dma_segment *segs; segs = (struct bus_dma_segment *)csio->data_ptr; if ((ccb_h->flags & CAM_SG_LIST_PHYS) != 0) { /* The SG list pointer is physical */ rv = setup_hcb_for_physical_sg_list(hcb, segs, csio->sglist_cnt); } else if (!(ccb_h->flags & CAM_DATA_PHYS)) { /* SG buffer pointers are virtual */ for (i = 0; i < csio->sglist_cnt; i++) { rv = add_virtual_chunk(hcb, segs[i].ds_addr, segs[i].ds_len, dir); if (rv != CAM_REQ_CMP) break; } } else { /* SG buffer pointers are physical */ for (i = 0; i < csio->sglist_cnt; i++) { rv = add_physical_chunk(hcb, segs[i].ds_addr, segs[i].ds_len, dir); if (rv != CAM_REQ_CMP) break; } } } if(rv != CAM_REQ_CMP) { /* we expect that add_*_chunk() functions return CAM_REQ_CMP * if they added a chunk successfully, CAM_REQ_TOO_BIG if * the request is too big (too many bytes or too many chunks), * CAM_REQ_INVALID in case of other troubles */ free_hcb_and_ccb_done(hcb, ccb, rv); return; } end_data: If disconnection is disabled for this CCB we pass this information to the hcb: if(ccb_h->flags & CAM_DIS_DISCONNECT) hcb_disable_disconnect(hcb); If the controller is able to run REQUEST SENSE command all by itself then the value of the flag CAM_DIS_AUTOSENSE should also be passed to it, to prevent automatic REQUEST SENSE if the CAM subsystem does not want it. The only thing left is to set up the timeout, pass our hcb to the hardware and return, the rest will be done by the interrupt handler (or timeout handler). ccb_h->timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, (ccb_h->timeout * hz) / 1000); /* convert milliseconds to ticks */ put_hcb_into_hardware_queue(hcb); return; And here is a possible implementation of the function returning CCB: static void free_hcb_and_ccb_done(struct xxx_hcb *hcb, union ccb *ccb, u_int32_t status) { struct xxx_softc *softc = hcb->softc; ccb->ccb_h.ccb_hcb = 0; if(hcb != NULL) { untimeout(xxx_timeout, (caddr_t) hcb, ccb->ccb_h.timeout_ch); /* we're about to free a hcb, so the shortage has ended */ if(softc->flags & RESOURCE_SHORTAGE) { softc->flags &= ~RESOURCE_SHORTAGE; status |= CAM_RELEASE_SIMQ; } free_hcb(hcb); /* also removes hcb from any internal lists */ } ccb->ccb_h.status = status | (ccb->ccb_h.status & ~(CAM_STATUS_MASK|CAM_SIM_QUEUED)); xpt_done(ccb); } XPT_RESET_DEV - send the SCSI "BUS DEVICE RESET" message to a device There is no data transferred in CCB except the header and the most interesting argument of it is target_id. Depending on the controller hardware a hardware control block just like for the XPT_SCSI_IO request may be constructed (see XPT_SCSI_IO request description) and sent to the controller or the SCSI controller may be immediately programmed to send this RESET message to the device or this request may be just not supported (and return the status CAM_REQ_INVALID). Also on completion of the request all the disconnected transactions for this target must be aborted (probably in the interrupt routine). Also all the current negotiations for the target are lost on reset, so they might be cleaned too. Or they clearing may be deferred, because anyway the target would request re-negotiation on the next transaction. XPT_RESET_BUS - send the RESET signal to the SCSI bus No arguments are passed in the CCB, the only interesting argument is the SCSI bus indicated by the struct sim pointer. A minimalistic implementation would forget the SCSI negotiations for all the devices on the bus and return the status CAM_REQ_CMP. The proper implementation would in addition actually reset the SCSI bus (possible also reset the SCSI controller) and mark all the CCBs being processed, both those in the hardware queue and those being disconnected, as done with the status CAM_SCSI_BUS_RESET. Like: int targ, lun; struct xxx_hcb *h, *hh; struct ccb_trans_settings neg; struct cam_path *path; /* The SCSI bus reset may take a long time, in this case its completion * should be checked by interrupt or timeout. But for simplicity * we assume here that it's really fast. */ reset_scsi_bus(softc); /* drop all enqueued CCBs */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* the clean values of negotiations to report */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); /* drop all disconnected CCBs and clean negotiations */ for(targ=0; targ <= OUR_MAX_SUPPORTED_TARGET; targ++) { clean_negotiations(softc, targ); /* report the event if possible */ if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), targ, CAM_LUN_WILDCARD) == CAM_REQ_CMP) { xpt_async(AC_TRANSFER_NEG, path, &neg); xpt_free_path(path); } for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[targ][lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); /* report the event */ xpt_async(AC_BUS_RESET, softc->wpath, NULL); return; Implementing the SCSI bus reset as a function may be a good idea because it would be re-used by the timeout function as a last resort if the things go wrong. XPT_ABORT - abort the specified CCB The arguments are transferred in the instance "struct ccb_abort cab" of the union ccb. The only argument field in it is: abort_ccb - pointer to the CCB to be aborted If the abort is not supported just return the status CAM_UA_ABORT. This is also the easy way to minimally implement this call, return CAM_UA_ABORT in any case. The hard way is to implement this request honestly. First check that abort applies to a SCSI transaction: struct ccb *abort_ccb; abort_ccb = ccb->cab.abort_ccb; if(abort_ccb->ccb_h.func_code != XPT_SCSI_IO) { ccb->ccb_h.status = CAM_UA_ABORT; xpt_done(ccb); return; } Then it is necessary to find this CCB in our queue. This can be done by walking the list of all our hardware control blocks in search for one associated with this CCB: struct xxx_hcb *hcb, *h; hcb = NULL; /* We assume that softc->first_hcb is the head of the list of all * HCBs associated with this bus, including those enqueued for * processing, being processed by hardware and disconnected ones. */ for(h = softc->first_hcb; h != NULL; h = h->next) { if(h->ccb == abort_ccb) { hcb = h; break; } } if(hcb == NULL) { /* no such CCB in our queue */ ccb->ccb_h.status = CAM_PATH_INVALID; xpt_done(ccb); return; } hcb=found_hcb; Now we look at the current processing status of the HCB. It may be either sitting in the queue waiting to be sent to the SCSI bus, being transferred right now, or disconnected and waiting for the result of the command, or actually completed by hardware but not yet marked as done by software. To make sure that we do not get in any races with hardware we mark the HCB as being aborted, so that if this HCB is about to be sent to the SCSI bus the SCSI controller will see this flag and skip it. int hstatus; /* shown as a function, in case special action is needed to make * this flag visible to hardware */ set_hcb_flags(hcb, HCB_BEING_ABORTED); abort_again: hstatus = get_hcb_status(hcb); switch(hstatus) { case HCB_SITTING_IN_QUEUE: remove_hcb_from_hardware_queue(hcb); /* FALLTHROUGH */ case HCB_COMPLETED: /* this is an easy case */ free_hcb_and_ccb_done(hcb, abort_ccb, CAM_REQ_ABORTED); break; If the CCB is being transferred right now we would like to signal to the SCSI controller in some hardware-dependent way that we want to abort the current transfer. The SCSI controller would set the SCSI ATTENTION signal and when the target responds to it send an ABORT message. We also reset the timeout to make sure that the target is not sleeping forever. If the command would not get aborted in some reasonable time like 10 seconds the timeout routine would go ahead and reset the whole SCSI bus. Because the command will be aborted in some reasonable time we can just return the abort request now as successfully completed, and mark the aborted CCB as aborted (but not mark it as done yet). case HCB_BEING_TRANSFERRED: untimeout(xxx_timeout, (caddr_t) hcb, abort_ccb->ccb_h.timeout_ch); abort_ccb->ccb_h.timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, 10 * hz); abort_ccb->ccb_h.status = CAM_REQ_ABORTED; /* ask the controller to abort that HCB, then generate * an interrupt and stop */ if(signal_hardware_to_abort_hcb_and_stop(hcb) < 0) { /* oops, we missed the race with hardware, this transaction * got off the bus before we aborted it, try again */ goto abort_again; } break; If the CCB is in the list of disconnected then set it up as an abort request and re-queue it at the front of hardware queue. Reset the timeout and report the abort request to be completed. case HCB_DISCONNECTED: untimeout(xxx_timeout, (caddr_t) hcb, abort_ccb->ccb_h.timeout_ch); abort_ccb->ccb_h.timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, 10 * hz); put_abort_message_into_hcb(hcb); put_hcb_at_the_front_of_hardware_queue(hcb); break; } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; That is all for the ABORT request, although there is one more issue. Because the ABORT message cleans all the ongoing transactions on a LUN we have to mark all the other active transactions on this LUN as aborted. That should be done in the interrupt routine, after the transaction gets aborted. Implementing the CCB abort as a function may be quite a good idea, this function can be re-used if an I/O transaction times out. The only difference would be that the timed out transaction would return the status CAM_CMD_TIMEOUT for the timed out request. Then the case XPT_ABORT would be small, like that: case XPT_ABORT: struct ccb *abort_ccb; abort_ccb = ccb->cab.abort_ccb; if(abort_ccb->ccb_h.func_code != XPT_SCSI_IO) { ccb->ccb_h.status = CAM_UA_ABORT; xpt_done(ccb); return; } if(xxx_abort_ccb(abort_ccb, CAM_REQ_ABORTED) < 0) /* no such CCB in our queue */ ccb->ccb_h.status = CAM_PATH_INVALID; else ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; XPT_SET_TRAN_SETTINGS - explicitly set values of SCSI transfer settings The arguments are transferred in the instance "struct ccb_trans_setting cts" of the union ccb: valid - a bitmask showing which settings should be updated: CCB_TRANS_SYNC_RATE_VALID - synchronous transfer rate CCB_TRANS_SYNC_OFFSET_VALID - synchronous offset CCB_TRANS_BUS_WIDTH_VALID - bus width CCB_TRANS_DISC_VALID - set enable/disable disconnection CCB_TRANS_TQ_VALID - set enable/disable tagged queuing flags - consists of two parts, binary arguments and identification of - sub-operations. The binary arguments are : + sub-operations. The binary arguments are: CCB_TRANS_DISC_ENB - enable disconnection CCB_TRANS_TAG_ENB - enable tagged queuing the sub-operations are: CCB_TRANS_CURRENT_SETTINGS - change the current negotiations CCB_TRANS_USER_SETTINGS - remember the desired user values sync_period, sync_offset - self-explanatory, if sync_offset==0 then the asynchronous mode is requested bus_width - bus width, in bits (not bytes) Two sets of negotiated parameters are supported, the user settings and the current settings. The user settings are not really used much in the SIM drivers, this is mostly just a piece of memory where the upper levels can store (and later recall) its ideas about the parameters. Setting the user parameters does not cause re-negotiation of the transfer rates. But when the SCSI controller does a negotiation it must never set the values higher than the user parameters, so it is essentially the top boundary. The current settings are, as the name says, current. Changing them means that the parameters must be re-negotiated on the next transfer. Again, these "new current settings" are not supposed to be forced on the device, just they are used as the initial step of negotiations. Also they must be limited by actual capabilities of the SCSI controller: for example, if the SCSI controller has 8-bit bus and the request asks to set 16-bit wide transfers this parameter must be silently truncated to 8-bit transfers before sending it to the device. One caveat is that the bus width and synchronous parameters are per target while the disconnection and tag enabling parameters are per lun. The recommended implementation is to keep 3 sets of negotiated (bus width and synchronous transfer) parameters: user - the user set, as above current - those actually in effect goal - those requested by setting of the "current" parameters The code looks like: struct ccb_trans_settings *cts; int targ, lun; int flags; cts = &ccb->cts; targ = ccb_h->target_id; lun = ccb_h->target_lun; flags = cts->flags; if(flags & CCB_TRANS_USER_SETTINGS) { if(flags & CCB_TRANS_SYNC_RATE_VALID) softc->user_sync_period[targ] = cts->sync_period; if(flags & CCB_TRANS_SYNC_OFFSET_VALID) softc->user_sync_offset[targ] = cts->sync_offset; if(flags & CCB_TRANS_BUS_WIDTH_VALID) softc->user_bus_width[targ] = cts->bus_width; if(flags & CCB_TRANS_DISC_VALID) { softc->user_tflags[targ][lun] &= ~CCB_TRANS_DISC_ENB; softc->user_tflags[targ][lun] |= flags & CCB_TRANS_DISC_ENB; } if(flags & CCB_TRANS_TQ_VALID) { softc->user_tflags[targ][lun] &= ~CCB_TRANS_TQ_ENB; softc->user_tflags[targ][lun] |= flags & CCB_TRANS_TQ_ENB; } } if(flags & CCB_TRANS_CURRENT_SETTINGS) { if(flags & CCB_TRANS_SYNC_RATE_VALID) softc->goal_sync_period[targ] = max(cts->sync_period, OUR_MIN_SUPPORTED_PERIOD); if(flags & CCB_TRANS_SYNC_OFFSET_VALID) softc->goal_sync_offset[targ] = min(cts->sync_offset, OUR_MAX_SUPPORTED_OFFSET); if(flags & CCB_TRANS_BUS_WIDTH_VALID) softc->goal_bus_width[targ] = min(cts->bus_width, OUR_BUS_WIDTH); if(flags & CCB_TRANS_DISC_VALID) { softc->current_tflags[targ][lun] &= ~CCB_TRANS_DISC_ENB; softc->current_tflags[targ][lun] |= flags & CCB_TRANS_DISC_ENB; } if(flags & CCB_TRANS_TQ_VALID) { softc->current_tflags[targ][lun] &= ~CCB_TRANS_TQ_ENB; softc->current_tflags[targ][lun] |= flags & CCB_TRANS_TQ_ENB; } } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; Then when the next I/O request will be processed it will check if it has to re-negotiate, for example by calling the function target_negotiated(hcb). It can be implemented like this: int target_negotiated(struct xxx_hcb *hcb) { struct softc *softc = hcb->softc; int targ = hcb->targ; if( softc->current_sync_period[targ] != softc->goal_sync_period[targ] || softc->current_sync_offset[targ] != softc->goal_sync_offset[targ] || softc->current_bus_width[targ] != softc->goal_bus_width[targ] ) return 0; /* FALSE */ else return 1; /* TRUE */ } After the values are re-negotiated the resulting values must be assigned to both current and goal parameters, so for future I/O transactions the current and goal parameters would be the same and target_negotiated() would return TRUE. When the card is initialized (in xxx_attach()) the current negotiation values must be initialized to narrow asynchronous mode, the goal and current values must be initialized to the maximal values supported by controller. XPT_GET_TRAN_SETTINGS - get values of SCSI transfer settings This operations is the reverse of XPT_SET_TRAN_SETTINGS. Fill up the CCB instance "struct ccb_trans_setting cts" with data as requested by the flags CCB_TRANS_CURRENT_SETTINGS or CCB_TRANS_USER_SETTINGS (if both are set then the existing drivers return the current settings). Set all the bits in the valid field. XPT_CALC_GEOMETRY - calculate logical (BIOS) geometry of the disk The arguments are transferred in the instance "struct ccb_calc_geometry ccg" of the union ccb: block_size - input, block (A.K.A sector) size in bytes volume_size - input, volume size in bytes cylinders - output, logical cylinders heads - output, logical heads secs_per_track - output, logical sectors per track If the returned geometry differs much enough from what the SCSI controller BIOS thinks and a disk on this SCSI controller is used as bootable the system may not be able to boot. The typical calculation example taken from the aic7xxx driver is: struct ccb_calc_geometry *ccg; u_int32_t size_mb; u_int32_t secs_per_cylinder; int extended; ccg = &ccb->ccg; size_mb = ccg->volume_size / ((1024L * 1024L) / ccg->block_size); extended = check_cards_EEPROM_for_extended_geometry(softc); if (size_mb > 1024 && extended) { ccg->heads = 255; ccg->secs_per_track = 63; } else { ccg->heads = 64; ccg->secs_per_track = 32; } secs_per_cylinder = ccg->heads * ccg->secs_per_track; ccg->cylinders = ccg->volume_size / secs_per_cylinder; ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; This gives the general idea, the exact calculation depends on the quirks of the particular BIOS. If BIOS provides no way set the "extended translation" flag in EEPROM this flag should normally be assumed equal to 1. Other popular geometries are: 128 heads, 63 sectors - Symbios controllers 16 heads, 63 sectors - old controllers Some system BIOSes and SCSI BIOSes fight with each other with variable success, for example a combination of Symbios 875/895 SCSI and Phoenix BIOS can give geometry 128/63 after power up and 255/63 after a hard reset or soft reboot. XPT_PATH_INQ - path inquiry, in other words get the SIM driver and SCSI controller (also known as HBA - Host Bus Adapter) properties The properties are returned in the instance "struct ccb_pathinq cpi" of the union ccb: version_num - the SIM driver version number, now all drivers use 1 hba_inquiry - bitmask of features supported by the controller: PI_MDP_ABLE - supports MDP message (something from SCSI3?) PI_WIDE_32 - supports 32 bit wide SCSI PI_WIDE_16 - supports 16 bit wide SCSI PI_SDTR_ABLE - can negotiate synchronous transfer rate PI_LINKED_CDB - supports linked commands PI_TAG_ABLE - supports tagged commands PI_SOFT_RST - supports soft reset alternative (hard reset and soft reset are mutually exclusive within a SCSI bus) target_sprt - flags for target mode support, 0 if unsupported hba_misc - miscellaneous controller features: PIM_SCANHILO - bus scans from high ID to low ID PIM_NOREMOVE - removable devices not included in scan PIM_NOINITIATOR - initiator role not supported PIM_NOBUSRESET - user has disabled initial BUS RESET hba_eng_cnt - mysterious HBA engine count, something related to compression, now is always set to 0 vuhba_flags - vendor-unique flags, unused now max_target - maximal supported target ID (7 for 8-bit bus, 15 for 16-bit bus, 127 for Fibre Channel) max_lun - maximal supported LUN ID (7 for older SCSI controllers, 63 for newer ones) async_flags - bitmask of installed Async handler, unused now hpath_id - highest Path ID in the subsystem, unused now unit_number - the controller unit number, cam_sim_unit(sim) bus_id - the bus number, cam_sim_bus(sim) initiator_id - the SCSI ID of the controller itself base_transfer_speed - nominal transfer speed in KB/s for asynchronous narrow transfers, equals to 3300 for SCSI sim_vid - SIM driver's vendor id, a zero-terminated string of maximal length SIM_IDLEN including the terminating zero hba_vid - SCSI controller's vendor id, a zero-terminated string of maximal length HBA_IDLEN including the terminating zero dev_name - device driver name, a zero-terminated string of maximal length DEV_IDLEN including the terminating zero, equal to cam_sim_name(sim) The recommended way of setting the string fields is using strncpy, like: strncpy(cpi->dev_name, cam_sim_name(sim), DEV_IDLEN); After setting the values set the status to CAM_REQ_CMP and mark the CCB as done. Polling static void xxx_poll struct cam_sim *sim The poll function is used to simulate the interrupts when the interrupt subsystem is not functioning (for example, when the system has crashed and is creating the system dump). The CAM subsystem sets the proper interrupt level before calling the poll routine. So all it needs to do is to call the interrupt routine (or the other way around, the poll routine may be doing the real action and the interrupt routine would just call the - poll routine). Why bother about a separate function then ? + poll routine). Why bother about a separate function then? Because of different calling conventions. The xxx_poll routine gets the struct cam_sim pointer as its argument when the PCI interrupt routine by common convention gets pointer to the struct xxx_softc and the ISA interrupt routine gets just the device unit number. So the poll routine would normally look as: static void xxx_poll(struct cam_sim *sim) { xxx_intr((struct xxx_softc *)cam_sim_softc(sim)); /* for PCI device */ } or static void xxx_poll(struct cam_sim *sim) { xxx_intr(cam_sim_unit(sim)); /* for ISA device */ } Asynchronous Events If an asynchronous event callback has been set up then the callback function should be defined. static void ahc_async(void *callback_arg, u_int32_t code, struct cam_path *path, void *arg) callback_arg - the value supplied when registering the callback code - identifies the type of event path - identifies the devices to which the event applies arg - event-specific argument Implementation for a single type of event, AC_LOST_DEVICE, looks like: struct xxx_softc *softc; struct cam_sim *sim; int targ; struct ccb_trans_settings neg; sim = (struct cam_sim *)callback_arg; softc = (struct xxx_softc *)cam_sim_softc(sim); switch (code) { case AC_LOST_DEVICE: targ = xpt_path_target_id(path); if(targ <= OUR_MAX_SUPPORTED_TARGET) { clean_negotiations(softc, targ); /* send indication to CAM */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); xpt_async(AC_TRANSFER_NEG, path, &neg); } break; default: break; } Interrupts The exact type of the interrupt routine depends on the type of the peripheral bus (PCI, ISA and so on) to which the SCSI controller is connected. The interrupt routines of the SIM drivers run at the interrupt level splcam. So splcam() should be used in the driver to synchronize activity between the interrupt routine and the rest of the driver (for a multiprocessor-aware driver things get yet more interesting but we ignore this case here). The pseudo-code in this document happily ignores the problems of synchronization. The real code must not ignore them. A simple-minded approach is to set splcam() on the entry to the other routines and reset it on return thus protecting them by one big critical section. To make sure that the interrupt level will be always restored a wrapper function can be defined, like: static void xxx_action(struct cam_sim *sim, union ccb *ccb) { int s; s = splcam(); xxx_action1(sim, ccb); splx(s); } static void xxx_action1(struct cam_sim *sim, union ccb *ccb) { ... process the request ... } This approach is simple and robust but the problem with it is that interrupts may get blocked for a relatively long time and this would negatively affect the system's performance. On the other hand the functions of the spl() family have rather high overhead, so vast amount of tiny critical sections may not be good either. The conditions handled by the interrupt routine and the details depend very much on the hardware. We consider the set of "typical" conditions. First, we check if a SCSI reset was encountered on the bus (probably caused by another SCSI controller on the same SCSI bus). If so we drop all the enqueued and disconnected requests, report the events and re-initialize our SCSI controller. It is important that during this initialization the controller will not issue another reset or else two controllers on the same SCSI bus could ping-pong resets forever. The case of fatal controller error/hang could be handled in the same place, but it will probably need also sending RESET signal to the SCSI bus to reset the status of the connections with the SCSI devices. int fatal=0; struct ccb_trans_settings neg; struct cam_path *path; if( detected_scsi_reset(softc) || (fatal = detected_fatal_controller_error(softc)) ) { int targ, lun; struct xxx_hcb *h, *hh; /* drop all enqueued CCBs */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* the clean values of negotiations to report */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); /* drop all disconnected CCBs and clean negotiations */ for(targ=0; targ <= OUR_MAX_SUPPORTED_TARGET; targ++) { clean_negotiations(softc, targ); /* report the event if possible */ if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), targ, CAM_LUN_WILDCARD) == CAM_REQ_CMP) { xpt_async(AC_TRANSFER_NEG, path, &neg); xpt_free_path(path); } for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[targ][lun]; h != NULL; h = hh) { hh=h->next; if(fatal) free_hcb_and_ccb_done(h, h->ccb, CAM_UNREC_HBA_ERROR); else free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } } /* report the event */ xpt_async(AC_BUS_RESET, softc->wpath, NULL); /* re-initialization may take a lot of time, in such case * its completion should be signaled by another interrupt or * checked on timeout - but for simplicity we assume here that * it's really fast */ if(!fatal) { reinitialize_controller_without_scsi_reset(softc); } else { reinitialize_controller_with_scsi_reset(softc); } schedule_next_hcb(softc); return; } If interrupt is not caused by a controller-wide condition then probably something has happened to the current hardware control block. Depending on the hardware there may be other non-HCB-related events, we just do not consider them here. Then we analyze what happened to this HCB: struct xxx_hcb *hcb, *h, *hh; int hcb_status, scsi_status; int ccb_status; int targ; int lun_to_freeze; hcb = get_current_hcb(softc); if(hcb == NULL) { /* either stray interrupt or something went very wrong * or this is something hardware-dependent */ handle as necessary; return; } targ = hcb->target; hcb_status = get_status_of_current_hcb(softc); First we check if the HCB has completed and if so we check the returned SCSI status. if(hcb_status == COMPLETED) { scsi_status = get_completion_status(hcb); Then look if this status is related to the REQUEST SENSE command and if so handle it in a simple way. if(hcb->flags & DOING_AUTOSENSE) { if(scsi_status == GOOD) { /* autosense was successful */ hcb->ccb->ccb_h.status |= CAM_AUTOSNS_VALID; free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_SCSI_STATUS_ERROR); } else { autosense_failed: free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_AUTOSENSE_FAIL); } schedule_next_hcb(softc); return; } Else the command itself has completed, pay more attention to details. If auto-sense is not disabled for this CCB and the command has failed with sense data then run REQUEST SENSE command to receive that data. hcb->ccb->csio.scsi_status = scsi_status; calculate_residue(hcb); if( (hcb->ccb->ccb_h.flags & CAM_DIS_AUTOSENSE)==0 && ( scsi_status == CHECK_CONDITION || scsi_status == COMMAND_TERMINATED) ) { /* start auto-SENSE */ hcb->flags |= DOING_AUTOSENSE; setup_autosense_command_in_hcb(hcb); restart_current_hcb(softc); return; } if(scsi_status == GOOD) free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_REQ_CMP); else free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_SCSI_STATUS_ERROR); schedule_next_hcb(softc); return; } One typical thing would be negotiation events: negotiation messages received from a SCSI target (in answer to our negotiation attempt or by target's initiative) or the target is unable to negotiate (rejects our negotiation messages or does not answer them). switch(hcb_status) { case TARGET_REJECTED_WIDE_NEG: /* revert to 8-bit bus */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = 8; /* report the event */ neg.bus_width = 8; neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); continue_current_hcb(softc); return; case TARGET_ANSWERED_WIDE_NEG: { int wd; wd = get_target_bus_width_request(softc); if(wd <= softc->goal_bus_width[targ]) { /* answer is acceptable */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = neg.bus_width = wd; /* report the event */ neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); } else { prepare_reject_message(hcb); } } continue_current_hcb(softc); return; case TARGET_REQUESTED_WIDE_NEG: { int wd; wd = get_target_bus_width_request(softc); wd = min (wd, OUR_BUS_WIDTH); wd = min (wd, softc->user_bus_width[targ]); if(wd != softc->current_bus_width[targ]) { /* the bus width has changed */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = neg.bus_width = wd; /* report the event */ neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); } prepare_width_nego_rsponse(hcb, wd); } continue_current_hcb(softc); return; } Then we handle any errors that could have happened during auto-sense in the same simple-minded way as before. Otherwise we look closer at the details again. if(hcb->flags & DOING_AUTOSENSE) goto autosense_failed; switch(hcb_status) { The next event we consider is unexpected disconnect. Which is considered normal after an ABORT or BUS DEVICE RESET message and abnormal in other cases. case UNEXPECTED_DISCONNECT: if(requested_abort(hcb)) { /* abort affects all commands on that target+LUN, so * mark all disconnected HCBs on that target+LUN as aborted too */ for(h = softc->first_discon_hcb[hcb->target][hcb->lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_REQ_ABORTED); } ccb_status = CAM_REQ_ABORTED; } else if(requested_bus_device_reset(hcb)) { int lun; /* reset affects all commands on that target, so * mark all disconnected HCBs on that target+LUN as reset */ for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[hcb->target][lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* send event */ xpt_async(AC_SENT_BDR, hcb->ccb->ccb_h.path_id, NULL); /* this was the CAM_RESET_DEV request itself, it's completed */ ccb_status = CAM_REQ_CMP; } else { calculate_residue(hcb); ccb_status = CAM_UNEXP_BUSFREE; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = hcb->lun; } break; If the target refuses to accept tags we notify CAM about that and return back all commands for this LUN: case TAGS_REJECTED: /* report the event */ neg.flags = 0 & ~CCB_TRANS_TAG_ENB; neg.valid = CCB_TRANS_TQ_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); ccb_status = CAM_MSG_REJECT_REC; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = hcb->lun; break; Then we check a number of other conditions, with processing basically limited to setting the CCB status: case SELECTION_TIMEOUT: ccb_status = CAM_SEL_TIMEOUT; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = CAM_LUN_WILDCARD; break; case PARITY_ERROR: ccb_status = CAM_UNCOR_PARITY; break; case DATA_OVERRUN: case ODD_WIDE_TRANSFER: ccb_status = CAM_DATA_RUN_ERR; break; default: /* all other errors are handled in a generic way */ ccb_status = CAM_REQ_CMP_ERR; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = CAM_LUN_WILDCARD; break; } Then we check if the error was serious enough to freeze the input queue until it gets proceeded and do so if it is: if(hcb->ccb->ccb_h.status & CAM_DEV_QFRZN) { /* freeze the queue */ xpt_freeze_devq(ccb->ccb_h.path, /*count*/1); /* re-queue all commands for this target/LUN back to CAM */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; if(targ == h->targ && (lun_to_freeze == CAM_LUN_WILDCARD || lun_to_freeze == h->lun) ) free_hcb_and_ccb_done(h, h->ccb, CAM_REQUEUE_REQ); } } free_hcb_and_ccb_done(hcb, hcb->ccb, ccb_status); schedule_next_hcb(softc); return; This concludes the generic interrupt handling although specific controllers may require some additions. Errors Summary When executing an I/O request many things may go wrong. The reason of error can be reported in the CCB status with great detail. Examples of use are spread throughout this document. For completeness here is the summary of recommended responses for the typical error conditions: CAM_RESRC_UNAVAIL - some resource is temporarily unavailable and the SIM driver cannot generate an event when it will become available. An example of this resource would be some intra-controller hardware resource for which the controller does not generate an interrupt when it becomes available. CAM_UNCOR_PARITY - unrecovered parity error occurred CAM_DATA_RUN_ERR - data overrun or unexpected data phase (going in other direction than specified in CAM_DIR_MASK) or odd transfer length for wide transfer CAM_SEL_TIMEOUT - selection timeout occurred (target does not respond) CAM_CMD_TIMEOUT - command timeout occurred (the timeout function ran) CAM_SCSI_STATUS_ERROR - the device returned error CAM_AUTOSENSE_FAIL - the device returned error and the REQUEST SENSE COMMAND failed CAM_MSG_REJECT_REC - MESSAGE REJECT message was received CAM_SCSI_BUS_RESET - received SCSI bus reset CAM_REQ_CMP_ERR - "impossible" SCSI phase occurred or something else as weird or just a generic error if further detail is not available CAM_UNEXP_BUSFREE - unexpected disconnect occurred CAM_BDR_SENT - BUS DEVICE RESET message was sent to the target CAM_UNREC_HBA_ERROR - unrecoverable Host Bus Adapter Error CAM_REQ_TOO_BIG - the request was too large for this controller CAM_REQUEUE_REQ - this request should be re-queued to preserve transaction ordering. This typically occurs when the SIM recognizes an error that should freeze the queue and must place other queued requests for the target at the sim level back into the XPT queue. Typical cases of such errors are selection timeouts, command timeouts and other like conditions. In such cases the troublesome command returns the status indicating the error, the and the other commands which have not be sent to the bus yet get re-queued. CAM_LUN_INVALID - the LUN ID in the request is not supported by the SCSI controller CAM_TID_INVALID - the target ID in the request is not supported by the SCSI controller Timeout Handling When the timeout for an HCB expires that request should be aborted, just like with an XPT_ABORT request. The only difference is that the returned status of aborted request should be CAM_CMD_TIMEOUT instead of CAM_REQ_ABORTED (that is why implementation of the abort better be done as a function). But there is one more possible problem: what if the abort request itself will get stuck? In this case the SCSI bus should be reset, just like with an XPT_RESET_BUS request (and the idea about implementing it as a function called from both places applies here too). Also we should reset the whole SCSI bus if a device reset request got stuck. So after all the timeout function would look like: static void xxx_timeout(void *arg) { struct xxx_hcb *hcb = (struct xxx_hcb *)arg; struct xxx_softc *softc; struct ccb_hdr *ccb_h; softc = hcb->softc; ccb_h = &hcb->ccb->ccb_h; if(hcb->flags & HCB_BEING_ABORTED || ccb_h->func_code == XPT_RESET_DEV) { xxx_reset_bus(softc); } else { xxx_abort_ccb(hcb->ccb, CAM_CMD_TIMEOUT); } } When we abort a request all the other disconnected requests to the same target/LUN get aborted too. So there appears a question, should we return them with status CAM_REQ_ABORTED or - CAM_CMD_TIMEOUT ? The current drivers use CAM_CMD_TIMEOUT. This + CAM_CMD_TIMEOUT? The current drivers use CAM_CMD_TIMEOUT. This seems logical because if one request got timed out then probably something really bad is happening to the device, so if they would not be disturbed they would time out by themselves. diff --git a/en_US.ISO8859-1/books/developers-handbook/driverbasics/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/driverbasics/chapter.sgml index 8dfde91426..72970d9733 100644 --- a/en_US.ISO8859-1/books/developers-handbook/driverbasics/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/driverbasics/chapter.sgml @@ -1,390 +1,390 @@ Writing FreeBSD Device Drivers This chapter was written by &a.murray; with selections from a variety of sources including the intro(4) man page by &a.joerg;. Introduction This chapter provides a brief introduction to writing device drivers for FreeBSD. A device in this context is a term used mostly for hardware-related stuff that belongs to the system, like disks, printers, or a graphics display with its keyboard. A device driver is the software component of the operating system that controls a specific device. There are also so-called pseudo-devices where a device driver emulates the behaviour of a device in software without any particular underlying hardware. Device drivers can be compiled into the system statically or loaded on demand through the dynamic kernel linker facility `kld'. Most devices in a Unix-like operating system are accessed through device-nodes, sometimes also called special files. These files are usually located under the directory /dev in the file system hierarchy. Until devfs is fully integrated into FreeBSD, each device node must be created statically and independent of the existence of the associated device driver. Most device nodes on the system are created by running MAKEDEV. Device drivers can roughly be broken down into two categories; character and network device drivers. Dynamic Kernel Linker Facility - KLD The kld interface allows system administrators to dynamically add and remove functionality from a running system. This allows device driver writers to load their new changes into a running kernel without constantly rebooting to test changes. The kld interface is used through the following - administrator commands : + administrator commands: kldload - loads a new kernel module kldunload - unloads a kernel module kldstat - lists the currently loaded modules Skeleton Layout of a kernel module /* * KLD Skeleton * Inspired by Andrew Reiter's Daemonnews article */ #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ /* * Load handler that deals with the loading and unloading of a KLD. */ static int skel_loader(struct module *m, int what, void *arg) { int err = 0; switch (what) { case MOD_LOAD: /* kldload */ uprintf("Skeleton KLD loaded.\n"); break; case MOD_UNLOAD: uprintf("Skeleton KLD unloaded.\n"); break; default: err = EINVAL; break; } return(err); } /* Declare this module to the rest of the kernel */ static moduledata_t skel_mod = { "skel", skel_loader, NULL }; DECLARE_MODULE(skeleton, skel_mod, SI_SUB_KLD, SI_ORDER_ANY); Makefile FreeBSD provides a makefile include that you can use to quickly compile your kernel addition. SRCS=skeleton.c KMOD=skeleton .include <bsd.kmod.mk> Simply running make with this makefile will create a file skeleton.ko that can - be loaded into your system by typing : + be loaded into your system by typing: &prompt.root; kldload -v ./skeleton.ko Accessing a device driver Unix provides a common set of system calls for user applications to use. The upper layers of the kernel dispatch these calls to the corresponding device driver when a user accesses a device node. The /dev/MAKEDEV script makes most of the device nodes for your system but if you are doing your own driver development it may be necessary to create your own device nodes with mknod Creating static device nodes The mknod command requires four arguments to create a device node. You must specify the name of this device node, the type of device, the major number of the device, and the minor number of the device. Dynamic device nodes The device filesystem, or devfs, provides access to the kernel's device namespace in the global filesystem namespace. This eliminates the problems of potentially having a device driver without a static device node, or a device node without an installed device driver. Devfs is still a work in progress, but it is already working quite nice. Character Devices A character device driver is one that transfers data directly to and from a user process. This is the most common type of device driver and there are plenty of simple examples in the source tree. This simple example pseudo-device remembers whatever values you write to it and can then supply them back to you when you read from it. /* * Simple `echo' pseudo-device KLD * * Murray Stokely */ #define MIN(a,b) (((a) < (b)) ? (a) : (b)) #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ #include <sys/conf.h> /* cdevsw struct */ #include <sys/uio.h> /* uio struct */ #include <sys/malloc.h> #define BUFFERSIZE 256 /* Function prototypes */ d_open_t echo_open; d_close_t echo_close; d_read_t echo_read; d_write_t echo_write; /* Character device entry points */ static struct cdevsw echo_cdevsw = { echo_open, echo_close, echo_read, echo_write, noioctl, nopoll, nommap, nostrategy, "echo", 33, /* reserved for lkms - /usr/src/sys/conf/majors */ nodump, nopsize, D_TTY, -1 }; typedef struct s_echo { char msg[BUFFERSIZE]; int len; } t_echo; /* vars */ static dev_t sdev; static int len; static int count; static t_echo *echomsg; MALLOC_DECLARE(M_ECHOBUF); MALLOC_DEFINE(M_ECHOBUF, "echobuffer", "buffer for echo module"); /* * This function acts is called by the kld[un]load(2) system calls to * determine what actions to take when a module is loaded or unloaded. */ static int echo_loader(struct module *m, int what, void *arg) { int err = 0; switch (what) { case MOD_LOAD: /* kldload */ sdev = make_dev(&echo_cdevsw, 0, UID_ROOT, GID_WHEEL, 0600, "echo"); /* kmalloc memory for use by this driver */ /* malloc(256,M_ECHOBUF,M_WAITOK); */ MALLOC(echomsg, t_echo *, sizeof(t_echo), M_ECHOBUF, M_WAITOK); printf("Echo device loaded.\n"); break; case MOD_UNLOAD: destroy_dev(sdev); FREE(echomsg,M_ECHOBUF); printf("Echo device unloaded.\n"); break; default: err = EINVAL; break; } return(err); } int echo_open(dev_t dev, int oflags, int devtype, struct proc *p) { int err = 0; uprintf("Opened device \"echo\" successfully.\n"); return(err); } int echo_close(dev_t dev, int fflag, int devtype, struct proc *p) { uprintf("Closing device \"echo.\"\n"); return(0); } /* * The read function just takes the buf that was saved via * echo_write() and returns it to userland for accessing. * uio(9) */ int echo_read(dev_t dev, struct uio *uio, int ioflag) { int err = 0; int amt; /* How big is this read operation? Either as big as the user wants, or as big as the remaining data */ amt = MIN(uio->uio_resid, (echomsg->len - uio->uio_offset > 0) ? echomsg->len - uio->uio_offset : 0); if ((err = uiomove(echomsg->msg + uio->uio_offset,amt,uio)) != 0) { uprintf("uiomove failed!\n"); } return err; } /* * echo_write takes in a character string and saves it * to buf for later accessing. */ int echo_write(dev_t dev, struct uio *uio, int ioflag) { int err = 0; /* Copy the string in from user memory to kernel memory */ err = copyin(uio->uio_iov->iov_base, echomsg->msg, MIN(uio->uio_iov->iov_len,BUFFERSIZE)); /* Now we need to null terminate */ *(echomsg->msg + MIN(uio->uio_iov->iov_len,BUFFERSIZE)) = 0; /* Record the length */ echomsg->len = MIN(uio->uio_iov->iov_len,BUFFERSIZE); if (err != 0) { uprintf("Write failed: bad address!\n"); } count++; return(err); } DEV_MODULE(echo,echo_loader,NULL); To install this driver you will first need to make a node on - your filesystem with a command such as : + your filesystem with a command such as: &prompt.root; mknod /dev/echo c 33 0 With this driver loaded you should now be able to type - something like : + something like: &prompt.root; echo -n "Test Data" > /dev/echo &prompt.root; cat /dev/echo Test Data Real hardware devices in the next chapter.. Additional Resources Dynamic Kernel Linker (KLD) Facility Programming Tutorial - Daemonnews October 2000 How to Write Kernel Drivers with NEWBUS - Daemonnews July 2000 Network Drivers Drivers for network devices do not use device nodes in order to be accessed. Their selection is based on other decisions made inside the kernel and instead of calling open(), use of a network device is generally introduced by using the system call socket(2). man ifnet(), loopback device, Bill Paul's drivers, etc.. diff --git a/en_US.ISO8859-1/books/developers-handbook/isa/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/isa/chapter.sgml index dde9cd3968..f90c0067c3 100644 --- a/en_US.ISO8859-1/books/developers-handbook/isa/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/isa/chapter.sgml @@ -1,2479 +1,2479 @@ ISA device drivers This chapter was written by &a.babkin; Modifications for the handbook made by &a.murray;, &a.wylie;, and &a.logo;. Synopsis This chapter introduces the issues relevant to writing a driver for an ISA device. The pseudo-code presented here is rather detailed and reminiscent of the real code but is still only pseudo-code. It avoids the details irrelevant to the subject of the discussion. The real-life examples can be found in the source code of real drivers. In particular the drivers "ep" and "aha" are good sources of information. Basic information A typical ISA driver would need the following include files: #include <sys/module.h> #include <sys/bus.h> #include <machine/bus.h> #include <machine/resource.h> #include <sys/rman.h> #include <isa/isavar.h> #include <isa/pnpvar.h> They describe the things specific to the ISA and generic bus subsystem. The bus subsystem is implemented in an object-oriented fashion, its main structures are accessed by associated method functions. The list of bus methods implemented by an ISA driver is like one for any other bus. For a hypothetical driver named "xxx" they would be: static void xxx_isa_identify (driver_t *, device_t); Normally used for bus drivers, not device drivers. But for ISA devices this method may have special use: if the device provides some device-specific (non-PnP) way to auto-detect devices this routine may implement it. static int xxx_isa_probe (device_t dev); Probe for a device at a known (or PnP) location. This routine can also accommodate device-specific auto-detection of parameters for partially configured devices. static int xxx_isa_attach (device_t dev); Attach and initialize device. static int xxx_isa_detach (device_t dev); Detach device before unloading the driver module. static int xxx_isa_shutdown (device_t dev); Execute shutdown of the device before system shutdown. static int xxx_isa_suspend (device_t dev); Suspend the device before the system goes to the power-save state. May also abort transition to the power-save state. static int xxx_isa_resume (device_t dev); Resume the device activity after return from power-save state. xxx_isa_probe() and xxx_isa_attach() are mandatory, the rest of the routines are optional, depending on the device's needs. The driver is linked to the system with the following set of descriptions. /* table of supported bus methods */ static device_method_t xxx_isa_methods[] = { /* list all the bus method functions supported by the driver */ /* omit the unsupported methods */ DEVMETHOD(device_identify, xxx_isa_identify), DEVMETHOD(device_probe, xxx_isa_probe), DEVMETHOD(device_attach, xxx_isa_attach), DEVMETHOD(device_detach, xxx_isa_detach), DEVMETHOD(device_shutdown, xxx_isa_shutdown), DEVMETHOD(device_suspend, xxx_isa_suspend), DEVMETHOD(device_resume, xxx_isa_resume), { 0, 0 } }; static driver_t xxx_isa_driver = { "xxx", xxx_isa_methods, sizeof(struct xxx_softc), }; static devclass_t xxx_devclass; DRIVER_MODULE(xxx, isa, xxx_isa_driver, xxx_devclass, load_function, load_argument); Here struct xxx_softc is a device-specific structure that contains private driver data and descriptors for the driver's resources. The bus code automatically allocates one softc descriptor per device as needed. If the driver is implemented as a loadable module then load_function() is called to do driver-specific initialization or clean-up when the driver is loaded or unloaded and load_argument is passed as one of its arguments. If the driver does not support dynamic loading (in other words it must always be linked into kernel) then these values should be set to 0 and the last definition would look like: DRIVER_MODULE(xxx, isa, xxx_isa_driver, xxx_devclass, 0, 0); If the driver is for a device which supports PnP then a table of supported PnP IDs must be defined. The table consists of a list of PnP IDs supported by this driver and human-readable descriptions of the hardware types and models having these IDs. It looks like: static struct isa_pnp_id xxx_pnp_ids[] = { /* a line for each supported PnP ID */ { 0x12345678, "Our device model 1234A" }, { 0x12345679, "Our device model 1234B" }, { 0, NULL }, /* end of table */ }; If the driver does not support PnP devices it still needs an empty PnP ID table, like: static struct isa_pnp_id xxx_pnp_ids[] = { { 0, NULL }, /* end of table */ }; Device_t pointer Device_t is the pointer type for the device structure. Here we consider only the methods interesting from the device driver writer's standpoint. The methods to manipulate values in the device structure are: device_t device_get_parent(dev) Get the parent bus of a device. driver_t device_get_driver(dev) Get pointer to its driver structure. char *device_get_name(dev) Get the driver name, such as "xxx" for our example. int device_get_unit(dev) Get the unit number (units are numbered from 0 for the devices associated with each driver). char *device_get_nameunit(dev) Get the device name - including the unit number, such as "xxx0" , "xxx1" and so + including the unit number, such as "xxx0", "xxx1" and so on. char *device_get_desc(dev) Get the device description. Normally it describes the exact model of device in human-readable form. device_set_desc(dev, desc) Set the description. This makes the device description point to the string desc which may not be deallocated or changed after that. device_set_desc_copy(dev, desc) Set the description. The description is copied into an internal dynamically allocated buffer, so the string desc may be changed afterwards without adverse effects. void *device_get_softc(dev) Get pointer to the device descriptor (struct xxx_softc) associated with this device. u_int32_t device_get_flags(dev) Get the flags specified for the device in the configuration file. A convenience function device_printf(dev, fmt, ...) may be used to print the messages from the device driver. It automatically prepends the unitname and colon to the message. The device_t methods are implemented in the file kern/bus_subr.c. Config file and the order of identifying and probing during auto-configuration The ISA devices are described in the kernel config file like: device xxx0 at isa? port 0x300 irq 10 drq 5 iomem 0xd0000 flags 0x1 sensitive The values of port, IRQ and so on are converted to the resource values associated with the device. They are optional, depending on the device needs and abilities for auto-configuration. For example, some devices do not need DRQ at all and some allow the driver to read the IRQ setting from the device configuration ports. If a machine has multiple ISA buses the exact bus may be specified in the configuration line, like "isa0" or "isa1", otherwise the device would be searched for on all the ISA buses. "sensitive" is a resource requesting that this device must be probed before all non-sensitive devices. It is supported but does not seem to be used in any current driver. For legacy ISA devices in many cases the drivers are still able to detect the configuration parameters. But each device to be configured in the system must have a config line. If two devices of some type are installed in the system but there is only one configuration line for the corresponding driver, ie: device xxx0 at isa? then only one device will be configured. But for the devices supporting automatic identification by the means of Plug-n-Play or some proprietary protocol one configuration line is enough to configure all the devices in the system, like the one above or just simply: device xxx at isa? If a driver supports both auto-identified and legacy devices and both kinds are installed at once in one machine then it is enough to describe in the config file the legacy devices only. The auto-identified devices will be added automatically. When an ISA bus is auto-configured the events happen as follows: All the drivers' identify routines (including the PnP identify routine which identifies all the PnP devices) are called in random order. As they identify the devices they add them to the list on the ISA bus. Normally the drivers' identify routines associate their drivers with the new devices. The PnP identify routine does not know about the other drivers yet so it does not associate any with the new devices it adds. The PnP devices are put to sleep using the PnP protocol to prevent them from being probed as legacy devices. The probe routines of non-PnP devices marked as "sensitive" are called. If probe for a device went successfully, the attach routine is called for it. The probe and attach routines of all non-PNP devices are called likewise. The PnP devices are brought back from the sleep state and assigned the resources they request: I/O and memory address ranges, IRQs and DRQs, all of them not conflicting with the attached legacy devices. Then for each PnP device the probe routines of all the present ISA drivers are called. The first one that claims the device gets attached. It is possible that multiple drivers would claim the device with different priority, the highest-priority driver wins. The probe routines must call ISA_PNP_PROBE() to compare the actual PnP ID with the list of the IDs supported by the driver and if the ID is not in the table return failure. That means that absolutely every driver, even the ones not supporting any PnP devices must call ISA_PNP_PROBE(), at least with an empty PnP ID table to return failure on unknown PnP devices. The probe routine returns a positive value (the error code) on error, zero or negative value on success. The negative return values are used when a PnP device supports multiple interfaces. For example, an older compatibility interface and a newer advanced interface which are supported by different drivers. Then both drivers would detect the device. The driver which returns a higher value in the probe routine takes precedence (in other words, the driver returning 0 has highest precedence, returning -1 is next, returning -2 is after it and so on). In result the devices which support only the old interface will be handled by the old driver (which should return -1 from the probe routine) while the devices supporting the new interface as well will be handled by the new driver (which should return 0 from the probe routine). If multiple drivers return the same value then the one called first wins. So if a driver returns value 0 it may be sure that it won the priority arbitration. The device-specific identify routines can also assign not a driver but a class of drivers to the device. Then all the drivers in the class are probed for this device, like the case with PnP. This feature is not implemented in any existing driver and is not considered further in this document. Because the PnP devices are disabled when probing the legacy devices they will not be attached twice (once as legacy and once as PnP). But in case of device-dependent identify routines it is the responsibility of the driver to make sure that the same device will not be attached by the driver twice: once as legacy user-configured and once as auto-identified. Another practical consequence for the auto-identified devices (both PnP and device-specific) is that the flags can not be passed to them from the kernel configuration file. So they must either not use the flags at all or use the flags from the device unit 0 for all the auto-identified devices or use the sysctl interface instead of flags. Other unusual configurations may be accommodated by accessing the configuration resources directly with functions of families resource_query_*() and resource_*_value(). Their implementations are located in kern/subr_bus.h. The old IDE disk driver i386/isa/wd.c contains examples of such use. But the standard means of configuration must always be preferred. Leave parsing the configuration resources to the bus configuration code. Resources The information that a user enters into the kernel configuration file is processed and passed to the kernel as configuration resources. This information is parsed by the bus configuration code and transformed into a value of structure device_t and the bus resources associated with it. The drivers may access the configuration resources directly using functions resource_* for more complex cases of configuration. But generally it is not needed nor recommended, so this issue is not discussed further. The bus resources are associated with each device. They are identified by type and number within the type. For the ISA bus the following types are defined: SYS_RES_IRQ - interrupt number SYS_RES_DRQ - ISA DMA channel number SYS_RES_MEMORY - range of device memory mapped into the system memory space SYS_RES_IOPORT - range of device I/O registers The enumeration within types starts from 0, so if a device has two memory regions if would have resources of type SYS_RES_MEMORY numbered 0 and 1. The resource type has nothing to do with the C language type, all the resource values have the C language type "unsigned long" and must be cast as necessary. The resource numbers do not have to be contiguous although for ISA they normally would be. The permitted resource numbers for ISA devices are: IRQ: 0-1 DRQ: 0-1 MEMORY: 0-3 IOPORT: 0-7 All the resources are represented as ranges, with a start value and count. For IRQ and DRQ resources the count would be normally equal to 1. The values for memory refer to the physical addresses. Three types of activities can be performed on resources: set/get allocate/release activate/deactivate Setting sets the range used by the resource. Allocation reserves the requested range that no other driver would be able to reserve it (and checking that no other driver reserved this range already). Activation makes the resource accessible to the driver doing whatever is necessary for that (for example, for memory it would be mapping into the kernel virtual address space). The functions to manipulate resources are: int bus_set_resource(device_t dev, int type, int rid, u_long start, u_long count) Set a range for a resource. Returns 0 if successful, error code otherwise. Normally the only reason this function would return an error is value of type, rid, start or count out of permitted range. dev - driver's device type - type of resource, SYS_RES_* rid - resource number (ID) within type start, count - resource range int bus_get_resource(device_t dev, int type, int rid, u_long *startp, u_long *countp) Get the range of resource. Returns 0 if successful, error code if the resource is not defined yet. u_long bus_get_resource_start(device_t dev, int type, int rid) u_long bus_get_resource_count (device_t dev, int type, int rid) Convenience functions to get only the start or count. Return 0 in case of error, so if the resource start has 0 among the legitimate values it would be impossible to tell if the value is 0 or an error occurred. Luckily, no ISA resources for add-on drivers may have a start value equal 0. void bus_delete_resource(device_t dev, int type, int rid) Delete a resource, make it undefined. struct resource * bus_alloc_resource(device_t dev, int type, int *rid, u_long start, u_long end, u_long count, u_int flags) Allocate a resource as a range of count values not allocated by anyone else, somewhere between start and end. Alas, alignment is not supported. If the resource was not set yet it is automatically created. The special values of start 0 and end ~0 (all ones) means that the fixed values previously set by bus_set_resource() must be used instead: start and count as themselves and end=(start+count), in this case if the resource was not defined before then an error is returned. Although rid is passed by reference it is not set anywhere by the resource allocation code of the ISA bus. (The other buses may use a different approach and modify it). Flags are a bitmap, the flags interesting for the caller are: RF_ACTIVE - causes the resource to be automatically activated after allocation. RF_SHAREABLE - resource may be shared at the same time by multiple drivers. RF_TIMESHARE - resource may be time-shared by multiple drivers, i.e. allocated at the same time by many but activated only by one at any given moment of time. Returns 0 on error. The allocated values may be obtained from the returned handle using methods rhand_*(). int bus_release_resource(device_t dev, int type, int rid, struct resource *r) Release the resource, r is the handle returned by bus_alloc_resource(). Returns 0 on success, error code otherwise. int bus_activate_resource(device_t dev, int type, int rid, struct resource *r) int bus_deactivate_resource(device_t dev, int type, int rid, struct resource *r) Activate or deactivate resource. Return 0 on success, error code otherwise. If the resource is time-shared and currently activated by another driver then EBUSY is returned. int bus_setup_intr(device_t dev, struct resource *r, int flags, driver_intr_t *handler, void *arg, void **cookiep) int bus_teardown_intr(device_t dev, struct resource *r, void *cookie) Associate or de-associate the interrupt handler with a device. Return 0 on success, error code otherwise. r - the activated resource handler describing the IRQ flags - the interrupt priority level, one of: INTR_TYPE_TTY - terminals and other likewise character-type devices. To mask them use spltty(). (INTR_TYPE_TTY | INTR_TYPE_FAST) - terminal type devices with small input buffer, critical to the data loss on input (such as the old-fashioned serial ports). To mask them use spltty(). INTR_TYPE_BIO - block-type devices, except those on the CAM controllers. To mask them use splbio(). INTR_TYPE_CAM - CAM (Common Access Method) bus controllers. To mask them use splcam(). INTR_TYPE_NET - network interface controllers. To mask them use splimp(). INTR_TYPE_MISC - miscellaneous devices. There is no other way to mask them than by splhigh() which masks all interrupts. When an interrupt handler executes all the other interrupts matching its priority level will be masked. The only exception is the MISC level for which no other interrupts are masked and which is not masked by any other interrupt. handler - pointer to the handler function, the type driver_intr_t is defined as "void driver_intr_t(void *)" arg - the argument passed to the handler to identify this particular device. It is cast from void* to any real type by the handler. The old convention for the ISA interrupt handlers was to use the unit number as argument, the new (recommended) convention is using a pointer to the device softc structure. cookie[p] - the value received from setup() is used to identify the handler when passed to teardown() A number of methods is defined to operate on the resource handlers (struct resource *). Those of interest to the device driver writers are: u_long rman_get_start(r) u_long rman_get_end(r) Get the start and end of allocated resource range. void *rman_get_virtual(r) Get the virtual address of activated memory resource. Bus memory mapping In many cases data is exchanged between the driver and the device through the memory. Two variants are possible: (a) memory is located on the device card (b) memory is the main memory of computer In the case (a) the driver always copies the data back and forth between the on-card memory and the main memory as necessary. To map the on-card memory into the kernel virtual address space the physical address and length of the on-card memory must be defined as a SYS_RES_MEMORY resource. That resource can then be allocated and activated, and its virtual address obtained using rman_get_virtual(). The older drivers used the function pmap_mapdev() for this purpose, which should not be used directly any more. Now it is one of the internal steps of resource activation. Most of the ISA cards will have their memory configured for physical location somewhere in range 640KB-1MB. Some of the ISA cards require larger memory ranges which should be placed somewhere under 16MB (because of the 24-bit address limitation on the ISA bus). In that case if the machine has more memory than the start address of the device memory (in other words, they overlap) a memory hole must be configured at the address range used by devices. Many BIOSes allow to configure a memory hole of 1MB starting at 14MB or 15MB. FreeBSD can handle the memory holes properly if the BIOS reports them properly (old BIOSes may have this feature broken). In the case (b) just the address of the data is sent to the device, and the device uses DMA to actually access the data in the main memory. Two limitations are present: First, ISA cards can only access memory below 16MB. Second, the contiguous pages in virtual address space may not be contiguous in physical address space, so the device may have to do scatter/gather operations. The bus subsystem provides ready solutions for some of these problems, the rest has to be done by the drivers themselves. Two structures are used for DMA memory allocation, bus_dma_tag_t and bus_dmamap_t. Tag describes the properties required for the DMA memory. Map represents a memory block allocated according to these properties. Multiple maps may be associated with the same tag. Tags are organized into a tree-like hierarchy with inheritance of the properties. A child tag inherits all the requirements of its parent tag or may make them more strict but never more loose. Normally one top-level tag (with no parent) is created for each device unit. If multiple memory areas with different requirements are needed for each device then a tag for each of them may be created as a child of the parent tag. The tags can be used to create a map in two ways. First, a chunk of contiguous memory conformant with the tag requirements may be allocated (and later may be freed). This is normally used to allocate relatively long-living areas of memory for communication with the device. Loading of such memory into a map is trivial: it is always considered as one chunk in the appropriate physical memory range. Second, an arbitrary area of virtual memory may be loaded into a map. Each page of this memory will be checked for conformance to the map requirement. If it conforms then it is left at its original location. If it is not then a fresh conformant "bounce page" is allocated and used as intermediate storage. When writing the data from the non-conformant original pages they will be copied to their bounce pages first and then transferred from the bounce pages to the device. When reading the data would go from the device to the bounce pages and then copied to their non-conformant original pages. The process of copying between the original and bounce pages is called synchronization. This is normally used on per-transfer basis: buffer for each transfer would be loaded, transfer done and buffer unloaded. The functions working on the DMA memory are: int bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment, bus_size_t boundary, bus_addr_t lowaddr, bus_addr_t highaddr, bus_dma_filter_t *filter, void *filterarg, bus_size_t maxsize, int nsegments, bus_size_t maxsegsz, int flags, bus_dma_tag_t *dmat) Create a new tag. Returns 0 on success, the error code otherwise. parent - parent tag, or NULL to create a top-level tag alignment - required physical alignment of the memory area to be allocated for this tag. Use value 1 for "no specific alignment". Applies only to the future bus_dmamem_alloc() but not bus_dmamap_create() calls. boundary - physical address boundary that must not be crossed when allocating the memory. Use value 0 for "no boundary". Applies only to the future bus_dmamem_alloc() but not bus_dmamap_create() calls. Must be power of 2. If the memory is planned to be used in non-cascaded DMA mode (i.e. the DMA addresses will be supplied not by the device itself but by the ISA DMA controller) then the boundary must be no larger than 64KB (64*1024) due to the limitations of the DMA hardware. lowaddr, highaddr - the names are slighlty misleading; these values are used to limit the permitted range of physical addresses used to allocate the memory. The exact meaning varies depending on the planned future use: For bus_dmamem_alloc() all the addresses from 0 to lowaddr-1 are considered permitted, the higher ones are forbidden. For bus_dmamap_create() all the addresses outside the inclusive range [lowaddr; highaddr] are considered accessible. The addresses of pages inside the range are passed to the filter function which decides if they are accessible. If no filter function is supplied then all the range is considered unaccessible. For the ISA devices the normal values (with no filter function) are: lowaddr = BUS_SPACE_MAXADDR_24BIT highaddr = BUS_SPACE_MAXADDR filter, filterarg - the filter function and its argument. If NULL is passed for filter then the whole range [lowaddr, highaddr] is considered unaccessible when doing bus_dmamap_create(). Otherwise the physical address of each attempted page in range [lowaddr; highaddr] is passed to the filter function which decides if it is accessible. The prototype of the filter function is: int filterfunc(void *arg, bus_addr_t paddr) It must return 0 if the page is accessible, non-zero otherwise. maxsize - the maximal size of memory (in bytes) that may be allocated through this tag. In case it is difficult to estimate or could be arbitrarily big, the value for ISA devices would be BUS_SPACE_MAXSIZE_24BIT. nsegments - maximal number of scatter-gather segments supported by the device. If unrestricted then the value BUS_SPACE_UNRESTRICTED should be used. This value is recommended for the parent tags, the actual restrictions would then be specified for the descendant tags. Tags with nsegments equal to BUS_SPACE_UNRESTRICTED may not be used to actually load maps, they may be used only as parent tags. The practical limit for nsegments seems to be about 250-300, higher values will cause kernel stack overflow. But anyway the hardware normally can not support that many scatter-gather buffers. maxsegsz - maximal size of a scatter-gather segment supported by the device. The maximal value for ISA device would be BUS_SPACE_MAXSIZE_24BIT. flags - a bitmap of flags. The only interesting flags are: BUS_DMA_ALLOCNOW - requests to allocate all the potentially needed bounce pages when creating the tag BUS_DMA_ISA - mysterious flag used only on Alpha machines. It is not defined for the i386 machines. Probably it should be used by all the ISA drivers for Alpha machines but it looks like there are no such drivers yet. dmat - pointer to the storage for the new tag to be returned int bus_dma_tag_destroy(bus_dma_tag_t dmat) Destroy a tag. Returns 0 on success, the error code otherwise. dmat - the tag to be destroyed int bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags, bus_dmamap_t *mapp) Allocate an area of contiguous memory described by the tag. The size of memory to be allocated is tag's maxsize. Returns 0 on success, the error code otherwise. The result still has to be loaded by bus_dmamap_load() before used to get the physical address of the memory. dmat - the tag vaddr - pointer to the storage for the kernel virtual address of the allocated area to be returned. flags - a bitmap of flags. The only interesting flag is: BUS_DMA_NOWAIT - if the memory is not immediately available return the error. If this flag is not set then the routine is allowed to sleep waiting until the memory will become available. mapp - pointer to the storage for the new map to be returned void bus_dmamem_free(bus_dma_tag_t dmat, void *vaddr, bus_dmamap_t map) Free the memory allocated by bus_dmamem_alloc(). As of now freeing of the memory allocated with ISA restrictions is not implemented. Because of this the recommended model of use is to keep and re-use the allocated areas for as long as possible. Do not lightly free some area and then shortly allocate it again. That does not mean that bus_dmamem_free() should not be used at all: hopefully it will be properly implemented soon. dmat - the tag vaddr - the kernel virtual address of the memory map - the map of the memory (as returned from bus_dmamem_alloc()) int bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp) Create a map for the tag, to be used in bus_dmamap_load() later. Returns 0 on success, the error code otherwise. dmat - the tag flags - theoretically, a bit map of flags. But no flags are defined yet, so as of now it will be always 0. mapp - pointer to the storage for the new map to be returned int bus_dmamap_destroy(bus_dma_tag_t dmat, bus_dmamap_t map) Destroy a map. Returns 0 on success, the error code otherwise. dmat - the tag to which the map is associated map - the map to be destroyed int bus_dmamap_load(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf, bus_size_t buflen, bus_dmamap_callback_t *callback, void *callback_arg, int flags) Load a buffer into the map (the map must be previously created by bus_dmamap_create() or bus_dmamem_alloc()). All the pages of the buffer are checked for conformance to the tag requirements and for those not conformant the bounce pages are allocated. An array of physical segment descriptors is built and passed to the callback routine. This callback routine is then expected to handle it in some way. The number of bounce buffers in the system is limited, so if the bounce buffers are needed but not immediately available the request will be queued and the callback will be called when the bounce buffers will become available. Returns 0 if the callback was executed immediately or EINPROGRESS if the request was queued for future execution. In the latter case the synchronization with queued callback routine is the responsibility of the driver. dmat - the tag map - the map buf - kernel virtual address of the buffer buflen - length of the buffer callback, callback_arg - the callback function and its argument The prototype of callback function is: void callback(void *arg, bus_dma_segment_t *seg, int nseg, int error) arg - the same as callback_arg passed to bus_dmamap_load() seg - array of the segment descriptors nseg - number of descriptors in array error - indication of the segment number overflow: if it is set to EFBIG then the buffer did not fit into the maximal number of segments permitted by the tag. In this case only the permitted number of descriptors will be in the array. Handling of this situation is up to the driver: depending on the desired semantics it can either consider this an error or split the buffer in two and handle the second part separately Each entry in the segments array contains the fields: ds_addr - physical bus address of the segment ds_len - length of the segment void bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map) unload the map. dmat - tag map - loaded map void bus_dmamap_sync (bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op) Synchronise a loaded buffer with its bounce pages before and after physical transfer to or from device. This is the function that does all the necessary copying of data between the original buffer and its mapped version. The buffers must be synchronized both before and after doing the transfer. dmat - tag map - loaded map op - type of synchronization operation to perform: BUS_DMASYNC_PREREAD - before reading from device into buffer BUS_DMASYNC_POSTREAD - after reading from device into buffer BUS_DMASYNC_PREWRITE - before writing the buffer to device BUS_DMASYNC_POSTWRITE - after writing the buffer to device As of now PREREAD and POSTWRITE are null operations but that may change in the future, so they must not be ignored in the driver. Synchronization is not needed for the memory obtained from bus_dmamem_alloc(). Before calling the callback function from bus_dmamap_load() the segment array is stored in the stack. And it gets pre-allocated for the maximal number of segments allowed by the tag. Because of this the practical limit for the number of segments on i386 architecture is about 250-300 (the kernel stack is 4KB minus the size of the user structure, size of a segment array entry is 8 bytes, and some space must be left). Because the array is allocated based on the maximal number this value must not be set higher than really needed. Fortunately, for most of hardware the maximal supported number of segments is much lower. But if the driver wants to handle buffers with a very large number of scatter-gather segments it should do that in portions: load part of the buffer, transfer it to the device, load next part of the buffer, and so on. Another practical consequence is that the number of segments may limit the size of the buffer. If all the pages in the buffer happen to be physically non-contiguous then the maximal supported buffer size for that fragmented case would be (nsegments * page_size). For example, if a maximal number of 10 segments is supported then on i386 maximal guaranteed supported buffer size would be 40K. If a higher size is desired then special tricks should be used in the driver. If the hardware does not support scatter-gather at all or the driver wants to support some buffer size even if it is heavily fragmented then the solution is to allocate a contiguous buffer in the driver and use it as intermediate storage if the original buffer does not fit. Below are the typical call sequences when using a map depend on the use of the map. The characters -> are used to show the flow of time. For a buffer which stays practically fixed during all the time between attachment and detachment of a device: bus_dmamem_alloc -> bus_dmamap_load -> ...use buffer... -> -> bus_dmamap_unload -> bus_dmamem_free For a buffer that changes frequently and is passed from outside the driver: bus_dmamap_create -> -> bus_dmamap_load -> bus_dmamap_sync(PRE...) -> do transfer -> -> bus_dmamap_sync(POST...) -> bus_dmamap_unload -> ... -> bus_dmamap_load -> bus_dmamap_sync(PRE...) -> do transfer -> -> bus_dmamap_sync(POST...) -> bus_dmamap_unload -> -> bus_dmamap_destroy When loading a map created by bus_dmamem_alloc() the passed address and size of the buffer must be the same as used in bus_dmamem_alloc(). In this case it is guaranteed that the whole buffer will be mapped as one segment (so the callback may be based on this assumption) and the request will be executed immediately (EINPROGRESS will never be returned). All the callback needs to do in this case is to save the physical address. A typical example would be: static void alloc_callback(void *arg, bus_dma_segment_t *seg, int nseg, int error) { *(bus_addr_t *)arg = seg[0].ds_addr; } ... int error; struct somedata { .... }; struct somedata *vsomedata; /* virtual address */ bus_addr_t psomedata; /* physical bus-relative address */ bus_dma_tag_t tag_somedata; bus_dmamap_t map_somedata; ... error=bus_dma_tag_create(parent_tag, alignment, boundary, lowaddr, highaddr, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ sizeof(struct somedata), /*nsegments*/ 1, /*maxsegsz*/ sizeof(struct somedata), /*flags*/ 0, &tag_somedata); if(error) return error; error = bus_dmamem_alloc(tag_somedata, &vsomedata, /* flags*/ 0, &map_somedata); if(error) return error; bus_dmamap_load(tag_somedata, map_somedata, (void *)vsomedata, sizeof (struct somedata), alloc_callback, (void *) &psomedata, /*flags*/0); Looks a bit long and complicated but that is the way to do it. The practical consequence is: if multiple memory areas are allocated always together it would be a really good idea to combine them all into one structure and allocate as one (if the alignment and boundary limitations permit). When loading an arbitrary buffer into the map created by bus_dmamap_create() special measures must be taken to synchronize with the callback in case it would be delayed. The code would look like: { int s; int error; s = splsoftvm(); error = bus_dmamap_load( dmat, dmamap, buffer_ptr, buffer_len, callback, /*callback_arg*/ buffer_descriptor, /*flags*/0); if (error == EINPROGRESS) { /* * Do whatever is needed to ensure synchronization * with callback. Callback is guaranteed not to be started * until we do splx() or tsleep(). */ } splx(s); } Two possible approaches for the processing of requests are: 1. If requests are completed by marking them explicitly as done (such as the CAM requests) then it would be simpler to put all the further processing into the callback driver which would mark the request when it is done. Then not much extra synchronization is needed. For the flow control reasons it may be a good idea to freeze the request queue until this request gets completed. 2. If requests are completed when the function returns (such as classic read or write requests on character devices) then a synchronization flag should be set in the buffer descriptor and tsleep() called. Later when the callback gets called it will do its processing and check this synchronization flag. If it is set then the callback should issue a wakeup. In this approach the callback function could either do all the needed processing (just like the previous case) or simply save the segments array in the buffer descriptor. Then after callback completes the calling function could use this saved segments array and do all the processing. DMA The Direct Memory Access (DMA) is implemented in the ISA bus through the DMA controller (actually, two of them but that is an irrelevant detail). To make the early ISA devices simple and cheap the logic of the bus control and address generation was concentrated in the DMA controller. Fortunately, FreeBSD provides a set of functions that mostly hide the annoying details of the DMA controller from the device drivers. The simplest case is for the fairly intelligent devices. Like the bus master devices on PCI they can generate the bus cycles and memory addresses all by themselves. The only thing they really need from the DMA controller is bus arbitration. So for this purpose they pretend to be cascaded slave DMA controllers. And the only thing needed from the system DMA controller is to enable the cascaded mode on a DMA channel by calling the following function when attaching the driver: void isa_dmacascade(int channel_number) All the further activity is done by programming the device. When detaching the driver no DMA-related functions need to be called. For the simpler devices things get more complicated. The functions used are: int isa_dma_acquire(int chanel_number) Reserve a DMA channel. Returns 0 on success or EBUSY if the channel was already reserved by this or a different driver. Most of the ISA devices are not able to share DMA channels anyway, so normally this function is called when attaching a device. This reservation was made redundant by the modern interface of bus resources but still must be used in addition to the latter. If not used then later, other DMA routines will panic. int isa_dma_release(int chanel_number) Release a previously reserved DMA channel. No transfers must be in progress when the channel is released (as well as the device must not try to initiate transfer after the channel is released). void isa_dmainit(int chan, u_int bouncebufsize) Allocate a bounce buffer for use with the specified channel. The requested size of the buffer can not exceed 64KB. This bounce buffer will be automatically used later if a transfer buffer happens to be not physically contiguous or outside of the memory accessible by the ISA bus or crossing the 64KB boundary. If the transfers will be always done from buffers which conform to these conditions (such as those allocated by bus_dmamem_alloc() with proper limitations) then isa_dmainit() does not have to be called. But it is quite convenient to transfer arbitrary data using the DMA controller. The bounce buffer will automatically care of the scatter-gather issues. chan - channel number bouncebufsize - size of the bounce buffer in bytes void isa_dmastart(int flags, caddr_t addr, u_int nbytes, int chan) Prepare to start a DMA transfer. This function must be called to set up the DMA controller before actually starting transfer on the device. It checks that the buffer is contiguous and falls into the ISA memory range, if not then the bounce buffer is automatically used. If bounce buffer is required but not set up by isa_dmainit() or too small for the requested transfer size then the system will panic. In case of a write request with bounce buffer the data will be automatically copied to the bounce buffer. flags - a bitmask determining the type of operation to be done. The direction bits B_READ and B_WRITE are mutually exclusive. B_READ - read from the ISA bus into memory B_WRITE - write from the memory to the ISA bus B_RAW - if set then the DMA controller will remember the buffer and after the end of transfer will automatically re-initialize itself to repeat transfer of the same buffer again (of course, the driver may change the data in the buffer before initiating another transfer in the device). If not set then the parameters will work only for one transfer, and isa_dmastart() will have to be called again before initiating the next transfer. Using B_RAW makes sense only if the bounce buffer is not used. addr - virtual address of the buffer nbytes - length of the buffer. Must be less or equal to 64KB. Length of 0 is not allowed: the DMA controller will understand it as 64KB while the kernel code will understand it as 0 and that would cause unpredictable effects. For channels number 4 and higher the length must be even because these channels transfer 2 bytes at a time. In case of an odd length the last byte will not be transferred. chan - channel number void isa_dmadone(int flags, caddr_t addr, int nbytes, int chan) Synchronize the memory after device reports that transfer is done. If that was a read operation with a bounce buffer then the data will be copied from the bounce buffer to the original buffer. Arguments are the same as for isa_dmastart(). Flag B_RAW is permitted but it does not affect isa_dmadone() in any way. int isa_dmastatus(int channel_number) Returns the number of bytes left in the current transfer to be transferred. In case the flag B_READ was set in isa_dmastart() the number returned will never be equal to zero. At the end of transfer it will be automatically reset back to the length of buffer. The normal use is to check the number of bytes left after the device signals that the transfer is completed. If the number of bytes is not 0 then probably something went wrong with that transfer. int isa_dmastop(int channel_number) Aborts the current transfer and returns the number of bytes left untransferred. xxx_isa_probe This function probes if a device is present. If the driver supports auto-detection of some part of device configuration (such as interrupt vector or memory address) this auto-detection must be done in this routine. As for any other bus, if the device cannot be detected or is detected but failed the self-test or some other problem happened then it returns a positive value of error. The value ENXIO must be returned if the device is not present. Other error values may mean other conditions. Zero or negative values mean success. Most of the drivers return zero as success. The negative return values are used when a PnP device supports multiple interfaces. For example, an older compatibility interface and a newer advanced interface which are supported by different drivers. Then both drivers would detect the device. The driver which returns a higher value in the probe routine takes precedence (in other words, the driver returning 0 has highest precedence, one returning -1 is next, one returning -2 is after it and so on). In result the devices which support only the old interface will be handled by the old driver (which should return -1 from the probe routine) while the devices supporting the new interface as well will be handled by the new driver (which should return 0 from the probe routine). The device descriptor struct xxx_softc is allocated by the system before calling the probe routine. If the probe routine returns an error the descriptor will be automatically deallocated by the system. So if a probing error occurs the driver must make sure that all the resources it used during probe are deallocated and that nothing keeps the descriptor from being safely deallocated. If the probe completes successfully the descriptor will be preserved by the system and later passed to the routine xxx_isa_attach(). If a driver returns a negative value it can not be sure that it will have the highest priority and its attach routine will be called. So in this case it also must release all the resources before returning and if necessary allocate them again in the attach routine. When xxx_isa_probe() returns 0 releasing the resources before returning is also a good idea, a well-behaved driver should do so. But in case if there is some problem with releasing the resources the driver is allowed to keep resources between returning 0 from the probe routine and execution of the attach routine. A typical probe routine starts with getting the device descriptor and unit: struct xxx_softc *sc = device_get_softc(dev); int unit = device_get_unit(dev); int pnperror; int error = 0; sc->dev = dev; /* link it back */ sc->unit = unit; Then check for the PnP devices. The check is carried out by a table containing the list of PnP IDs supported by this driver and human-readable descriptions of the device models corresponding to these IDs. pnperror=ISA_PNP_PROBE(device_get_parent(dev), dev, xxx_pnp_ids); if(pnperror == ENXIO) return ENXIO; The logic of ISA_PNP_PROBE is the following: If this card (device unit) was not detected as PnP then ENOENT will be returned. If it was detected as PnP but its detected ID does not match any of the IDs in the table then ENXIO is returned. Finally, if it has PnP support and it matches on of the IDs in the table, 0 is returned and the appropriate description from the table is set by device_set_desc(). If a driver supports only PnP devices then the condition would look like: if(pnperror != 0) return pnperror; No special treatment is required for the drivers which do not support PnP because they pass an empty PnP ID table and will always get ENXIO if called on a PnP card. The probe routine normally needs at least some minimal set of resources, such as I/O port number to find the card and probe it. Depending on the hardware the driver may be able to discover the other necessary resources automatically. The PnP devices have all the resources pre-set by the PnP subsystem, so the driver does not need to discover them by itself. Typically the minimal information required to get access to the device is the I/O port number. Then some devices allow to get the rest of information from the device configuration registers (though not all devices do that). So first we try to get the port start value: sc->port0 = bus_get_resource_start(dev, SYS_RES_IOPORT, 0 /*rid*/); if(sc->port0 == 0) return ENXIO; The base port address is saved in the structure softc for future use. If it will be used very often then calling the resource function each time would be prohibitively slow. If we do not get a port we just return an error. Some device drivers can instead be clever and try to probe all the possible ports, like this: /* table of all possible base I/O port addresses for this device */ static struct xxx_allports { u_short port; /* port address */ short used; /* flag: if this port is already used by some unit */ } xxx_allports = { { 0x300, 0 }, { 0x320, 0 }, { 0x340, 0 }, { 0, 0 } /* end of table */ }; ... int port, i; ... port = bus_get_resource_start(dev, SYS_RES_IOPORT, 0 /*rid*/); if(port !=0 ) { for(i=0; xxx_allports[i].port!=0; i++) { if(xxx_allports[i].used || xxx_allports[i].port != port) continue; /* found it */ xxx_allports[i].used = 1; /* do probe on a known port */ return xxx_really_probe(dev, port); } return ENXIO; /* port is unknown or already used */ } /* we get here only if we need to guess the port */ for(i=0; xxx_allports[i].port!=0; i++) { if(xxx_allports[i].used) continue; /* mark as used - even if we find nothing at this port * at least we won't probe it in future */ xxx_allports[i].used = 1; error = xxx_really_probe(dev, xxx_allports[i].port); if(error == 0) /* found a device at that port */ return 0; } /* probed all possible addresses, none worked */ return ENXIO; Of course, normally the driver's identify() routine should be used for such things. But there may be one valid reason why it may be better to be done in probe(): if this probe would drive some other sensitive device crazy. The probe routines are ordered with consideration of the "sensitive" flag: the sensitive devices get probed first and the rest of devices later. But the identify() routines are called before any probes, so they show no respect to the sensitive devices and may upset them. Now, after we got the starting port we need to set the port count (except for PnP devices) because the kernel does not have this information in the configuration file. if(pnperror /* only for non-PnP devices */ && bus_set_resource(dev, SYS_RES_IOPORT, 0, sc->port0, XXX_PORT_COUNT)<0) return ENXIO; Finally allocate and activate a piece of port address space (special values of start and end mean "use those we set by bus_set_resource()"): sc->port0_rid = 0; sc->port0_r = bus_alloc_resource(dev, SYS_RES_IOPORT, &sc->port0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->port0_r == NULL) return ENXIO; Now having access to the port-mapped registers we can poke the device in some way and check if it reacts like it is expected to. If it does not then there is probably some other device or no device at all at this address. Normally drivers do not set up the interrupt handlers until the attach routine. Instead they do probes in the polling mode using the DELAY() function for timeout. The probe routine must never hang forever, all the waits for the device must be done with timeouts. If the device does not respond within the time it is probably broken or misconfigured and the driver must return error. When determining the timeout interval give the device some extra time to be on the safe side: although DELAY() is supposed to delay for the same amount of time on any machine it has some margin of error, depending on the exact CPU. If the probe routine really wants to check that the interrupts really work it may configure and probe the interrupts too. But that is not recommended. /* implemented in some very device-specific way */ if(error = xxx_probe_ports(sc)) goto bad; /* will deallocate the resources before returning */ The function xxx_probe_ports() may also set the device description depending on the exact model of device it discovers. But if there is only one supported device model this can be as well done in a hardcoded way. Of course, for the PnP devices the PnP support sets the description from the table automatically. if(pnperror) device_set_desc(dev, "Our device model 1234"); Then the probe routine should either discover the ranges of all the resources by reading the device configuration registers or make sure that they were set explicitly by the user. We will consider it with an example of on-board memory. The probe routine should be as non-intrusive as possible, so allocation and check of functionality of the rest of resources (besides the ports) would be better left to the attach routine. The memory address may be specified in the kernel configuration file or on some devices it may be pre-configured in non-volatile configuration registers. If both sources are available and different, which one should be used? Probably if the user bothered to set the address explicitly in the kernel configuration file they know what they are doing and this one should take precedence. An example of implementation could be: /* try to find out the config address first */ sc->mem0_p = bus_get_resource_start(dev, SYS_RES_MEMORY, 0 /*rid*/); if(sc->mem0_p == 0) { /* nope, not specified by user */ sc->mem0_p = xxx_read_mem0_from_device_config(sc); if(sc->mem0_p == 0) /* can't get it from device config registers either */ goto bad; } else { if(xxx_set_mem0_address_on_device(sc) < 0) goto bad; /* device does not support that address */ } /* just like the port, set the memory size, * for some devices the memory size would not be constant * but should be read from the device configuration registers instead * to accommodate different models of devices. Another option would * be to let the user set the memory size as "msize" configuration * resource which will be automatically handled by the ISA bus. */ if(pnperror) { /* only for non-PnP devices */ sc->mem0_size = bus_get_resource_count(dev, SYS_RES_MEMORY, 0 /*rid*/); if(sc->mem0_size == 0) /* not specified by user */ sc->mem0_size = xxx_read_mem0_size_from_device_config(sc); if(sc->mem0_size == 0) { /* suppose this is a very old model of device without * auto-configuration features and the user gave no preference, * so assume the minimalistic case * (of course, the real value will vary with the driver) */ sc->mem0_size = 8*1024; } if(xxx_set_mem0_size_on_device(sc) < 0) goto bad; /* device does not support that size */ if(bus_set_resource(dev, SYS_RES_MEMORY, /*rid*/0, sc->mem0_p, sc->mem0_size)<0) goto bad; } else { sc->mem0_size = bus_get_resource_count(dev, SYS_RES_MEMORY, 0 /*rid*/); } Resources for IRQ and DRQ are easy to check by analogy. If all went well then release all the resources and return success. xxx_free_resources(sc); return 0; Finally, handle the troublesome situations. All the resources should be deallocated before returning. We make use of the fact that before the structure softc is passed to us it gets zeroed out, so we can find out if some resource was allocated: then its descriptor is non-zero. bad: xxx_free_resources(sc); if(error) return error; else /* exact error is unknown */ return ENXIO; That would be all for the probe routine. Freeing of resources is done from multiple places, so it is moved to a function which may look like: static void xxx_free_resources(sc) struct xxx_softc *sc; { /* check every resource and free if not zero */ /* interrupt handler */ if(sc->intr_r) { bus_teardown_intr(sc->dev, sc->intr_r, sc->intr_cookie); bus_release_resource(sc->dev, SYS_RES_IRQ, sc->intr_rid, sc->intr_r); sc->intr_r = 0; } /* all kinds of memory maps we could have allocated */ if(sc->data_p) { bus_dmamap_unload(sc->data_tag, sc->data_map); sc->data_p = 0; } if(sc->data) { /* sc->data_map may be legitimately equal to 0 */ /* the map will also be freed */ bus_dmamem_free(sc->data_tag, sc->data, sc->data_map); sc->data = 0; } if(sc->data_tag) { bus_dma_tag_destroy(sc->data_tag); sc->data_tag = 0; } ... free other maps and tags if we have them ... if(sc->parent_tag) { bus_dma_tag_destroy(sc->parent_tag); sc->parent_tag = 0; } /* release all the bus resources */ if(sc->mem0_r) { bus_release_resource(sc->dev, SYS_RES_MEMORY, sc->mem0_rid, sc->mem0_r); sc->mem0_r = 0; } ... if(sc->port0_r) { bus_release_resource(sc->dev, SYS_RES_IOPORT, sc->port0_rid, sc->port0_r); sc->port0_r = 0; } } xxx_isa_attach The attach routine actually connects the driver to the system if the probe routine returned success and the system had chosen to attach that driver. If the probe routine returned 0 then the attach routine may expect to receive the device structure softc intact, as it was set by the probe routine. Also if the probe routine returns 0 it may expect that the attach routine for this device shall be called at some point in the future. If the probe routine returns a negative value then the driver may make none of these assumptions. The attach routine returns 0 if it completed successfully or error code otherwise. The attach routine starts just like the probe routine, with getting some frequently used data into more accessible variables. struct xxx_softc *sc = device_get_softc(dev); int unit = device_get_unit(dev); int error = 0; Then allocate and activate all the necessary resources. Because normally the port range will be released before returning from probe, it has to be allocated again. We expect that the probe routine had properly set all the resource ranges, as well as saved them in the structure softc. If the probe routine had left some resource allocated then it does not need to be allocated again (which would be considered an error). sc->port0_rid = 0; sc->port0_r = bus_alloc_resource(dev, SYS_RES_IOPORT, &sc->port0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->port0_r == NULL) return ENXIO; /* on-board memory */ sc->mem0_rid = 0; sc->mem0_r = bus_alloc_resource(dev, SYS_RES_MEMORY, &sc->mem0_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->mem0_r == NULL) goto bad; /* get its virtual address */ sc->mem0_v = rman_get_virtual(sc->mem0_r); The DMA request channel (DRQ) is allocated likewise. To initialize it use functions of the isa_dma*() family. For example: isa_dmacascade(sc->drq0); The interrupt request line (IRQ) is a bit special. Besides allocation the driver's interrupt handler should be associated with it. Historically in the old ISA drivers the argument passed by the system to the interrupt handler was the device unit number. But in modern drivers the convention suggests passing the pointer to structure softc. The important reason is that when the structures softc are allocated dynamically then getting the unit number from softc is easy while getting softc from unit number is difficult. Also this convention makes the drivers for different buses look more uniform and allows them to share the code: each bus gets its own probe, attach, detach and other bus-specific routines while the bulk of the driver code may be shared among them. sc->intr_rid = 0; sc->intr_r = bus_alloc_resource(dev, SYS_RES_MEMORY, &sc->intr_rid, /*start*/ 0, /*end*/ ~0, /*count*/ 0, RF_ACTIVE); if(sc->intr_r == NULL) goto bad; /* * XXX_INTR_TYPE is supposed to be defined depending on the type of * the driver, for example as INTR_TYPE_CAM for a CAM driver */ error = bus_setup_intr(dev, sc->intr_r, XXX_INTR_TYPE, (driver_intr_t *) xxx_intr, (void *) sc, &sc->intr_cookie); if(error) goto bad; If the device needs to make DMA to the main memory then this memory should be allocated like described before: error=bus_dma_tag_create(NULL, /*alignment*/ 4, /*boundary*/ 0, /*lowaddr*/ BUS_SPACE_MAXADDR_24BIT, /*highaddr*/ BUS_SPACE_MAXADDR, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ BUS_SPACE_MAXSIZE_24BIT, /*nsegments*/ BUS_SPACE_UNRESTRICTED, /*maxsegsz*/ BUS_SPACE_MAXSIZE_24BIT, /*flags*/ 0, &sc->parent_tag); if(error) goto bad; /* many things get inherited from the parent tag * sc->data is supposed to point to the structure with the shared data, * for example for a ring buffer it could be: * struct { * u_short rd_pos; * u_short wr_pos; * char bf[XXX_RING_BUFFER_SIZE] * } *data; */ error=bus_dma_tag_create(sc->parent_tag, 1, 0, BUS_SPACE_MAXADDR, 0, /*filter*/ NULL, /*filterarg*/ NULL, /*maxsize*/ sizeof(* sc->data), /*nsegments*/ 1, /*maxsegsz*/ sizeof(* sc->data), /*flags*/ 0, &sc->data_tag); if(error) goto bad; error = bus_dmamem_alloc(sc->data_tag, &sc->data, /* flags*/ 0, &sc->data_map); if(error) goto bad; /* xxx_alloc_callback() just saves the physical address at * the pointer passed as its argument, in this case &sc->data_p. * See details in the section on bus memory mapping. * It can be implemented like: * * static void * xxx_alloc_callback(void *arg, bus_dma_segment_t *seg, * int nseg, int error) * { * *(bus_addr_t *)arg = seg[0].ds_addr; * } */ bus_dmamap_load(sc->data_tag, sc->data_map, (void *)sc->data, sizeof (* sc->data), xxx_alloc_callback, (void *) &sc->data_p, /*flags*/0); After all the necessary resources are allocated the device should be initialized. The initialization may include testing that all the expected features are functional. if(xxx_initialize(sc) < 0) goto bad; The bus subsystem will automatically print on the console the device description set by probe. But if the driver wants to print some extra information about the device it may do so, for example: device_printf(dev, "has on-card FIFO buffer of %d bytes\n", sc->fifosize); If the initialization routine experiences any problems then printing messages about them before returning error is also recommended. The final step of the attach routine is attaching the device to its functional subsystem in the kernel. The exact way to do it depends on the type of the driver: a character device, a block device, a network device, a CAM SCSI bus device and so on. If all went well then return success. error = xxx_attach_subsystem(sc); if(error) goto bad; return 0; Finally, handle the troublesome situations. All the resources should be deallocated before returning an error. We make use of the fact that before the structure softc is passed to us it gets zeroed out, so we can find out if some resource was allocated: then its descriptor is non-zero. bad: xxx_free_resources(sc); if(error) return error; else /* exact error is unknown */ return ENXIO; That would be all for the attach routine. xxx_isa_detach If this function is present in the driver and the driver is compiled as a loadable module then the driver gets the ability to be unloaded. This is an important feature if the hardware supports hot plug. But the ISA bus does not support hot plug, so this feature is not particularly important for the ISA devices. The ability to unload a driver may be useful when debugging it, but in many cases installation of the new version of the driver would be required only after the old version somehow wedges the system and reboot will be needed anyway, so the efforts spent on writing the detach routine may not be worth it. Another argument is that unloading would allow upgrading the drivers on a production machine seems to be mostly theoretical. Installing a new version of a driver is a dangerous operation which should never be performed on a production machine (and which is not permitted when the system is running in secure mode). Still the detach routine may be provided for the sake of completeness. The detach routine returns 0 if the driver was successfully detached or the error code otherwise. The logic of detach is a mirror of the attach. The first thing to do is to detach the driver from its kernel subsystem. If the device is currently open then the driver has two choices: refuse to be detached or forcibly close and proceed with detach. The choice used depends on the ability of the particular kernel subsystem to do a forced close and on the preferences of the driver's author. Generally the forced close seems to be the preferred alternative. struct xxx_softc *sc = device_get_softc(dev); int error; error = xxx_detach_subsystem(sc); if(error) return error; Next the driver may want to reset the hardware to some consistent state. That includes stopping any ongoing transfers, disabling the DMA channels and interrupts to avoid memory corruption by the device. For most of the drivers this is exactly what the shutdown routine does, so if it is included in the driver we can as well just call it. xxx_isa_shutdown(dev); And finally release all the resources and return success. xxx_free_resources(sc); return 0; xxx_isa_shutdown This routine is called when the system is about to be shut down. It is expected to bring the hardware to some consistent state. For most of the ISA devices no special action is required, so the function is not really necessary because the device will be re-initialized on reboot anyway. But some devices have to be shut down with a special procedure, to make sure that they will be properly detected after soft reboot (this is especially true for many devices with proprietary identification protocols). In any case disabling DMA and interrupts in the device registers and stopping any ongoing transfers is a good idea. The exact action depends on the hardware, so we do not consider it here in any details. xxx_intr The interrupt handler is called when an interrupt is received which may be from this particular device. The ISA bus does not support interrupt sharing (except some special cases) so in practice if the interrupt handler is called then the interrupt almost for sure came from its device. Still the interrupt handler must poll the device registers and make sure that the interrupt was generated by its device. If not it should just return. The old convention for the ISA drivers was getting the device unit number as an argument. It is obsolete, and the new drivers receive whatever argument was specified for them in the attach routine when calling bus_setup_intr(). By the new convention it should be the pointer to the structure softc. So the interrupt handler commonly starts as: static void xxx_intr(struct xxx_softc *sc) { It runs at the interrupt priority level specified by the interrupt type parameter of bus_setup_intr(). That means that all the other interrupts of the same type as well as all the software interrupts are disabled. To avoid races it is commonly written as a loop: while(xxx_interrupt_pending(sc)) { xxx_process_interrupt(sc); xxx_acknowledge_interrupt(sc); } The interrupt handler has to acknowledge interrupt to the device only but not to the interrupt controller, the system takes care of the latter. diff --git a/en_US.ISO8859-1/books/developers-handbook/jail/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/jail/chapter.sgml index df99e3e2a4..47f6523a78 100644 --- a/en_US.ISO8859-1/books/developers-handbook/jail/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/jail/chapter.sgml @@ -1,611 +1,611 @@ Evan Sarmiento
evms@cs.bu.edu
 2001 Evan Sarmiento The Jail Subsystem On most UNIX systems, root has omnipotent power. This promotes insecurity. If an attacker were to gain root on a system, he would have every function at his fingertips. In FreeBSD there are sysctls which dilute the power of root, in order to minimize the damage caused by an attacker. Specifically, one of these functions is called secure levels. Similarly, another function which is present from FreeBSD 4.0 and onward, is a utility called &man.jail.8;. Jail chroots an environment and sets certain restrictions on processes which are forked from within. For example, a jailed process cannot affect processes outside of the jail, utilize certain system calls, or inflict any damage on the main computer. Jail is becoming the new security model. People are running potentially vulnerable servers such as Apache, BIND, and sendmail within jails, so that if an attacker gains root within the Jail, it is only an annoyance, and not a devastation. This article focuses on the internals (source code) of Jail and Jail NG. It will also suggest improvements upon the jail code base which are already being worked on. If you are looking for a how-to on setting up a Jail, I suggest you look at my other article in Sys Admin Magazine, May 2001, entitled "Securing FreeBSD using Jail." Architecture Jail consists of two realms: the user-space program, jail, and the code implemented within the kernel: the jail() system call and associated restrictions. I will be discussing the user-space program and then how jail is implemented within the kernel. Userland code The source for the user-land jail is located in - /usr/src/usr.sbin/jail , consisting of + /usr/src/usr.sbin/jail, consisting of one file, jail.c. The program takes these arguments: the path of the jail, hostname, ip address, and the command to be executed. Data Structures In jail.c, the first thing I would note is the declaration of an important structure struct jail j; which was included from /usr/include/sys/jail.h . The definition of the jail structure is: /usr/include/sys/jail.h: struct jail { u_int32_t version; char *path; char *hostname; u_int32_t ip_number; }; As you can see, there is an entry for each of the arguments passed to the jail program, and indeed, they are set during it's execution. /usr/src/usr.sbin/jail.c j.version = 0; j.path = argv[1]; j.hostname = argv[2]; Networking One of the arguments passed to the Jail program is an IP address with which the jail can be accessed over the network. Jail translates the ip address given into network byte order and then stores it in j (the jail structure). /usr/src/usr.sbin/jail/jail.c: struct in.addr in; ... i = inet.aton(argv[3], ); ... j.ip.number = ntohl(in.s.addr); The inet_aton3 function "interprets the specified character string as an Internet address, placing the address into the structure provided." The ip number node in the jail structure is set only when the ip address placed onto the in structure by inet aton is translated into network byte order by ntohl(). Jailing The Process Finally, the userland program jails the process, and executes the command specified. Jail now becomes an imprisoned process itself and forks a child process which then executes the command given using &man.execv.3; /usr/src/sys/usr.sbin/jail/jail.c i = jail(); ... i = execv(argv[4], argv + 4); As you can see, the jail function is being called, and its argument is the jail structure which has been filled with the arguments given to the program. Finally, the program you specify is executed. I will now discuss how Jail is implemented within the kernel. Kernel Space We will now be looking at the file /usr/src/sys/kern/kern_jail.c. This is the file where the jail system call, appropriate sysctls, and networking functions are defined. sysctls In kern_jail.c, the following sysctls are defined: /usr/src/sys/kern/kern_jail.c: int jail_set_hostname_allowed = 1; SYSCTL_INT(_jail, OID_AUTO, set_hostname_allowed, CTLFLAG_RW, _set_hostname_allowed, 0, "Processes in jail can set their hostnames"); int jail_socket_unixiproute_only = 1; SYSCTL_INT(_jail, OID_AUTO, socket_unixiproute_only, CTLFLAG_RW, _socket_unixiproute_only, 0, "Processes in jail are limited to creating UNIX/IPv4/route sockets only "); int jail_sysvipc_allowed = 0; SYSCTL_INT(_jail, OID_AUTO, sysvipc_allowed, CTLFLAG_RW, _sysvipc_allowed, 0, "Processes in jail can use System V IPC primitives"); Each of these sysctls can be accessed by the user through the sysctl program. Throughout the kernel, these specific sysctls are recognized by their name. For example, the name of the first sysctl is jail.set.hostname.allowed. &man.jail.2; system call Like all system calls, the &man.jail.2; system call takes two arguments, struct proc *p and struct jail_args *uap. p is a pointer to a proc structure which describes the calling process. In this context, uap is a pointer to a structure which specifies the arguments given to &man.jail.2; from the userland program jail.c. When I described the userland program before, you saw that the &man.jail.2; system call was given a jail structure as its own argument. /usr/src/sys/kern/kern_jail.c: int jail(p, uap) struct proc *p; struct jail_args /* { syscallarg(struct jail *) jail; } */ *uap; Therefore, uap->jail would access the jail structure which was passed to the system call. Next, the system call copies the jail structure into kernel space using the copyin() function. copyin() takes three arguments: the data which is to be copied into kernel space, uap->jail, where to store it, j and the size of the storage. The jail structure uap->jail is copied into kernel space and stored in another jail structure, j. /usr/src/sys/kern/kern_jail.c: error = copyin(uap->jail, , sizeof j); There is another important structure defined in jail.h. It is the prison structure (pr). The prison structure is used exclusively within kernel space. The &man.jail.2; system call copies everything from the jail structure onto the prison structure. Here is the definition of the prison structure. /usr/include/sys/jail.h: struct prison { int pr_ref; char pr_host[MAXHOSTNAMELEN]; u_int32_t pr_ip; void *pr_linux; }; The jail() system call then allocates memory for a pointer to a prison structure and copies data between the two structures. /usr/src/sys/kern/kern_jail.c: MALLOC(pr, struct prison *, sizeof *pr , M_PRISON, M_WAITOK); bzero((caddr_t)pr, sizeof *pr); error = copyinstr(j.hostname, pr_host]]>, sizeof pr->pr_host, 0); if (error) goto bail; Finally, the jail system call chroots the path specified. The chroot function is given two arguments. The first is p, which represents the calling process, the second is a pointer to the structure chroot args. The structure chroot args contains the path which is to be chrooted. As you can see, the path specified in the jail structure is copied to the chroot args structure and used. /usr/src/sys/kern/kern_jail.c: ca.path = j.path; error = chroot(p, ); These next three lines in the source are very important, as they specify how the kernel recognizes a process as jailed. Each process on a Unix system is described by its own proc structure. You can see the whole proc structure in /usr/include/sys/proc.h. For example, the p argument in any system call is actually a pointer to that process' proc structure, as stated before. The proc structure contains nodes which can describe the owner's identity (p_cred), the process resource limits (p_limit), and so on. In the definition of the process structure, there is a pointer to a prison structure. (p_prison). /usr/include/sys/proc.h: struct proc { ... struct prison *p_prison; ... }; In kern_jail.c, the function then copies the pr structure, which is filled with all the information from the original jail structure, over to the p->p_prison structure. It then does a bitwise OR of p->p_flag with the constant P_JAILED, meaning that the calling process is now recognized as jailed. The parent process of each process, forked within the jail, is the program jail itself, as it calls the &man.jail.2; system call. When the program is executed through execve, it inherits the properties of its parents proc structure, therefore it has the p->p_flag set, and the p->p_prison structure is filled. /usr/src/sys/kern/kern_jail.c p->p.prison = pr; p->p.flag --= P.JAILED; When a process is forked from a parent process, the &man.fork.2; system call deals differently with imprisoned processes. In the fork system call, there are two pointers to a proc structure p1 and p2. p1 points to the parent's proc structure and p2 points to the child's unfilled proc structure. After copying all relevant data between the structures, &man.fork.2; checks if the structure p->p_prison is filled on p2. If it is, it increments the pr.ref by one, and sets the p_flag to one on the child process. /usr/src/sys/kern/kern_fork.c: if (p2->p_prison) { p2->p_prison->pr_ref++; p2->p_flag |= P_JAILED; } Restrictions Throughout the kernel there are access restrictions relating to jailed processes. Usually, these restrictions only check if the process is jailed, and if so, returns an error. For example: if (p->p_prison) return EPERM; SysV IPC System V IPC is based on messages. Processes can send each other these messages which tell them how to act. The functions which deal with messages are: msgsys, msgctl, msgget, msgsend and msgrcv. Earlier, I mentioned that there were certain sysctls you could turn on or off in order to affect the behavior of Jail. One of these sysctls was jail_sysvipc_allowed. On most systems, this sysctl is set to 0. If it were set to 1, it would defeat the whole purpose of having a jail; privleged users from within the jail would be able to affect processes outside of the environment. The difference between a message and a signal is that the message only consists of the signal number. /usr/src/sys/kern/sysv_msg.c: &man.msgget.3;: msgget returns (and possibly creates) a message descriptor that designates a message queue for use in other system calls. &man.msgctl.3;: Using this function, a process can query the status of a message descriptor. &man.msgsnd.3;: msgsnd sends a message to a process. &man.msgrcv.3;: a process receives messages using this function In each of these system calls, there is this conditional: /usr/src/sys/kern/sysv msg.c: if (!jail.sysvipc.allowed && p->p_prison != NULL) return (ENOSYS); Semaphore system calls allow processes to synchronize execution by doing a set of operations atomically on a set of semaphores. Basically semaphores provide another way for processes lock resources. However, process waiting on a semaphore, that is being used, will sleep until the resources are relinquished. The following semaphore system calls are blocked inside a jail: semsys, semget, semctl and semop. /usr/src/sys/kern/sysv_sem.c: &man.semctl.2;(id, num, cmd, arg): Semctl does the specified cmd on the semaphore queue indicated by id. &man.semget.2;(key, nsems, flag): Semget creates an array of semaphores, corresponding to key. Key and flag take on the same meaning as they do in msgget. &man.semop.2;(id, ops, num): Semop does the set of semaphore operations in the array of structures ops, to the set of semaphores identified by id. System V IPC allows for processes to share memory. Processes can communicate directly with each other by sharing parts of their virtual address space and then reading and writing data stored in the shared memory. These system calls are blocked within a jailed environment: shmdt, shmat, oshmctl, shmctl, shmget, and shmsys. /usr/src/sys/kern/sysv shm.c: &man.shmctl.2;(id, cmd, buf): shmctl does various control operations on the shared memory region identified by id. &man.shmget.2;(key, size, flag): shmget accesses or creates a shared memory region of size bytes. &man.shmat.2;(id, addr, flag): shmat attaches a shared memory region identified by id to the address space of a process. &man.shmdt.2;(addr): shmdt detaches the shared memory region previously attached at addr. Sockets Jail treats the &man.socket.2; system call and related lower-level socket functions in a special manner. In order to determine whether a certain socket is allowed to be created, it first checks to see if the sysctl jail.socket.unixiproute.only is set. If set, sockets are only allowed to be created if the family specified is either PF_LOCAL, PF_INET or PF_ROUTE. Otherwise, it returns an error. /usr/src/sys/kern/uipc_socket.c: int socreate(dom, aso, type, proto, p) ... register struct protosw *prp; ... { if (p->p_prison && jail_socket_unixiproute_only && prp->pr_domain->dom_family != PR_LOCAL && prp->pr_domain->dom_family != PF_INET && prp->pr_domain->dom_family != PF_ROUTE) return (EPROTONOSUPPORT); ... } Berkeley Packet Filter The Berkeley Packet Filter provides a raw interface to data link layers in a protocol independent fashion. The function bpfopen() opens an Ethernet device. There is a conditional which disallows any jailed processes from accessing this function. /usr/src/sys/net/bpf.c: static int bpfopen(dev, flags, fmt, p) ... { if (p->p_prison) return (EPERM); ... } Protocols There are certain protocols which are very common, such as TCP, UDP, IP and ICMP. IP and ICMP are on the same level: the network layer 2 . There are certain precautions which are taken in order to prevent a jailed process from binding a protocol to a certain port only if the nam parameter is set. nam is a pointer to a sockaddr structure, which describes the address on which to bind the service. A more exact definition is that sockaddr "may be used as a template for reffering to the identifying tag and length of each address"[2] . In the function in pcbbind, sin is a pointer to a sockaddr.in structure, which contains the port, address, length and domain family of the socket which is to be bound. Basically, this disallows any processes from jail to be able to specify the domain family. /usr/src/sys/kern/netinet/in_pcb.c: int in.pcbbind(int, nam, p) ... struct sockaddr *nam; struct proc *p; { ... struct sockaddr.in *sin; ... if (nam) { sin = (struct sockaddr.in *)nam; ... if (sin->sin_addr.s_addr != INADDR_ANY) if (prison.ip(p, 0, ->sin.addr.s_addr)) return (EINVAL); .... } ... } You might be wondering what function prison_ip() does. prison.ip is given three arguments, the current process (represented by p), any flags, and an ip address. It returns 1 if the ip address belongs to a jail or 0 if it does not. As you can see from the code, if it is indeed an ip address belonging to a jail, the protcol is not allowed to bind to a certain port. /usr/src/sys/kern/kern_jail.c: int prison_ip(struct proc *p, int flag, u_int32_t *ip) { u_int32_t tmp; if (!p->p_prison) return (0); if (flag) tmp = *ip; else tmp = ntohl (*ip); if (tmp == INADDR_ANY) { if (flag) *ip = p->p_prison->pr_ip; else *ip = htonl(p->p_prison->pr_ip); return (0); } if (p->p_prison->pr_ip != tmp) return (1); return (0); } Jailed users are not allowed to bind services to an ip which does not belong to the jail. The restriction is also written within the function in_pcbbind: /usr/src/sys/net inet/in_pcb.c if (nam) { ... lport = sin->sin.port; ... if (lport) { ... if (p && p->p_prison) prison = 1; if (prison && prison_ip(p, 0, ->sin_addr.s_addr)) return (EADDRNOTAVAIL); Filesystem Even root users within the jail are not allowed to set any file flags, such as immutable, append, and no unlink flags, if the securelevel is greater than 0. /usr/src/sys/ufs/ufs/ufs_vnops.c: int ufs.setattr(ap) ... { if ((cred->cr.uid == 0) && (p->prison == NULL)) { if ((ip->i_flags & (SF_NOUNLINK | SF_IMMUTABLE | SF_APPEND)) && securelevel > 0) return (EPERM); } Jail NG Jail NG is a "from-scratch re-implementation of Jail" by Robert Watson, a FreeBSD committer. Some of the new features include the ability to add processes to a jail, an improved management tool, and per-jail sysctls. For example, you could have sysvipc_permitted set on one jail while another jail may be allowed to use System V IPC. You can download the kernel patches and utilities for Jail NG from his website at: . diff --git a/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml index 5268f992d4..dc4f4e0ce8 100644 --- a/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/kerneldebug/chapter.sgml @@ -1,651 +1,651 @@ Kernel Debugging Contributed by &a.paul; and &a.joerg; Debugging a Kernel Crash Dump with <command>gdb</command> Here are some instructions for getting kernel debugging working on a crash dump. They assume that you have enough swap space for a crash dump. If you have multiple swap partitions and the first one is too small to hold the dump, you can configure your kernel to use an alternate dump device (in the config kernel line), or you can specify an alternate using the &man.dumpon.8; command. The best way to use &man.dumpon.8; is to set the dumpdev variable in /etc/rc.conf. Typically you want to specify one of the swap devices specified in /etc/fstab. Dumps to non-swap devices, tapes for example, are currently not supported. Config your kernel using config . See The FreeBSD Handbook for details on configuring the FreeBSD kernel. Use the &man.dumpon.8; command to tell the kernel where to dump to (note that this will have to be done after configuring the partition in question as swap space via &man.swapon.8;). This is normally arranged via /etc/rc.conf and /etc/rc. Alternatively, you can hard-code the dump device via the dump clause in the config line of your kernel config file. This is deprecated and should be used only if you want a crash dump from a kernel that crashes during booting. In the following, the term gdb refers to the debugger gdb run in kernel debug mode. This can be accomplished by starting the gdb with the option . In kernel debug mode, gdb changes its prompt to (kgdb). If you are using FreeBSD 3 or earlier, you should make a stripped copy of the debug kernel, rather than installing the large debug kernel itself: &prompt.root; cp kernel kernel.debug &prompt.root; strip -g kernel This stage is not necessary, but it is recommended. (In FreeBSD 4 and later releases this step is performed automatically at the end of the kernel make process.) When the kernel has been stripped, either automatically or by using the commands above, you may install it as usual by typing make install. Note that older releases of FreeBSD (up to but not including 3.1) used a.out kernels by default, which must have their symbol tables permanently resident in physical memory. With the larger symbol table in an unstripped debug kernel, this is wasteful. Recent FreeBSD releases use ELF kernels where this is no longer a problem. If you are testing a new kernel, for example by typing the new kernel's name at the boot prompt, but need to boot a different one in order to get your system up and running again, boot it only into single user state using the flag at the boot prompt, and then perform the following steps: &prompt.root; fsck -p &prompt.root; mount -a -t ufs # so your file system for /var/crash is writable &prompt.root; savecore -N /kernel.panicked /var/crash &prompt.root; exit # ...to multi-user This instructs &man.savecore.8; to use another kernel for symbol name extraction. It would otherwise default to the currently running kernel and most likely not do anything at all since the crash dump and the kernel symbols differ. Now, after a crash dump, go to /sys/compile/WHATEVER and run gdb . From gdb do: symbol-file kernel.debug exec-file /var/crash/kernel.0 core-file /var/crash/vmcore.0 and voila, you can debug the crash dump using the kernel sources just like you can for any other program. Here is a script log of a gdb session illustrating the procedure. Long lines have been folded to improve readability, and the lines are numbered for reference. Despite this, it is a real-world error trace taken during the development of the pcvt console driver. 1:Script started on Fri Dec 30 23:15:22 1994 2:&prompt.root; cd /sys/compile/URIAH 3:&prompt.root; gdb -k kernel /var/crash/vmcore.1 4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel ...done. 5:IdlePTD 1f3000 6:panic: because you said to! 7:current pcb at 1e3f70 8:Reading in symbols for ../../i386/i386/machdep.c...done. 9:(kgdb) where 10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767) 11:#1 0xf0115159 in panic () 12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698) 13:#3 0xf010185e in db_fncall () 14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073) 15:#5 0xf0101711 in db_command_loop () 16:#6 0xf01040a0 in db_trap () 17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723) 18:#8 0xf019d2eb in trap_fatal (...) 19:#9 0xf019ce60 in trap_pfault (...) 20:#10 0xf019cb2f in trap (...) 21:#11 0xf01932a1 in exception:calltrap () 22:#12 0xf0191503 in cnopen (...) 23:#13 0xf0132c34 in spec_open () 24:#14 0xf012d014 in vn_open () 25:#15 0xf012a183 in open () 26:#16 0xf019d4eb in syscall (...) 27:(kgdb) up 10 28:Reading in symbols for ../../i386/i386/trap.c...done. 29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\ 30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\ 31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\ 32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\ 33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\ 34:ss = -266427884}) (../../i386/i386/trap.c line 283) 35:283 (void) trap_pfault(&frame, FALSE); 36:(kgdb) frame frame->tf_ebp frame->tf_eip 37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done. 38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\ 39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403) 40:403 return ((*linesw[tp->t_line].l_open)(dev, tp)); 41:(kgdb) list 42:398 43:399 tp->t_state |= TS_CARR_ON; 44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */ 45:401 46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200) 47:403 return ((*linesw[tp->t_line].l_open)(dev, tp)); 48:404 #else 49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag)); 50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */ 51:407 } 52:(kgdb) print tp 53:Reading in symbols for ../../i386/i386/cons.c...done. 54:$1 = (struct tty *) 0x1bae 55:(kgdb) print tp->t_line 56:$2 = 1767990816 57:(kgdb) up 58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\ 59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126) 60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p)); 61:(kgdb) up 62:#2 0xf0132c34 in spec_open () 63:(kgdb) up 64:#3 0xf012d014 in vn_open () 65:(kgdb) up 66:#4 0xf012a183 in open () 67:(kgdb) up 68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\ 69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\ 70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \ 71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \ 72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673) 73:673 error = (*callp->sy_call)(p, args, rval); 74:(kgdb) up 75:Initial frame selected; you cannot go up. 76:(kgdb) quit 77:&prompt.root; exit 78:exit 79: 80:Script done on Fri Dec 30 23:18:04 1994 Comments to the above script: line 6: This is a dump taken from within DDB (see below), hence the panic comment because you said to!, and a rather long stack trace; the initial reason for going into DDB has been a page fault trap though. line 20: This is the location of function trap() in the stack trace. line 36: Force usage of a new stack frame; this is no longer necessary now. The stack frames are supposed to point to the right locations now, even in case of a trap. From looking at the code in source line 403, there is a high probability that either the pointer access for tp was messed up, or the array access was out of bounds. line 52: The pointer looks suspicious, but happens to be a valid address. line 56: However, it obviously points to garbage, so we have found our error! (For those unfamiliar with that particular piece of code: tp->t_line refers to the line discipline of the console device here, which must be a rather small integer number.) Debugging a Crash Dump with DDD Examining a kernel crash dump with a graphical debugger like ddd is also possible. Add the option to the ddd command line you would use normally. For example; &prompt.root; ddd -k /var/crash/kernel.0 /var/crash/vmcore.0 You should then be able to go about looking at the crash dump using ddd's graphical interface. Post-Mortem Analysis of a Dump What do you do if a kernel dumped core but you did not expect it, and it is therefore not compiled using config -g? Not everything is lost here. Do not panic! Of course, you still need to enable crash dumps. See above on the options you have to specify in order to do this. Go to your kernel config directory (/usr/src/sys/arch/conf) and edit your configuration file. Uncomment (or add, if it does not exist) the following line makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols Rebuild the kernel. Due to the time stamp change on the Makefile, there will be some other object files rebuild, for example trap.o. With a bit of luck, the added option will not change anything for the generated code, so you will finally get a new kernel with similar code to the faulting one but some debugging symbols. You should at least verify the old and new sizes with the &man.size.1; command. If there is a mismatch, you probably need to give up here. Go and examine the dump as described above. The debugging symbols might be incomplete for some places, as can be seen in the stack trace in the example above where some functions are displayed without line numbers and argument lists. If you need more debugging symbols, remove the appropriate object files and repeat the gdb session until you know enough. All this is not guaranteed to work, but it will do it fine in most cases. On-Line Kernel Debugging Using DDB While gdb as an off-line debugger provides a very high level of user interface, there are some things it cannot do. The most important ones being breakpointing and single-stepping kernel code. If you need to do low-level debugging on your kernel, there is an on-line debugger available called DDB. It allows to setting breakpoints, single-stepping kernel functions, examining and changing kernel variables, etc. However, it cannot access kernel source files, and only has access to the global and static symbols, not to the full debug information like gdb. To configure your kernel to include DDB, add the option line options DDB to your config file, and rebuild. (See The FreeBSD Handbook for details on configuring the FreeBSD kernel. If you have an older version of the boot blocks, your debugger symbols might not be loaded at all. Update the boot blocks; the recent ones load the DDB symbols automagically.) Once your DDB kernel is running, there are several ways to enter DDB. The first, and earliest way is to type the boot flag right at the boot prompt. The kernel will start up in debug mode and enter DDB prior to any device probing. Hence you can even debug the device probe/attach functions. The second scenario is to drop to the debugger once the system has booted. There are two simple ways to accomplish this. If you would like to break to the debugger from the - command prompt, simply type the command : + command prompt, simply type the command: &prompt.root; sysctl debug.enter_debugger=ddb Alternatively, if you are at the system console, you may use a hot-key on the keyboard. The default break-to-debugger sequence is Ctrl AltESC. For syscons, this sequence can be remapped and some of the distributed maps out there do this, so check to make sure you know the right sequence to use. There is an option available for serial consoles that allows the use of a serial line BREAK on the console line to enter DDB (options BREAK_TO_DEBUGGER in the kernel config file). It is not the default since there are a lot of crappy serial adapters around that gratuitously generate a BREAK condition, for example when pulling the cable. The third way is that any panic condition will branch to DDB if the kernel is configured to use it. For this reason, it is not wise to configure a kernel with DDB for a machine running unattended. The DDB commands roughly resemble some gdb commands. The first thing you probably need to do is to set a breakpoint: b function-name b address Numbers are taken hexadecimal by default, but to make them distinct from symbol names; hexadecimal numbers starting with the letters a-f need to be preceded with 0x (this is optional for other numbers). Simple expressions are allowed, for example: function-name + 0x103. To continue the operation of an interrupted kernel, simply type: c To get a stack trace, use: trace Note that when entering DDB via a hot-key, the kernel is currently servicing an interrupt, so the stack trace might be not of much use for you. If you want to remove a breakpoint, use del del address-expression The first form will be accepted immediately after a breakpoint hit, and deletes the current breakpoint. The second form can remove any breakpoint, but you need to specify the exact address; this can be obtained from: show b To single-step the kernel, try: s This will step into functions, but you can make DDB trace them until the matching return statement is reached by: n This is different from gdb's next statement; it is like gdb's finish. To examine data from memory, use (for example): x/wx 0xf0133fe0,40 x/hd db_symtab_space x/bc termbuf,10 x/s stringbuf for word/halfword/byte access, and hexadecimal/decimal/character/ string display. The number after the comma is the object count. To display the next 0x10 items, simply use: x ,10 Similarly, use x/ia foofunc,10 to disassemble the first 0x10 instructions of foofunc, and display them along with their offset from the beginning of foofunc. To modify memory, use the write command: w/b termbuf 0xa 0xb 0 w/w 0xf0010030 0 0 The command modifier (b/h/w) specifies the size of the data to be written, the first following expression is the address to write to and the remainder is interpreted as data to write to successive memory locations. If you need to know the current registers, use: show reg Alternatively, you can display a single register value by e.g. p $eax and modify it by: set $eax new-value Should you need to call some kernel functions from DDB, simply say: call func(arg1, arg2, ...) The return value will be printed. For a &man.ps.1; style summary of all running processes, use: ps Now you have examined why your kernel failed, and you wish to reboot. Remember that, depending on the severity of previous malfunctioning, not all parts of the kernel might still be working as expected. Perform one of the following actions to shut down and reboot your system: panic This will cause your kernel to dump core and reboot, so you can later analyze the core on a higher level with gdb. This command usually must be followed by another continue statement. call boot(0) Which might be a good way to cleanly shut down the running system, sync() all disks, and finally reboot. As long as the disk and file system interfaces of the kernel are not damaged, this might be a good way for an almost clean shutdown. call cpu_reset() is the final way out of disaster and almost the same as hitting the Big Red Button. If you need a short command summary, simply type: help However, it is highly recommended to have a printed copy of the &man.ddb.4; manual page ready for a debugging session. Remember that it is hard to read the on-line manual while single-stepping the kernel. On-Line Kernel Debugging Using Remote GDB This feature has been supported since FreeBSD 2.2, and it is actually a very neat one. GDB has already supported remote debugging for a long time. This is done using a very simple protocol along a serial line. Unlike the other methods described above, you will need two machines for doing this. One is the host providing the debugging environment, including all the sources, and a copy of the kernel binary with all the symbols in it, and the other one is the target machine that simply runs a similar copy of the very same kernel (but stripped of the debugging information). You should configure the kernel in question with config -g, include into the configuration, and compile it as usual. This gives a large blurb of a binary, due to the debugging information. Copy this kernel to the target machine, strip the debugging symbols off with strip -x, and boot it using the boot option. Connect the serial line of the target machine that has "flags 080" set on its sio device to any serial line of the debugging host. Now, on the debugging machine, go to the compile directory of the target kernel, and start gdb: &prompt.user; gdb -k kernel GDB is free software and you are welcome to distribute copies of it under certain conditions; type "show copying" to see the conditions. There is absolutely no warranty for GDB; type "show warranty" for details. GDB 4.16 (i386-unknown-freebsd), Copyright 1996 Free Software Foundation, Inc... (kgdb) Initialize the remote debugging session (assuming the first serial port is being used) by: (kgdb) target remote /dev/cuaa0 Now, on the target host (the one that entered DDB right before even starting the device probe), type: Debugger("Boot flags requested debugger") Stopped at Debugger+0x35: movb $0, edata+0x51bc db> gdb DDB will respond with: Next trap will enter GDB remote protocol mode Every time you type gdb, the mode will be toggled between remote GDB and local DDB. In order to force a next trap immediately, simply type s (step). Your hosting GDB will now gain control over the target kernel: Remote debugging using /dev/cuaa0 Debugger (msg=0xf01b0383 "Boot flags requested debugger") at ../../i386/i386/db_interface.c:257 (kgdb) You can use this session almost as any other GDB session, including full access to the source, running it in gud-mode inside an Emacs window (which gives you an automatic source code display in another Emacs window) etc. Debugging Loadable Modules Using GDB When debugging a panic that occurred within a module, or using remote GDB against a machine that uses dynamic modules, you need to tell GDB how to obtain symbol information for those modules. First, you need to build the module(s) with debugging information: &prompt.root; cd /sys/modules/linux &prompt.root; make clean; make COPTS=-g If you are using remote GDB, you can run kldstat on the target machine to find out where the module was loaded: &prompt.root; kldstat Id Refs Address Size Name 1 4 0xc0100000 1c1678 kernel 2 1 0xc0a9e000 6000 linprocfs.ko 3 1 0xc0ad7000 2000 warp_saver.ko 4 1 0xc0adc000 11000 linux.ko If you are debugging a crash dump, you will need to walk the linker_files list, starting at linker_files->tqh_first and following the link.tqe_next pointers until you find the entry with the filename you are looking for. The address member of that entry is the load address of the module. Next, you need to find out the offset of the text section within the module: &prompt.root; objdump --section-headers /sys/modules/linux/linux.ko | grep text 3 .rel.text 000016e0 000038e0 000038e0 000038e0 2**2 10 .text 00007f34 000062d0 000062d0 000062d0 2**2 The one you want is the .text section, section 10 in the above example. The fourth hexadecimal field (sixth field overall) is the offset of the text section within the file. Add this offset to the load address of the module to obtain the relocation address for the module's code. In our example, we get 0xc0adc000 + 0x62d0 = 0xc0ae22d0. Use the add-symbol-file command in GDB to tell the debugger about the module: (kgdb) add-symbol-file /sys/modules/linux/linux.ko 0xc0ae22d0 add symbol table from file "/sys/modules/linux/linux.ko" at text_addr = 0xc0ae22d0? (y or n) y Reading symbols from /sys/modules/linux/linux.ko...done. (kgdb) You should now have access to all the symbols in the module. Debugging a Console Driver Since you need a console driver to run DDB on, things are more complicated if the console driver itself is failing. You might remember the use of a serial console (either with modified boot blocks, or by specifying at the Boot: prompt), and hook up a standard terminal onto your first serial port. DDB works on any configured console driver, of course also on a serial console. diff --git a/en_US.ISO8859-1/books/developers-handbook/pci/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/pci/chapter.sgml index d1085cce39..15146dc9e9 100644 --- a/en_US.ISO8859-1/books/developers-handbook/pci/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/pci/chapter.sgml @@ -1,370 +1,369 @@ PCI Devices This chapter will talk about the FreeBSD mechanisms for writing a device driver for a device on a PCI bus. Probe and Attach Information here about how the PCI bus code iterates through the unattached devices and see if a newly loaded kld will attach to any of them. /* * Simple KLD to play with the PCI functions. * * Murray Stokely */ #define MIN(a,b) (((a) < (b)) ? (a) : (b)) #include <sys/types.h> #include <sys/module.h> #include <sys/systm.h> /* uprintf */ #include <sys/errno.h> #include <sys/param.h> /* defines used in kernel.h */ #include <sys/kernel.h> /* types used in module initialization */ #include <sys/conf.h> /* cdevsw struct */ #include <sys/uio.h> /* uio struct */ #include <sys/malloc.h> #include <sys/bus.h> /* structs, prototypes for pci bus stuff */ #include <pci/pcivar.h> /* For get_pci macros! */ /* Function prototypes */ d_open_t mypci_open; d_close_t mypci_close; d_read_t mypci_read; d_write_t mypci_write; /* Character device entry points */ static struct cdevsw mypci_cdevsw = { mypci_open, mypci_close, mypci_read, mypci_write, noioctl, nopoll, nommap, nostrategy, "mypci", 36, /* reserved for lkms - /usr/src/sys/conf/majors */ nodump, nopsize, D_TTY, -1 }; /* vars */ static dev_t sdev; /* We're more interested in probe/attach than with open/close/read/write at this point */ int mypci_open(dev_t dev, int oflags, int devtype, struct proc *p) { int err = 0; uprintf("Opened device \"mypci\" successfully.\n"); return(err); } int mypci_close(dev_t dev, int fflag, int devtype, struct proc *p) { int err=0; uprintf("Closing device \"mypci.\"\n"); return(err); } int mypci_read(dev_t dev, struct uio *uio, int ioflag) { int err = 0; uprintf("mypci read!\n"); return err; } int mypci_write(dev_t dev, struct uio *uio, int ioflag) { int err = 0; uprintf("mypci write!\n"); return(err); } /* PCI Support Functions */ /* * Return identification string if this is device is ours. */ static int mypci_probe(device_t dev) { uprintf("MyPCI Probe\n" "Vendor ID : 0x%x\n" "Device ID : 0x%x\n",pci_get_vendor(dev),pci_get_device(dev)); if (pci_get_vendor(dev) == 0x11c1) { uprintf("We've got the Winmodem, probe successful!\n"); return 0; } return ENXIO; } /* Attach function is only called if the probe is successful */ static int mypci_attach(device_t dev) { uprintf("MyPCI Attach for : deviceID : 0x%x\n",pci_get_vendor(dev)); sdev = make_dev(&mypci_cdevsw, 0, UID_ROOT, GID_WHEEL, 0600, "mypci"); uprintf("Mypci device loaded.\n"); return ENXIO; } /* Detach device. */ static int mypci_detach(device_t dev) { uprintf("Mypci detach!\n"); return 0; } /* Called during system shutdown after sync. */ static int mypci_shutdown(device_t dev) { uprintf("Mypci shutdown!\n"); return 0; } /* * Device suspend routine. */ static int mypci_suspend(device_t dev) { uprintf("Mypci suspend!\n"); return 0; } /* * Device resume routine. */ static int mypci_resume(device_t dev) { uprintf("Mypci resume!\n"); return 0; } static device_method_t mypci_methods[] = { /* Device interface */ DEVMETHOD(device_probe, mypci_probe), DEVMETHOD(device_attach, mypci_attach), DEVMETHOD(device_detach, mypci_detach), DEVMETHOD(device_shutdown, mypci_shutdown), DEVMETHOD(device_suspend, mypci_suspend), DEVMETHOD(device_resume, mypci_resume), { 0, 0 } }; static driver_t mypci_driver = { "mypci", mypci_methods, 0, /* sizeof(struct mypci_softc), */ }; static devclass_t mypci_devclass; DRIVER_MODULE(mypci, pci, mypci_driver, mypci_devclass, 0, 0); Additional Resources PCI Special Interest Group PCI System Architecture, Fourth Edition by Tom Shanley, et al. Bus Resources FreeBSD provides an object-oriented mechanism for requesting resources from a parent bus. Almost all devices will be a child member of some sort of bus (PCI, ISA, USB, SCSI, etc) and these devices need to acquire resources from their parent bus (such as memory segments, interrupt lines, or DMA channels). Base Address Registers To do anything particularly useful with a PCI device you will need to obtain the Base Address Registers (BARs) from the PCI Configuration space. The PCI-specific details of obtaining the BAR is abstracted in the bus_alloc_resource() function. For example, a typical driver might have something similar - to this in the attach() function. : + to this in the attach() function: sc->bar0id = 0x10; sc->bar0res = bus_alloc_resource(dev, SYS_RES_MEMORY, &(sc->bar0id), 0, ~0, 1, RF_ACTIVE); if (sc->bar0res == NULL) { uprintf("Memory allocation of PCI base register 0 failed!\n"); error = ENXIO; goto fail1; } sc->bar1id = 0x14; sc->bar1res = bus_alloc_resource(dev, SYS_RES_MEMORY, &(sc->bar1id), 0, ~0, 1, RF_ACTIVE); if (sc->bar1res == NULL) { uprintf("Memory allocation of PCI base register 1 failed!\n"); error = ENXIO; goto fail2; } sc->bar0_bt = rman_get_bustag(sc->bar0res); sc->bar0_bh = rman_get_bushandle(sc->bar0res); sc->bar1_bt = rman_get_bustag(sc->bar1res); sc->bar1_bh = rman_get_bushandle(sc->bar1res); Handles for each base address register are kept in the softc structure so that they can be used to write to the device later. These handles can then be used to read or write from the device registers with the bus_space_* functions. For example, a driver might contain a shorthand - function to read from a board specific register like this : - + function to read from a board specific register like this: uint16_t board_read(struct ni_softc *sc, uint16_t address) { return bus_space_read_2(sc->bar1_bt, sc->bar1_bh, address); } - Similarly, one could write to the registers with : + Similarly, one could write to the registers with: void board_write(struct ni_softc *sc, uint16_t address, uint16_t value) { bus_space_write_2(sc->bar1_bt, sc->bar1_bh, address, value); } These functions exist in 8bit, 16bit, and 32bit versions and you should use bus_space_{read|write}_{1|2|4} accordingly. Interrupts Interrupts are allocated from the object-oriented bus code in a way similar to the memory resources. First an IRQ resource must be allocated from the parent bus, and then the interrupt handler must be setup to deal with this IRQ. Again, a sample from a device attach() function says more than words. /* Get the IRQ resource */ sc->irqid = 0x0; sc->irqres = bus_alloc_resource(dev, SYS_RES_IRQ, &(sc->irqid), 0, ~0, 1, RF_SHAREABLE | RF_ACTIVE); if (sc->irqres == NULL) { uprintf("IRQ allocation failed!\n"); error = ENXIO; goto fail3; } /* Now we should setup the interrupt handler */ error = bus_setup_intr(dev, sc->irqres, INTR_TYPE_MISC, my_handler, sc, &(sc->handler)); if (error) { printf("Couldn't set up irq\n"); goto fail4; } sc->irq_bt = rman_get_bustag(sc->irqres); sc->irq_bh = rman_get_bushandle(sc->irqres); DMA On the PC, peripherals that want to do bus-mastering DMA must deal with physical addresses. This is a problem since FreeBSD uses virtual memory and deals almost exclusively with virtual addresses. Fortunately, there is a function, vtophys() to help. #include <vm/vm.h> #include <vm/pmap.h> #define vtophys(virtual_address) (...) The solution is a bit different on the alpha however, and what we really want is a function called vtobus(). #if defined(__alpha__) #define vtobus(va) alpha_XXX_dmamap((vm_offset_t)va) #else #define vtobus(va) vtophys(va) #endif Deallocating Resources It is very important to deallocate all of the resources that were allocated during attach(). Care must be taken to deallocate the correct stuff even on a failure condition so that the system will remain usable while your driver dies. diff --git a/en_US.ISO8859-1/books/developers-handbook/scsi/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/scsi/chapter.sgml index a3881b407e..4e9c6e5332 100644 --- a/en_US.ISO8859-1/books/developers-handbook/scsi/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/scsi/chapter.sgml @@ -1,1983 +1,1983 @@ Common Access Method SCSI Controllers This chapter was written by &a.babkin; Modifications for the handbook made by &a.murray;. Synopsis This document assumes that the reader has a general understanding of device drivers in FreeBSD and of the SCSI protocol. Much of the information in this document was - extracted from the drivers : + extracted from the drivers: ncr (/sys/pci/ncr.c) by Wolfgang Stanglmeier and Stefan Esser sym (/sys/pci/sym.c) by Gerard Roudier aic7xxx (/sys/dev/aic7xxx/aic7xxx.c) by Justin T. Gibbs and from the CAM code itself (by Justing T. Gibbs, see /sys/cam/*). When some solution looked the most logical and was essentially verbatim extracted from the code by Justin Gibbs, I marked it as "recommended". The document is illustrated with examples in pseudo-code. Although sometimes the examples have many details and look like real code, it is still pseudo-code. It was written to demonstrate the concepts in an understandable way. For a real driver other approaches may be more modular and efficient. It also abstracts from the hardware details, as well as issues that would cloud the demonstrated concepts or that are supposed to be described in the other chapters of the developers handbook. Such details are commonly shown as calls to functions with descriptive names, comments or pseudo-statements. Fortunately real life full-size examples with all the details can be found in the real drivers. General architecture CAM stands for Common Access Method. It is a generic way to address the I/O buses in a SCSI-like way. This allows a separation of the generic device drivers from the drivers controlling the I/O bus: for example the disk driver becomes able to control disks on both SCSI, IDE, and/or any other bus so the disk driver portion does not have to be rewritten (or copied and modified) for every new I/O bus. Thus the two most important active entities are: Peripheral Modules - a driver for peripheral devices (disk, tape, CDROM, etc.) SCSI Interface Modules (SIM) - a Host Bus Adapter drivers for connecting to an I/O bus such as SCSI or IDE. A peripheral driver receives requests from the OS, converts them to a sequence of SCSI commands and passes these SCSI commands to a SCSI Interface Module. The SCSI Interface Module is responsible for passing these commands to the actual hardware (or if the actual hardware is not SCSI but, for example, IDE then also converting the SCSI commands to the native commands of the hardware). Because we are interested in writing a SCSI adapter driver here, from this point on we will consider everything from the SIM standpoint. A typical SIM driver needs to include the following CAM-related header files: #include <cam/cam.h> #include <cam/cam_ccb.h> #include <cam/cam_sim.h> #include <cam/cam_xpt_sim.h> #include <cam/cam_debug.h> #include <cam/scsi/scsi_all.h> The first thing each SIM driver must do is register itself with the CAM subsystem. This is done during the driver's xxx_attach() function (here and further xxx_ is used to denote the unique driver name prefix). The xxx_attach() function itself is called by the system bus auto-configuration code which we do not describe here. This is achieved in multiple steps: first it is necessary to allocate the queue of requests associated with this SIM: struct cam_devq *devq; if(( devq = cam_simq_alloc(SIZE) )==NULL) { error; /* some code to handle the error */ } Here SIZE is the size of the queue to be allocated, maximal number of requests it could contain. It is the number of requests that the SIM driver can handle in parallel on one SCSI card. Commonly it can be calculated as: SIZE = NUMBER_OF_SUPPORTED_TARGETS * MAX_SIMULTANEOUS_COMMANDS_PER_TARGET Next we create a descriptor of our SIM: struct cam_sim *sim; if(( sim = cam_sim_alloc(action_func, poll_func, driver_name, softc, unit, max_dev_transactions, max_tagged_dev_transactions, devq) )==NULL) { cam_simq_free(devq); error; /* some code to handle the error */ } Note that if we are not able to create a SIM descriptor we free the devq also because we can do nothing else with it and we want to conserve memory. If a SCSI card has multiple SCSI buses on it then each bus requires its own cam_sim structure. An interesting question is what to do if a SCSI card has more than one SCSI bus, do we need one devq structure per card or per SCSI bus? The answer given in the comments to the CAM code is: either way, as the driver's author prefers. - The arguments are : + The arguments are: action_func - pointer to the driver's xxx_action function. static void xxx_action struct cam_sim *sim, union ccb *ccb poll_func - pointer to the driver's xxx_poll() static void xxx_poll struct cam_sim *sim driver_name - the name of the actual driver, such as "ncr" or "wds" softc - pointer to the driver's internal descriptor for this SCSI card. This pointer will be used by the driver in future to get private data. unit - the controller unit number, for example for controller "wds0" this number will be 0 max_dev_transactions - maximal number of simultaneous transactions per SCSI target in the non-tagged mode. This value will be almost universally equal to 1, with possible exceptions only for the non-SCSI cards. Also the drivers that hope to take advantage by preparing one transaction while another one is executed may set it to 2 but this does not seem to be worth the complexity. max_tagged_dev_transactions - the same thing, but in the tagged mode. Tags are the SCSI way to initiate multiple transactions on a device: each transaction is assigned a unique tag and the transaction is sent to the device. When the device completes some transaction it sends back the result together with the tag so that the SCSI adapter (and the driver) can tell which transaction was completed. This argument is also known as the maximal tag depth. It depends on the abilities of the SCSI adapter. Finally we register the SCSI buses associated with our SCSI adapter: if(xpt_bus_register(sim, bus_number) != CAM_SUCCESS) { cam_sim_free(sim, /*free_devq*/ TRUE); error; /* some code to handle the error */ } If there is one devq structure per SCSI bus (i.e. we consider a card with multiple buses as multiple cards with one bus each) then the bus number will always be 0, otherwise each bus on the SCSI card should be get a distinct number. Each bus needs its own separate structure cam_sim. After that our controller is completely hooked to the CAM system. The value of devq can be discarded now: sim will be passed as an argument in all further calls from CAM and devq can be derived from it. CAM provides the framework for such asynchronous events. Some events originate from the lower levels (the SIM drivers), some events originate from the peripheral drivers, some events originate from the CAM subsystem itself. Any driver can register callbacks for some types of the asynchronous events, so that it would be notified if these events occur. A typical example of such an event is a device reset. Each transaction and event identifies the devices to which it applies by the means of "path". The target-specific events normally occur during a transaction with this device. So the path from that transaction may be re-used to report this event (this is safe because the event path is copied in the event reporting routine but not deallocated nor passed anywhere further). Also it is safe to allocate paths dynamically at any time including the interrupt routines, although that incurs certain overhead, and a possible problem with this approach is that there may be no free memory at that time. For a bus reset event we need to define a wildcard path including all devices on the bus. So we can create the path for the future bus reset events in advance and avoid problems with the future memory shortage: struct cam_path *path; if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), CAM_TARGET_WILDCARD, CAM_LUN_WILDCARD) != CAM_REQ_CMP) { xpt_bus_deregister(cam_sim_path(sim)); cam_sim_free(sim, /*free_devq*/TRUE); error; /* some code to handle the error */ } softc->wpath = path; softc->sim = sim; As you can see the path includes: ID of the peripheral driver (NULL here because we have none) ID of the SIM driver (cam_sim_path(sim)) SCSI target number of the device (CAM_TARGET_WILDCARD means "all devices") SCSI LUN number of the subdevice (CAM_LUN_WILDCARD means "all LUNs") If the driver can not allocate this path it will not be able to work normally, so in that case we dismantle that SCSI bus. And we save the path pointer in the softc structure for future use. After that we save the value of sim (or we can also discard it on the exit from xxx_probe() if we wish). That is all for a minimalistic initialization. To do things right there is one more issue left. For a SIM driver there is one particularly interesting event: when a target device is considered lost. In this case resetting the SCSI negotiations with this device may be a good idea. So we register a callback for this event with CAM. The request is passed to CAM by requesting CAM action on a CAM control block for this type of request: struct ccb_setasync csa; xpt_setup_ccb(&csa.ccb_h, path, /*priority*/5); csa.ccb_h.func_code = XPT_SASYNC_CB; csa.event_enable = AC_LOST_DEVICE; csa.callback = xxx_async; csa.callback_arg = sim; xpt_action((union ccb *)&csa); Now we take a look at the xxx_action() and xxx_poll() driver entry points. static void xxx_action struct cam_sim *sim, union ccb *ccb Do some action on request of the CAM subsystem. Sim describes the SIM for the request, CCB is the request itself. CCB stands for "CAM Control Block". It is a union of many specific instances, each describing arguments for some type of transactions. All of these instances share the CCB header where the common part of arguments is stored. CAM supports the SCSI controllers working in both initiator ("normal") mode and target (simulating a SCSI device) mode. Here we only consider the part relevant to the initiator mode. There are a few function and macros (in other words, methods) defined to access the public data in the struct sim: cam_sim_path(sim) - the path ID (see above) cam_sim_name(sim) - the name of the sim cam_sim_softc(sim) - the pointer to the softc (driver private data) structure cam_sim_unit(sim) - the unit number cam_sim_bus(sim) - the bus ID To identify the device, xxx_action() can get the unit number and pointer to its structure softc using these functions. The type of request is stored in ccb->ccb_h.func_code. So generally xxx_action() consists of a big switch: struct xxx_softc *softc = (struct xxx_softc *) cam_sim_softc(sim); struct ccb_hdr *ccb_h = &ccb->ccb_h; int unit = cam_sim_unit(sim); int bus = cam_sim_bus(sim); switch(ccb_h->func_code) { case ...: ... default: ccb_h->status = CAM_REQ_INVALID; xpt_done(ccb); break; } As can be seen from the default case (if an unknown command was received) the return code of the command is set into ccb->ccb_h.status and the completed CCB is returned back to CAM by calling xpt_done(ccb). xpt_done() does not have to be called from xxx_action(): For example an I/O request may be enqueued inside the SIM driver and/or its SCSI controller. Then when the device would post an interrupt signaling that the processing of this request is complete xpt_done() may be called from the interrupt handling routine. Actually, the CCB status is not only assigned as a return code but a CCB has some status all the time. Before CCB is passed to the xxx_action() routine it gets the status CCB_REQ_INPROG meaning that it is in progress. There are a surprising number of status values defined in /sys/cam/cam.h which should be able to represent the status of a request in great detail. More interesting yet, the status is in fact a "bitwise or" of an enumerated status value (the lower 6 bits) and possible additional flag-like bits (the upper bits). The enumerated values will be discussed later in more detail. The summary of them can be found in the Errors Summary section. The possible status flags are: CAM_DEV_QFRZN - if the SIM driver gets a serious error (for example, the device does not respond to the selection or breaks the SCSI protocol) when processing a CCB it should freeze the request queue by calling xpt_freeze_simq(), return the other enqueued but not processed yet CCBs for this device back to the CAM queue, then set this flag for the troublesome CCB and call xpt_done(). This flag causes the CAM subsystem to unfreeze the queue after it handles the error. CAM_AUTOSNS_VALID - if the device returned an error condition and the flag CAM_DIS_AUTOSENSE is not set in CCB the SIM driver must execute the REQUEST SENSE command automatically to extract the sense (extended error information) data from the device. If this attempt was successful the sense data should be saved in the CCB and this flag set. CAM_RELEASE_SIMQ - like CAM_DEV_QFRZN but used in case there is some problem (or resource shortage) with the SCSI controller itself. Then all the future requests to the controller should be stopped by xpt_freeze_simq(). The controller queue will be restarted after the SIM driver overcomes the shortage and informs CAM by returning some CCB with this flag set. CAM_SIM_QUEUED - when SIM puts a CCB into its request queue this flag should be set (and removed when this CCB gets dequeued before being returned back to CAM). This flag is not used anywhere in the CAM code now, so its purpose is purely diagnostic. The function xxx_action() is not allowed to sleep, so all the synchronization for resource access must be done using SIM or device queue freezing. Besides the aforementioned flags the CAM subsystem provides functions xpt_selease_simq() and xpt_release_devq() to unfreeze the queues directly, without passing a CCB to CAM. The CCB header contains the following fields: path - path ID for the request target_id - target device ID for the request target_lun - LUN ID of the target device timeout - timeout interval for this command, in milliseconds timeout_ch - a convenience place for the SIM driver to store the timeout handle (the CAM subsystem itself does not make any assumptions about it) flags - various bits of information about the request spriv_ptr0, spriv_ptr1 - fields reserved for private use by the SIM driver (such as linking to the SIM queues or SIM private control blocks); actually, they exist as unions: spriv_ptr0 and spriv_ptr1 have the type (void *), spriv_field0 and spriv_field1 have the type unsigned long, sim_priv.entries[0].bytes and sim_priv.entries[1].bytes are byte arrays of the size consistent with the other incarnations of the union and sim_priv.bytes is one array, twice bigger. The recommended way of using the SIM private fields of CCB is to define some meaningful names for them and use these meaningful names in the driver, like: #define ccb_some_meaningful_name sim_priv.entries[0].bytes #define ccb_hcb spriv_ptr1 /* for hardware control block */ The most common initiator mode requests are: XPT_SCSI_IO - execute an I/O transaction The instance "struct ccb_scsiio csio" of the union ccb is used to transfer the arguments. They are: cdb_io - pointer to the SCSI command buffer or the buffer itself cdb_len - SCSI command length data_ptr - pointer to the data buffer (gets a bit complicated if scatter/gather is used) dxfer_len - length of the data to transfer sglist_cnt - counter of the scatter/gather segments scsi_status - place to return the SCSI status sense_data - buffer for the SCSI sense information if the command returns an error (the SIM driver is supposed to run the REQUEST SENSE command automatically in this case if the CCB flag CAM_DIS_AUTOSENSE is not set) sense_len - the length of that buffer (if it happens to be higher than size of sense_data the SIM driver must silently assume the smaller value) resid, sense_resid - if the transfer of data or SCSI sense returned an error these are the returned counters of the residual (not transferred) data. They do not seem to be especially meaningful, so in a case when they are difficult to compute (say, counting bytes in the SCSI controller's FIFO buffer) an approximate value will do as well. For a successfully completed transfer they must be set to zero. tag_action - the kind of tag to use: CAM_TAG_ACTION_NONE - do not use tags for this transaction MSG_SIMPLE_Q_TAG, MSG_HEAD_OF_Q_TAG, MSG_ORDERED_Q_TAG - value equal to the appropriate tag message (see /sys/cam/scsi/scsi_message.h); this gives only the tag type, the SIM driver must assign the tag value itself The general logic of handling this request is the following: The first thing to do is to check for possible races, to make sure that the command did not get aborted when it was sitting in the queue: struct ccb_scsiio *csio = &ccb->csio; if ((ccb_h->status & CAM_STATUS_MASK) != CAM_REQ_INPROG) { xpt_done(ccb); return; } Also we check that the device is supported at all by our controller: if(ccb_h->target_id > OUR_MAX_SUPPORTED_TARGET_ID || cch_h->target_id == OUR_SCSI_CONTROLLERS_OWN_ID) { ccb_h->status = CAM_TID_INVALID; xpt_done(ccb); return; } if(ccb_h->target_lun > OUR_MAX_SUPPORTED_LUN) { ccb_h->status = CAM_LUN_INVALID; xpt_done(ccb); return; } Then allocate whatever data structures (such as card-dependent hardware control block) we need to process this request. If we ca not then freeze the SIM queue and remember that we have a pending operation, return the CCB back and ask CAM to re-queue it. Later when the resources become available the SIM queue must be unfrozen by returning a ccb with the CAM_SIMQ_RELEASE bit set in its status. Otherwise, if all went well, link the CCB with the hardware control block (HCB) and mark it as queued. struct xxx_hcb *hcb = allocate_hcb(softc, unit, bus); if(hcb == NULL) { softc->flags |= RESOURCE_SHORTAGE; xpt_freeze_simq(sim, /*count*/1); ccb_h->status = CAM_REQUEUE_REQ; xpt_done(ccb); return; } hcb->ccb = ccb; ccb_h->ccb_hcb = (void *)hcb; ccb_h->status |= CAM_SIM_QUEUED; Extract the target data from CCB into the hardware control block. Check if we are asked to assign a tag and if yes then generate an unique tag and build the SCSI tag messages. The SIM driver is also responsible for negotiations with the devices to set the maximal mutually supported bus width, synchronous rate and offset. hcb->target = ccb_h->target_id; hcb->lun = ccb_h->target_lun; generate_identify_message(hcb); if( ccb_h->tag_action != CAM_TAG_ACTION_NONE ) generate_unique_tag_message(hcb, ccb_h->tag_action); if( !target_negotiated(hcb) ) generate_negotiation_messages(hcb); Then set up the SCSI command. The command storage may be specified in the CCB in many interesting ways, specified by the CCB flags. The command buffer can be contained in CCB or pointed to, in the latter case the pointer may be physical or virtual. Since the hardware commonly needs physical address we always convert the address to the physical one. A NOT-QUITE RELATED NOTE: Normally this is done by a call to vtophys(), but for the PCI device (which account for most of the SCSI controllers now) drivers' portability to the Alpha architecture the conversion must be done by vtobus() instead due to special Alpha quirks. [IMHO it would be much better to have two separate functions, vtop() and ptobus() then vtobus() would be a simple superposition of them.] In case if a physical address is requested it is OK to return the CCB with the status CAM_REQ_INVALID, the current drivers do that. But it is also possible to compile the Alpha-specific piece of code, as in this example (there should be a more direct way to do that, without conditional compilation in the drivers). If necessary a physical address can be also converted or mapped back to a virtual address but with big pain, so we do not do that. if(ccb_h->flags & CAM_CDB_POINTER) { /* CDB is a pointer */ if(!(ccb_h->flags & CAM_CDB_PHYS)) { /* CDB pointer is virtual */ hcb->cmd = vtobus(csio->cdb_io.cdb_ptr); } else { /* CDB pointer is physical */ #if defined(__alpha__) hcb->cmd = csio->cdb_io.cdb_ptr | alpha_XXX_dmamap_or ; #else hcb->cmd = csio->cdb_io.cdb_ptr ; #endif } } else { /* CDB is in the ccb (buffer) */ hcb->cmd = vtobus(csio->cdb_io.cdb_bytes); } hcb->cmdlen = csio->cdb_len; Now it is time to set up the data. Again, the data storage may be specified in the CCB in many interesting ways, specified by the CCB flags. First we get the direction of the data transfer. The simplest case is if there is no data to transfer: int dir = (ccb_h->flags & CAM_DIR_MASK); if (dir == CAM_DIR_NONE) goto end_data; Then we check if the data is in one chunk or in a scatter-gather list, and the addresses are physical or virtual. The SCSI controller may be able to handle only a limited number of chunks of limited length. If the request hits this limitation we return an error. We use a special function to return the CCB to handle in one place the HCB resource shortages. The functions to add chunks are driver-dependent, and here we leave them without detailed implementation. See description of the SCSI command (CDB) handling for the details on the address-translation issues. If some variation is too difficult or impossible to implement with a particular card it is OK to return the status CAM_REQ_INVALID. Actually, it seems like the scatter-gather ability is not used anywhere in the CAM code now. But at least the case for a single non-scattered virtual buffer must be implemented, it is actively used by CAM. int rv; initialize_hcb_for_data(hcb); if((!(ccb_h->flags & CAM_SCATTER_VALID)) { /* single buffer */ if(!(ccb_h->flags & CAM_DATA_PHYS)) { rv = add_virtual_chunk(hcb, csio->data_ptr, csio->dxfer_len, dir); } } else { rv = add_physical_chunk(hcb, csio->data_ptr, csio->dxfer_len, dir); } } else { int i; struct bus_dma_segment *segs; segs = (struct bus_dma_segment *)csio->data_ptr; if ((ccb_h->flags & CAM_SG_LIST_PHYS) != 0) { /* The SG list pointer is physical */ rv = setup_hcb_for_physical_sg_list(hcb, segs, csio->sglist_cnt); } else if (!(ccb_h->flags & CAM_DATA_PHYS)) { /* SG buffer pointers are virtual */ for (i = 0; i < csio->sglist_cnt; i++) { rv = add_virtual_chunk(hcb, segs[i].ds_addr, segs[i].ds_len, dir); if (rv != CAM_REQ_CMP) break; } } else { /* SG buffer pointers are physical */ for (i = 0; i < csio->sglist_cnt; i++) { rv = add_physical_chunk(hcb, segs[i].ds_addr, segs[i].ds_len, dir); if (rv != CAM_REQ_CMP) break; } } } if(rv != CAM_REQ_CMP) { /* we expect that add_*_chunk() functions return CAM_REQ_CMP * if they added a chunk successfully, CAM_REQ_TOO_BIG if * the request is too big (too many bytes or too many chunks), * CAM_REQ_INVALID in case of other troubles */ free_hcb_and_ccb_done(hcb, ccb, rv); return; } end_data: If disconnection is disabled for this CCB we pass this information to the hcb: if(ccb_h->flags & CAM_DIS_DISCONNECT) hcb_disable_disconnect(hcb); If the controller is able to run REQUEST SENSE command all by itself then the value of the flag CAM_DIS_AUTOSENSE should also be passed to it, to prevent automatic REQUEST SENSE if the CAM subsystem does not want it. The only thing left is to set up the timeout, pass our hcb to the hardware and return, the rest will be done by the interrupt handler (or timeout handler). ccb_h->timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, (ccb_h->timeout * hz) / 1000); /* convert milliseconds to ticks */ put_hcb_into_hardware_queue(hcb); return; And here is a possible implementation of the function returning CCB: static void free_hcb_and_ccb_done(struct xxx_hcb *hcb, union ccb *ccb, u_int32_t status) { struct xxx_softc *softc = hcb->softc; ccb->ccb_h.ccb_hcb = 0; if(hcb != NULL) { untimeout(xxx_timeout, (caddr_t) hcb, ccb->ccb_h.timeout_ch); /* we're about to free a hcb, so the shortage has ended */ if(softc->flags & RESOURCE_SHORTAGE) { softc->flags &= ~RESOURCE_SHORTAGE; status |= CAM_RELEASE_SIMQ; } free_hcb(hcb); /* also removes hcb from any internal lists */ } ccb->ccb_h.status = status | (ccb->ccb_h.status & ~(CAM_STATUS_MASK|CAM_SIM_QUEUED)); xpt_done(ccb); } XPT_RESET_DEV - send the SCSI "BUS DEVICE RESET" message to a device There is no data transferred in CCB except the header and the most interesting argument of it is target_id. Depending on the controller hardware a hardware control block just like for the XPT_SCSI_IO request may be constructed (see XPT_SCSI_IO request description) and sent to the controller or the SCSI controller may be immediately programmed to send this RESET message to the device or this request may be just not supported (and return the status CAM_REQ_INVALID). Also on completion of the request all the disconnected transactions for this target must be aborted (probably in the interrupt routine). Also all the current negotiations for the target are lost on reset, so they might be cleaned too. Or they clearing may be deferred, because anyway the target would request re-negotiation on the next transaction. XPT_RESET_BUS - send the RESET signal to the SCSI bus No arguments are passed in the CCB, the only interesting argument is the SCSI bus indicated by the struct sim pointer. A minimalistic implementation would forget the SCSI negotiations for all the devices on the bus and return the status CAM_REQ_CMP. The proper implementation would in addition actually reset the SCSI bus (possible also reset the SCSI controller) and mark all the CCBs being processed, both those in the hardware queue and those being disconnected, as done with the status CAM_SCSI_BUS_RESET. Like: int targ, lun; struct xxx_hcb *h, *hh; struct ccb_trans_settings neg; struct cam_path *path; /* The SCSI bus reset may take a long time, in this case its completion * should be checked by interrupt or timeout. But for simplicity * we assume here that it's really fast. */ reset_scsi_bus(softc); /* drop all enqueued CCBs */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* the clean values of negotiations to report */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); /* drop all disconnected CCBs and clean negotiations */ for(targ=0; targ <= OUR_MAX_SUPPORTED_TARGET; targ++) { clean_negotiations(softc, targ); /* report the event if possible */ if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), targ, CAM_LUN_WILDCARD) == CAM_REQ_CMP) { xpt_async(AC_TRANSFER_NEG, path, &neg); xpt_free_path(path); } for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[targ][lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); /* report the event */ xpt_async(AC_BUS_RESET, softc->wpath, NULL); return; Implementing the SCSI bus reset as a function may be a good idea because it would be re-used by the timeout function as a last resort if the things go wrong. XPT_ABORT - abort the specified CCB The arguments are transferred in the instance "struct ccb_abort cab" of the union ccb. The only argument field in it is: abort_ccb - pointer to the CCB to be aborted If the abort is not supported just return the status CAM_UA_ABORT. This is also the easy way to minimally implement this call, return CAM_UA_ABORT in any case. The hard way is to implement this request honestly. First check that abort applies to a SCSI transaction: struct ccb *abort_ccb; abort_ccb = ccb->cab.abort_ccb; if(abort_ccb->ccb_h.func_code != XPT_SCSI_IO) { ccb->ccb_h.status = CAM_UA_ABORT; xpt_done(ccb); return; } Then it is necessary to find this CCB in our queue. This can be done by walking the list of all our hardware control blocks in search for one associated with this CCB: struct xxx_hcb *hcb, *h; hcb = NULL; /* We assume that softc->first_hcb is the head of the list of all * HCBs associated with this bus, including those enqueued for * processing, being processed by hardware and disconnected ones. */ for(h = softc->first_hcb; h != NULL; h = h->next) { if(h->ccb == abort_ccb) { hcb = h; break; } } if(hcb == NULL) { /* no such CCB in our queue */ ccb->ccb_h.status = CAM_PATH_INVALID; xpt_done(ccb); return; } hcb=found_hcb; Now we look at the current processing status of the HCB. It may be either sitting in the queue waiting to be sent to the SCSI bus, being transferred right now, or disconnected and waiting for the result of the command, or actually completed by hardware but not yet marked as done by software. To make sure that we do not get in any races with hardware we mark the HCB as being aborted, so that if this HCB is about to be sent to the SCSI bus the SCSI controller will see this flag and skip it. int hstatus; /* shown as a function, in case special action is needed to make * this flag visible to hardware */ set_hcb_flags(hcb, HCB_BEING_ABORTED); abort_again: hstatus = get_hcb_status(hcb); switch(hstatus) { case HCB_SITTING_IN_QUEUE: remove_hcb_from_hardware_queue(hcb); /* FALLTHROUGH */ case HCB_COMPLETED: /* this is an easy case */ free_hcb_and_ccb_done(hcb, abort_ccb, CAM_REQ_ABORTED); break; If the CCB is being transferred right now we would like to signal to the SCSI controller in some hardware-dependent way that we want to abort the current transfer. The SCSI controller would set the SCSI ATTENTION signal and when the target responds to it send an ABORT message. We also reset the timeout to make sure that the target is not sleeping forever. If the command would not get aborted in some reasonable time like 10 seconds the timeout routine would go ahead and reset the whole SCSI bus. Because the command will be aborted in some reasonable time we can just return the abort request now as successfully completed, and mark the aborted CCB as aborted (but not mark it as done yet). case HCB_BEING_TRANSFERRED: untimeout(xxx_timeout, (caddr_t) hcb, abort_ccb->ccb_h.timeout_ch); abort_ccb->ccb_h.timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, 10 * hz); abort_ccb->ccb_h.status = CAM_REQ_ABORTED; /* ask the controller to abort that HCB, then generate * an interrupt and stop */ if(signal_hardware_to_abort_hcb_and_stop(hcb) < 0) { /* oops, we missed the race with hardware, this transaction * got off the bus before we aborted it, try again */ goto abort_again; } break; If the CCB is in the list of disconnected then set it up as an abort request and re-queue it at the front of hardware queue. Reset the timeout and report the abort request to be completed. case HCB_DISCONNECTED: untimeout(xxx_timeout, (caddr_t) hcb, abort_ccb->ccb_h.timeout_ch); abort_ccb->ccb_h.timeout_ch = timeout(xxx_timeout, (caddr_t) hcb, 10 * hz); put_abort_message_into_hcb(hcb); put_hcb_at_the_front_of_hardware_queue(hcb); break; } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; That is all for the ABORT request, although there is one more issue. Because the ABORT message cleans all the ongoing transactions on a LUN we have to mark all the other active transactions on this LUN as aborted. That should be done in the interrupt routine, after the transaction gets aborted. Implementing the CCB abort as a function may be quite a good idea, this function can be re-used if an I/O transaction times out. The only difference would be that the timed out transaction would return the status CAM_CMD_TIMEOUT for the timed out request. Then the case XPT_ABORT would be small, like that: case XPT_ABORT: struct ccb *abort_ccb; abort_ccb = ccb->cab.abort_ccb; if(abort_ccb->ccb_h.func_code != XPT_SCSI_IO) { ccb->ccb_h.status = CAM_UA_ABORT; xpt_done(ccb); return; } if(xxx_abort_ccb(abort_ccb, CAM_REQ_ABORTED) < 0) /* no such CCB in our queue */ ccb->ccb_h.status = CAM_PATH_INVALID; else ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; XPT_SET_TRAN_SETTINGS - explicitly set values of SCSI transfer settings The arguments are transferred in the instance "struct ccb_trans_setting cts" of the union ccb: valid - a bitmask showing which settings should be updated: CCB_TRANS_SYNC_RATE_VALID - synchronous transfer rate CCB_TRANS_SYNC_OFFSET_VALID - synchronous offset CCB_TRANS_BUS_WIDTH_VALID - bus width CCB_TRANS_DISC_VALID - set enable/disable disconnection CCB_TRANS_TQ_VALID - set enable/disable tagged queuing flags - consists of two parts, binary arguments and identification of - sub-operations. The binary arguments are : + sub-operations. The binary arguments are: CCB_TRANS_DISC_ENB - enable disconnection CCB_TRANS_TAG_ENB - enable tagged queuing the sub-operations are: CCB_TRANS_CURRENT_SETTINGS - change the current negotiations CCB_TRANS_USER_SETTINGS - remember the desired user values sync_period, sync_offset - self-explanatory, if sync_offset==0 then the asynchronous mode is requested bus_width - bus width, in bits (not bytes) Two sets of negotiated parameters are supported, the user settings and the current settings. The user settings are not really used much in the SIM drivers, this is mostly just a piece of memory where the upper levels can store (and later recall) its ideas about the parameters. Setting the user parameters does not cause re-negotiation of the transfer rates. But when the SCSI controller does a negotiation it must never set the values higher than the user parameters, so it is essentially the top boundary. The current settings are, as the name says, current. Changing them means that the parameters must be re-negotiated on the next transfer. Again, these "new current settings" are not supposed to be forced on the device, just they are used as the initial step of negotiations. Also they must be limited by actual capabilities of the SCSI controller: for example, if the SCSI controller has 8-bit bus and the request asks to set 16-bit wide transfers this parameter must be silently truncated to 8-bit transfers before sending it to the device. One caveat is that the bus width and synchronous parameters are per target while the disconnection and tag enabling parameters are per lun. The recommended implementation is to keep 3 sets of negotiated (bus width and synchronous transfer) parameters: user - the user set, as above current - those actually in effect goal - those requested by setting of the "current" parameters The code looks like: struct ccb_trans_settings *cts; int targ, lun; int flags; cts = &ccb->cts; targ = ccb_h->target_id; lun = ccb_h->target_lun; flags = cts->flags; if(flags & CCB_TRANS_USER_SETTINGS) { if(flags & CCB_TRANS_SYNC_RATE_VALID) softc->user_sync_period[targ] = cts->sync_period; if(flags & CCB_TRANS_SYNC_OFFSET_VALID) softc->user_sync_offset[targ] = cts->sync_offset; if(flags & CCB_TRANS_BUS_WIDTH_VALID) softc->user_bus_width[targ] = cts->bus_width; if(flags & CCB_TRANS_DISC_VALID) { softc->user_tflags[targ][lun] &= ~CCB_TRANS_DISC_ENB; softc->user_tflags[targ][lun] |= flags & CCB_TRANS_DISC_ENB; } if(flags & CCB_TRANS_TQ_VALID) { softc->user_tflags[targ][lun] &= ~CCB_TRANS_TQ_ENB; softc->user_tflags[targ][lun] |= flags & CCB_TRANS_TQ_ENB; } } if(flags & CCB_TRANS_CURRENT_SETTINGS) { if(flags & CCB_TRANS_SYNC_RATE_VALID) softc->goal_sync_period[targ] = max(cts->sync_period, OUR_MIN_SUPPORTED_PERIOD); if(flags & CCB_TRANS_SYNC_OFFSET_VALID) softc->goal_sync_offset[targ] = min(cts->sync_offset, OUR_MAX_SUPPORTED_OFFSET); if(flags & CCB_TRANS_BUS_WIDTH_VALID) softc->goal_bus_width[targ] = min(cts->bus_width, OUR_BUS_WIDTH); if(flags & CCB_TRANS_DISC_VALID) { softc->current_tflags[targ][lun] &= ~CCB_TRANS_DISC_ENB; softc->current_tflags[targ][lun] |= flags & CCB_TRANS_DISC_ENB; } if(flags & CCB_TRANS_TQ_VALID) { softc->current_tflags[targ][lun] &= ~CCB_TRANS_TQ_ENB; softc->current_tflags[targ][lun] |= flags & CCB_TRANS_TQ_ENB; } } ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; Then when the next I/O request will be processed it will check if it has to re-negotiate, for example by calling the function target_negotiated(hcb). It can be implemented like this: int target_negotiated(struct xxx_hcb *hcb) { struct softc *softc = hcb->softc; int targ = hcb->targ; if( softc->current_sync_period[targ] != softc->goal_sync_period[targ] || softc->current_sync_offset[targ] != softc->goal_sync_offset[targ] || softc->current_bus_width[targ] != softc->goal_bus_width[targ] ) return 0; /* FALSE */ else return 1; /* TRUE */ } After the values are re-negotiated the resulting values must be assigned to both current and goal parameters, so for future I/O transactions the current and goal parameters would be the same and target_negotiated() would return TRUE. When the card is initialized (in xxx_attach()) the current negotiation values must be initialized to narrow asynchronous mode, the goal and current values must be initialized to the maximal values supported by controller. XPT_GET_TRAN_SETTINGS - get values of SCSI transfer settings This operations is the reverse of XPT_SET_TRAN_SETTINGS. Fill up the CCB instance "struct ccb_trans_setting cts" with data as requested by the flags CCB_TRANS_CURRENT_SETTINGS or CCB_TRANS_USER_SETTINGS (if both are set then the existing drivers return the current settings). Set all the bits in the valid field. XPT_CALC_GEOMETRY - calculate logical (BIOS) geometry of the disk The arguments are transferred in the instance "struct ccb_calc_geometry ccg" of the union ccb: block_size - input, block (A.K.A sector) size in bytes volume_size - input, volume size in bytes cylinders - output, logical cylinders heads - output, logical heads secs_per_track - output, logical sectors per track If the returned geometry differs much enough from what the SCSI controller BIOS thinks and a disk on this SCSI controller is used as bootable the system may not be able to boot. The typical calculation example taken from the aic7xxx driver is: struct ccb_calc_geometry *ccg; u_int32_t size_mb; u_int32_t secs_per_cylinder; int extended; ccg = &ccb->ccg; size_mb = ccg->volume_size / ((1024L * 1024L) / ccg->block_size); extended = check_cards_EEPROM_for_extended_geometry(softc); if (size_mb > 1024 && extended) { ccg->heads = 255; ccg->secs_per_track = 63; } else { ccg->heads = 64; ccg->secs_per_track = 32; } secs_per_cylinder = ccg->heads * ccg->secs_per_track; ccg->cylinders = ccg->volume_size / secs_per_cylinder; ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); return; This gives the general idea, the exact calculation depends on the quirks of the particular BIOS. If BIOS provides no way set the "extended translation" flag in EEPROM this flag should normally be assumed equal to 1. Other popular geometries are: 128 heads, 63 sectors - Symbios controllers 16 heads, 63 sectors - old controllers Some system BIOSes and SCSI BIOSes fight with each other with variable success, for example a combination of Symbios 875/895 SCSI and Phoenix BIOS can give geometry 128/63 after power up and 255/63 after a hard reset or soft reboot. XPT_PATH_INQ - path inquiry, in other words get the SIM driver and SCSI controller (also known as HBA - Host Bus Adapter) properties The properties are returned in the instance "struct ccb_pathinq cpi" of the union ccb: version_num - the SIM driver version number, now all drivers use 1 hba_inquiry - bitmask of features supported by the controller: PI_MDP_ABLE - supports MDP message (something from SCSI3?) PI_WIDE_32 - supports 32 bit wide SCSI PI_WIDE_16 - supports 16 bit wide SCSI PI_SDTR_ABLE - can negotiate synchronous transfer rate PI_LINKED_CDB - supports linked commands PI_TAG_ABLE - supports tagged commands PI_SOFT_RST - supports soft reset alternative (hard reset and soft reset are mutually exclusive within a SCSI bus) target_sprt - flags for target mode support, 0 if unsupported hba_misc - miscellaneous controller features: PIM_SCANHILO - bus scans from high ID to low ID PIM_NOREMOVE - removable devices not included in scan PIM_NOINITIATOR - initiator role not supported PIM_NOBUSRESET - user has disabled initial BUS RESET hba_eng_cnt - mysterious HBA engine count, something related to compression, now is always set to 0 vuhba_flags - vendor-unique flags, unused now max_target - maximal supported target ID (7 for 8-bit bus, 15 for 16-bit bus, 127 for Fibre Channel) max_lun - maximal supported LUN ID (7 for older SCSI controllers, 63 for newer ones) async_flags - bitmask of installed Async handler, unused now hpath_id - highest Path ID in the subsystem, unused now unit_number - the controller unit number, cam_sim_unit(sim) bus_id - the bus number, cam_sim_bus(sim) initiator_id - the SCSI ID of the controller itself base_transfer_speed - nominal transfer speed in KB/s for asynchronous narrow transfers, equals to 3300 for SCSI sim_vid - SIM driver's vendor id, a zero-terminated string of maximal length SIM_IDLEN including the terminating zero hba_vid - SCSI controller's vendor id, a zero-terminated string of maximal length HBA_IDLEN including the terminating zero dev_name - device driver name, a zero-terminated string of maximal length DEV_IDLEN including the terminating zero, equal to cam_sim_name(sim) The recommended way of setting the string fields is using strncpy, like: strncpy(cpi->dev_name, cam_sim_name(sim), DEV_IDLEN); After setting the values set the status to CAM_REQ_CMP and mark the CCB as done. Polling static void xxx_poll struct cam_sim *sim The poll function is used to simulate the interrupts when the interrupt subsystem is not functioning (for example, when the system has crashed and is creating the system dump). The CAM subsystem sets the proper interrupt level before calling the poll routine. So all it needs to do is to call the interrupt routine (or the other way around, the poll routine may be doing the real action and the interrupt routine would just call the - poll routine). Why bother about a separate function then ? + poll routine). Why bother about a separate function then? Because of different calling conventions. The xxx_poll routine gets the struct cam_sim pointer as its argument when the PCI interrupt routine by common convention gets pointer to the struct xxx_softc and the ISA interrupt routine gets just the device unit number. So the poll routine would normally look as: static void xxx_poll(struct cam_sim *sim) { xxx_intr((struct xxx_softc *)cam_sim_softc(sim)); /* for PCI device */ } or static void xxx_poll(struct cam_sim *sim) { xxx_intr(cam_sim_unit(sim)); /* for ISA device */ } Asynchronous Events If an asynchronous event callback has been set up then the callback function should be defined. static void ahc_async(void *callback_arg, u_int32_t code, struct cam_path *path, void *arg) callback_arg - the value supplied when registering the callback code - identifies the type of event path - identifies the devices to which the event applies arg - event-specific argument Implementation for a single type of event, AC_LOST_DEVICE, looks like: struct xxx_softc *softc; struct cam_sim *sim; int targ; struct ccb_trans_settings neg; sim = (struct cam_sim *)callback_arg; softc = (struct xxx_softc *)cam_sim_softc(sim); switch (code) { case AC_LOST_DEVICE: targ = xpt_path_target_id(path); if(targ <= OUR_MAX_SUPPORTED_TARGET) { clean_negotiations(softc, targ); /* send indication to CAM */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); xpt_async(AC_TRANSFER_NEG, path, &neg); } break; default: break; } Interrupts The exact type of the interrupt routine depends on the type of the peripheral bus (PCI, ISA and so on) to which the SCSI controller is connected. The interrupt routines of the SIM drivers run at the interrupt level splcam. So splcam() should be used in the driver to synchronize activity between the interrupt routine and the rest of the driver (for a multiprocessor-aware driver things get yet more interesting but we ignore this case here). The pseudo-code in this document happily ignores the problems of synchronization. The real code must not ignore them. A simple-minded approach is to set splcam() on the entry to the other routines and reset it on return thus protecting them by one big critical section. To make sure that the interrupt level will be always restored a wrapper function can be defined, like: static void xxx_action(struct cam_sim *sim, union ccb *ccb) { int s; s = splcam(); xxx_action1(sim, ccb); splx(s); } static void xxx_action1(struct cam_sim *sim, union ccb *ccb) { ... process the request ... } This approach is simple and robust but the problem with it is that interrupts may get blocked for a relatively long time and this would negatively affect the system's performance. On the other hand the functions of the spl() family have rather high overhead, so vast amount of tiny critical sections may not be good either. The conditions handled by the interrupt routine and the details depend very much on the hardware. We consider the set of "typical" conditions. First, we check if a SCSI reset was encountered on the bus (probably caused by another SCSI controller on the same SCSI bus). If so we drop all the enqueued and disconnected requests, report the events and re-initialize our SCSI controller. It is important that during this initialization the controller will not issue another reset or else two controllers on the same SCSI bus could ping-pong resets forever. The case of fatal controller error/hang could be handled in the same place, but it will probably need also sending RESET signal to the SCSI bus to reset the status of the connections with the SCSI devices. int fatal=0; struct ccb_trans_settings neg; struct cam_path *path; if( detected_scsi_reset(softc) || (fatal = detected_fatal_controller_error(softc)) ) { int targ, lun; struct xxx_hcb *h, *hh; /* drop all enqueued CCBs */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* the clean values of negotiations to report */ neg.bus_width = 8; neg.sync_period = neg.sync_offset = 0; neg.valid = (CCB_TRANS_BUS_WIDTH_VALID | CCB_TRANS_SYNC_RATE_VALID | CCB_TRANS_SYNC_OFFSET_VALID); /* drop all disconnected CCBs and clean negotiations */ for(targ=0; targ <= OUR_MAX_SUPPORTED_TARGET; targ++) { clean_negotiations(softc, targ); /* report the event if possible */ if(xpt_create_path(&path, /*periph*/NULL, cam_sim_path(sim), targ, CAM_LUN_WILDCARD) == CAM_REQ_CMP) { xpt_async(AC_TRANSFER_NEG, path, &neg); xpt_free_path(path); } for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[targ][lun]; h != NULL; h = hh) { hh=h->next; if(fatal) free_hcb_and_ccb_done(h, h->ccb, CAM_UNREC_HBA_ERROR); else free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } } /* report the event */ xpt_async(AC_BUS_RESET, softc->wpath, NULL); /* re-initialization may take a lot of time, in such case * its completion should be signaled by another interrupt or * checked on timeout - but for simplicity we assume here that * it's really fast */ if(!fatal) { reinitialize_controller_without_scsi_reset(softc); } else { reinitialize_controller_with_scsi_reset(softc); } schedule_next_hcb(softc); return; } If interrupt is not caused by a controller-wide condition then probably something has happened to the current hardware control block. Depending on the hardware there may be other non-HCB-related events, we just do not consider them here. Then we analyze what happened to this HCB: struct xxx_hcb *hcb, *h, *hh; int hcb_status, scsi_status; int ccb_status; int targ; int lun_to_freeze; hcb = get_current_hcb(softc); if(hcb == NULL) { /* either stray interrupt or something went very wrong * or this is something hardware-dependent */ handle as necessary; return; } targ = hcb->target; hcb_status = get_status_of_current_hcb(softc); First we check if the HCB has completed and if so we check the returned SCSI status. if(hcb_status == COMPLETED) { scsi_status = get_completion_status(hcb); Then look if this status is related to the REQUEST SENSE command and if so handle it in a simple way. if(hcb->flags & DOING_AUTOSENSE) { if(scsi_status == GOOD) { /* autosense was successful */ hcb->ccb->ccb_h.status |= CAM_AUTOSNS_VALID; free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_SCSI_STATUS_ERROR); } else { autosense_failed: free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_AUTOSENSE_FAIL); } schedule_next_hcb(softc); return; } Else the command itself has completed, pay more attention to details. If auto-sense is not disabled for this CCB and the command has failed with sense data then run REQUEST SENSE command to receive that data. hcb->ccb->csio.scsi_status = scsi_status; calculate_residue(hcb); if( (hcb->ccb->ccb_h.flags & CAM_DIS_AUTOSENSE)==0 && ( scsi_status == CHECK_CONDITION || scsi_status == COMMAND_TERMINATED) ) { /* start auto-SENSE */ hcb->flags |= DOING_AUTOSENSE; setup_autosense_command_in_hcb(hcb); restart_current_hcb(softc); return; } if(scsi_status == GOOD) free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_REQ_CMP); else free_hcb_and_ccb_done(hcb, hcb->ccb, CAM_SCSI_STATUS_ERROR); schedule_next_hcb(softc); return; } One typical thing would be negotiation events: negotiation messages received from a SCSI target (in answer to our negotiation attempt or by target's initiative) or the target is unable to negotiate (rejects our negotiation messages or does not answer them). switch(hcb_status) { case TARGET_REJECTED_WIDE_NEG: /* revert to 8-bit bus */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = 8; /* report the event */ neg.bus_width = 8; neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); continue_current_hcb(softc); return; case TARGET_ANSWERED_WIDE_NEG: { int wd; wd = get_target_bus_width_request(softc); if(wd <= softc->goal_bus_width[targ]) { /* answer is acceptable */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = neg.bus_width = wd; /* report the event */ neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); } else { prepare_reject_message(hcb); } } continue_current_hcb(softc); return; case TARGET_REQUESTED_WIDE_NEG: { int wd; wd = get_target_bus_width_request(softc); wd = min (wd, OUR_BUS_WIDTH); wd = min (wd, softc->user_bus_width[targ]); if(wd != softc->current_bus_width[targ]) { /* the bus width has changed */ softc->current_bus_width[targ] = softc->goal_bus_width[targ] = neg.bus_width = wd; /* report the event */ neg.valid = CCB_TRANS_BUS_WIDTH_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); } prepare_width_nego_rsponse(hcb, wd); } continue_current_hcb(softc); return; } Then we handle any errors that could have happened during auto-sense in the same simple-minded way as before. Otherwise we look closer at the details again. if(hcb->flags & DOING_AUTOSENSE) goto autosense_failed; switch(hcb_status) { The next event we consider is unexpected disconnect. Which is considered normal after an ABORT or BUS DEVICE RESET message and abnormal in other cases. case UNEXPECTED_DISCONNECT: if(requested_abort(hcb)) { /* abort affects all commands on that target+LUN, so * mark all disconnected HCBs on that target+LUN as aborted too */ for(h = softc->first_discon_hcb[hcb->target][hcb->lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_REQ_ABORTED); } ccb_status = CAM_REQ_ABORTED; } else if(requested_bus_device_reset(hcb)) { int lun; /* reset affects all commands on that target, so * mark all disconnected HCBs on that target+LUN as reset */ for(lun=0; lun <= OUR_MAX_SUPPORTED_LUN; lun++) for(h = softc->first_discon_hcb[hcb->target][lun]; h != NULL; h = hh) { hh=h->next; free_hcb_and_ccb_done(h, h->ccb, CAM_SCSI_BUS_RESET); } /* send event */ xpt_async(AC_SENT_BDR, hcb->ccb->ccb_h.path_id, NULL); /* this was the CAM_RESET_DEV request itself, it's completed */ ccb_status = CAM_REQ_CMP; } else { calculate_residue(hcb); ccb_status = CAM_UNEXP_BUSFREE; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = hcb->lun; } break; If the target refuses to accept tags we notify CAM about that and return back all commands for this LUN: case TAGS_REJECTED: /* report the event */ neg.flags = 0 & ~CCB_TRANS_TAG_ENB; neg.valid = CCB_TRANS_TQ_VALID; xpt_async(AC_TRANSFER_NEG, hcb->ccb.ccb_h.path_id, &neg); ccb_status = CAM_MSG_REJECT_REC; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = hcb->lun; break; Then we check a number of other conditions, with processing basically limited to setting the CCB status: case SELECTION_TIMEOUT: ccb_status = CAM_SEL_TIMEOUT; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = CAM_LUN_WILDCARD; break; case PARITY_ERROR: ccb_status = CAM_UNCOR_PARITY; break; case DATA_OVERRUN: case ODD_WIDE_TRANSFER: ccb_status = CAM_DATA_RUN_ERR; break; default: /* all other errors are handled in a generic way */ ccb_status = CAM_REQ_CMP_ERR; /* request the further code to freeze the queue */ hcb->ccb->ccb_h.status |= CAM_DEV_QFRZN; lun_to_freeze = CAM_LUN_WILDCARD; break; } Then we check if the error was serious enough to freeze the input queue until it gets proceeded and do so if it is: if(hcb->ccb->ccb_h.status & CAM_DEV_QFRZN) { /* freeze the queue */ xpt_freeze_devq(ccb->ccb_h.path, /*count*/1); /* re-queue all commands for this target/LUN back to CAM */ for(h = softc->first_queued_hcb; h != NULL; h = hh) { hh = h->next; if(targ == h->targ && (lun_to_freeze == CAM_LUN_WILDCARD || lun_to_freeze == h->lun) ) free_hcb_and_ccb_done(h, h->ccb, CAM_REQUEUE_REQ); } } free_hcb_and_ccb_done(hcb, hcb->ccb, ccb_status); schedule_next_hcb(softc); return; This concludes the generic interrupt handling although specific controllers may require some additions. Errors Summary When executing an I/O request many things may go wrong. The reason of error can be reported in the CCB status with great detail. Examples of use are spread throughout this document. For completeness here is the summary of recommended responses for the typical error conditions: CAM_RESRC_UNAVAIL - some resource is temporarily unavailable and the SIM driver cannot generate an event when it will become available. An example of this resource would be some intra-controller hardware resource for which the controller does not generate an interrupt when it becomes available. CAM_UNCOR_PARITY - unrecovered parity error occurred CAM_DATA_RUN_ERR - data overrun or unexpected data phase (going in other direction than specified in CAM_DIR_MASK) or odd transfer length for wide transfer CAM_SEL_TIMEOUT - selection timeout occurred (target does not respond) CAM_CMD_TIMEOUT - command timeout occurred (the timeout function ran) CAM_SCSI_STATUS_ERROR - the device returned error CAM_AUTOSENSE_FAIL - the device returned error and the REQUEST SENSE COMMAND failed CAM_MSG_REJECT_REC - MESSAGE REJECT message was received CAM_SCSI_BUS_RESET - received SCSI bus reset CAM_REQ_CMP_ERR - "impossible" SCSI phase occurred or something else as weird or just a generic error if further detail is not available CAM_UNEXP_BUSFREE - unexpected disconnect occurred CAM_BDR_SENT - BUS DEVICE RESET message was sent to the target CAM_UNREC_HBA_ERROR - unrecoverable Host Bus Adapter Error CAM_REQ_TOO_BIG - the request was too large for this controller CAM_REQUEUE_REQ - this request should be re-queued to preserve transaction ordering. This typically occurs when the SIM recognizes an error that should freeze the queue and must place other queued requests for the target at the sim level back into the XPT queue. Typical cases of such errors are selection timeouts, command timeouts and other like conditions. In such cases the troublesome command returns the status indicating the error, the and the other commands which have not be sent to the bus yet get re-queued. CAM_LUN_INVALID - the LUN ID in the request is not supported by the SCSI controller CAM_TID_INVALID - the target ID in the request is not supported by the SCSI controller Timeout Handling When the timeout for an HCB expires that request should be aborted, just like with an XPT_ABORT request. The only difference is that the returned status of aborted request should be CAM_CMD_TIMEOUT instead of CAM_REQ_ABORTED (that is why implementation of the abort better be done as a function). But there is one more possible problem: what if the abort request itself will get stuck? In this case the SCSI bus should be reset, just like with an XPT_RESET_BUS request (and the idea about implementing it as a function called from both places applies here too). Also we should reset the whole SCSI bus if a device reset request got stuck. So after all the timeout function would look like: static void xxx_timeout(void *arg) { struct xxx_hcb *hcb = (struct xxx_hcb *)arg; struct xxx_softc *softc; struct ccb_hdr *ccb_h; softc = hcb->softc; ccb_h = &hcb->ccb->ccb_h; if(hcb->flags & HCB_BEING_ABORTED || ccb_h->func_code == XPT_RESET_DEV) { xxx_reset_bus(softc); } else { xxx_abort_ccb(hcb->ccb, CAM_CMD_TIMEOUT); } } When we abort a request all the other disconnected requests to the same target/LUN get aborted too. So there appears a question, should we return them with status CAM_REQ_ABORTED or - CAM_CMD_TIMEOUT ? The current drivers use CAM_CMD_TIMEOUT. This + CAM_CMD_TIMEOUT? The current drivers use CAM_CMD_TIMEOUT. This seems logical because if one request got timed out then probably something really bad is happening to the device, so if they would not be disturbed they would time out by themselves. diff --git a/en_US.ISO8859-1/books/developers-handbook/secure/chapter.sgml b/en_US.ISO8859-1/books/developers-handbook/secure/chapter.sgml index ad82dadf93..a8c1eb3e18 100644 --- a/en_US.ISO8859-1/books/developers-handbook/secure/chapter.sgml +++ b/en_US.ISO8859-1/books/developers-handbook/secure/chapter.sgml @@ -1,516 +1,517 @@ Secure Programming This chapter was written by &a.murray;. Synopsis This chapter describes some of the security issues that have plagued Unix programmers for decades and some of the new tools available to help programmers avoid writing exploitable code. Secure Design Methodology Writing secure applications takes a very scrutinous and pessimistic outlook on life. Applications should be run with the principle of least privilege so that no process is ever running with more than the bare minimum access that it needs to accomplish its function. Previously tested code should be reused whenever possible to avoid common mistakes that others may have already fixed. One of the pitfalls of the Unix environment is how easy it is to make assumptions about the sanity of the environment. Applications should never trust user input (in all its forms), system resources, inter-process communication, or the timing of events. Unix processes do not execute synchronously so logical operations are rarely atomic. Buffer Overflows Buffer Overflows have been around since the very beginnings of the Von-Neuman architecture. buffer overflow Von-Neuman They first gained widespread notoriety in 1988 with the Morris Internet worm. Unfortunately, the same basic attack remains Morris Internet worm effective today. Of the 17 CERT security advisories of 1999, 10 CERTsecurity advisories of them were directly caused by buffer-overflow software bugs. By far the most common type of buffer overflow attack is based on corrupting the stack. stack arguments Most modern computer systems use a stack to pass arguments to procedures and to store local variables. A stack is a last in first out (LIFO) buffer in the high memory area of a process image. When a program invokes a function a new "stack frame" is LIFO process image stack pointer created. This stack frame consists of the arguments passed to the function as well as a dynamic amount of local variable space. The "stack pointer" is a register that holds the current stack frame stack pointer location of the top of the stack. Since this value is constantly changing as new values are pushed onto the top of the stack, many implementations also provide a "frame pointer" that is located near the beginning of a stack frame so that local variables can more easily be addressed relative to this value. The return address for function frame pointer process image frame pointer return address stack-overflow calls is also stored on the stack, and this is the cause of stack-overflow exploits since overflowing a local variable in a function can overwrite the return address of that function, potentially allowing a malicious user to execute any code he or she wants. Although stack-based attacks are by far the most common, it would also be possible to overrun the stack with a heap-based (malloc/free) attack. The C programming language does not perform automatic bounds checking on arrays or pointers as many other languages do. In addition, the standard C library is filled with a handful of very dangerous functions. strcpy(char *dest, const char *src) May overflow the dest buffer strcat(char *dest, const char *src) May overflow the dest buffer getwd(char *buf) May overflow the buf buffer gets(char *s) May overflow the s buffer [vf]scanf(const char *format, ...) May overflow its arguments. realpath(char *path, char resolved_path[]) May overflow the path buffer [v]sprintf(char *str, const char *format, ...) May overflow the str buffer. Example Buffer Overflow The following example code contains a buffer overflow designed to overwrite the return address and skip the instruction immediately following the function call. (Inspired by ) #include stdio.h void manipulate(char *buffer) { char newbuffer[80]; strcpy(newbuffer,buffer); } int main() { char ch,buffer[4096]; int i=0; while ((buffer[i++] = getchar()) != '\n') {}; i=1; manipulate(buffer); i=2; printf("The value of i is : %d\n",i); return 0; } Let us examine what the memory image of this process would look like if we were to input 160 spaces into our little program before hitting return. [XXX figure here!] Obviously more malicious input can be devised to execute actual compiled instructions (such as exec(/bin/sh)). Avoiding Buffer Overflows The most straightforward solution to the problem of stack-overflows is to always use length restricted memory and string copy functions. strncpy and strncat are part of the standard C library. string copy functions strncpy string copy functions strncat These functions accept a length value as a parameter which should be no larger than the size of the destination buffer. These functions will then copy up to `length' bytes from the source to the destination. However there are a number of problems with these functions. Neither function guarantees NUL termination if the size of the input buffer is as large as the NUL termination destination. The length parameter is also used inconsistently between strncpy and strncat so it is easy for programmers to get confused as to their proper usage. There is also a significant performance loss compared to strcpy when copying a short string into a large buffer since strncpy NUL fills up the size specified. In OpenBSD, another memory copy implementation has been OpenBSD created to get around these problem. The strlcpy and strlcat functions guarantee that they will always null terminate the destination string when given a non-zero length argument. For more information about these functions see . The OpenBSD strlcpy and strlcat instructions have been in FreeBSD since 3.3. string copy functions strlcpy string copy functions strlcat Compiler based run-time bounds checking bounds checking compiler-based Unfortunately there is still a very large assortment of code in public use which blindly copies memory around without using any of the bounded copy routines we just discussed. Fortunately, there is another solution. Several compiler add-ons and libraries exist to do Run-time bounds checking in C/C++. StackGuard gcc StackGuard is one such add-on that is implemented as a small patch to the gcc code generator. From the StackGuard website:
"StackGuard detects and defeats stack smashing attacks by protecting the return address on the stack from being altered. StackGuard places a "canary" word next to the return address when a function is called. If the canary word has been altered when the function returns, then a stack smashing attack has been attempted, and the program responds by emitting an intruder alert into syslog, and then halts."
"StackGuard is implemented as a small patch to the gcc code generator, specifically the function_prolog() and function_epilog() routines. function_prolog() has been enhanced to lay down canaries on the stack when functions start, and function_epilog() checks canary integrity when the function exits. Any attempt at corrupting the return address is thus detected before the function returns."
 buffer overflow Recompiling your application with StackGuard is an effective means of stopping most buffer-overflow attacks, but it can still be compromised. Library based run-time bounds checking bounds checking library-based Compiler-based mechanisms are completely useless for binary-only software for which you cannot recompile. For these situations there are a number of libraries which re-implement the unsafe functions of the C-library (strcpy, fscanf, getwd, etc..) and ensure that these functions can never write past the stack pointer. libsafe libverify libparnoia Unfortunately these library-based defenses have a number of shortcomings. These libraries only protect against a very small set of security related issues and they neglect to fix the actual problem. These defenses may fail if the application was compiled with -fomit-frame-pointer. Also, the LD_PRELOAD and LD_LIBRARY_PATH environment variables can be overwritten/unset by the user. SetUID issues seteuid There are at least 6 different IDs associated with any given process. Because of this you have to be very careful with the access that your process has at any given time. In particular, all seteuid applications should give up their privileges as soon as it is no longer required. user IDs real user ID user IDs effective user ID The real user ID can only be changed by a superuser process. The login program sets this when a user initially logs in and it is seldom changed. The effective user ID is set by the exec() functions if a program has its seteuid bit set. An application can call seteuid() at any time to set the effective user ID to either the real user ID or the saved set-user-ID. When the effective user ID is set by exec() functions, the previous value is saved in the saved set-user-ID. Limiting your program's environment chroot() The traditional method of restricting a process is with the chroot() system call. This system call changes the root directory from which all other paths are referenced for a process and any child processes. For this call to succeed the process must have execute (search) permission on the directory being referenced. The new environment does not actually take effect until you chdir() into your new environment. It should also be noted that a process can easily break out of a chroot environment if it has root privilege. This could be accomplished by creating device nodes to read kernel memory, attaching a debugger to a process outside of the jail, or in many other creative ways. The behavior of the chroot() system call can be controlled somewhat with the kern.chroot_allow_open_directories sysctl variable. When this value is set to 0, chroot() will fail with EPERM if there are any directories open. If set to the default value of 1, then chroot() will fail with EPERM if there are any directories open and the process is already subject to a chroot() call. For any other value, the check for open directories will be bypassed completely. FreeBSD's jail functionality jail The concept of a Jail extends upon the chroot() by limiting the powers of the superuser to create a true `virtual server'. Once a prison is setup all network communication must take place through the specified IP address, and the power of "root privilege" in this jail is severely constrained. While in a prison, any tests of superuser power within the kernel using the suser() call will fail. However, some calls to suser() have been changed to a new interface suser_xxx(). This function is responsible for recognizing or denying access to superuser power for imprisoned processes. A superuser process within a jailed environment has the - power to : + power to: + Manipulate credential with setuid, seteuid, setgid, setegid, setgroups, setreuid, setregid, setlogin Set resource limits with setrlimit Modify some sysctl nodes (kern.hostname) chroot() Set flags on a vnode: chflags, fchflags Set attributes of a vnode such as file permission, owner, group, size, access time, and modification time. Bind to privileged ports in the Internet domain (ports < 1024) Jail is a very useful tool for running applications in a secure environment but it does have some shortcomings. Currently, the IPC mechanisms have not been converted to the suser_xxx so applications such as MySQL cannot be run within a jail. Superuser access may have a very limited meaning within a jail, but there is no way to specify exactly what "very limited" means. POSIX.1e Process Capabilities POSIX.1e Process Capabilities TrustedBSD Posix has released a working draft that adds event auditing, access control lists, fine grained privileges, information labeling, and mandatory access control. This is a work in progress and is the focus of the TrustedBSD project. Some of the initial work has been committed to FreeBSD-current (cap_set_proc(3)). Trust An application should never assume that anything about the users environment is sane. This includes (but is certainly not - limited to) : user input, signals, environment variables, + limited to): user input, signals, environment variables, resources, IPC, mmaps, the file system working directory, file descriptors, the # of open files, etc. positive filtering data validation You should never assume that you can catch all forms of invalid input that a user might supply. Instead, your application should use positive filtering to only allow a specific subset of inputs that you deem safe. Improper data validation has been the cause of many exploits, especially with CGI scripts on the world wide web. For filenames you need to be extra careful about paths ("../", "/"), symbolic links, and shell escape characters. Perl Taint mode Perl has a really cool feature called "Taint" mode which can be used to prevent scripts from using data derived outside the program in an unsafe way. This mode will check command line arguments, environment variables, locale information, the results of certain syscalls (readdir(), readlink(), getpwxxx(), and all file input. Race Conditions A race condition is anomalous behavior caused by the unexpected dependence on the relative timing of events. In other words, a programmer incorrectly assumed that a particular event would always happen before another. race conditions signals race conditions access checks race conditions file opens Some of the common causes of race conditions are signals, access checks, and file opens. Signals are asynchronous events by nature so special care must be taken in dealing with them. Checking access with access(2) then open(2) is clearly non-atomic. Users can move files in between the two calls. Instead, privileged applications should seteuid() and then call open() directly. Along the same lines, an application should always set a proper umask before open() to obviate the need for spurious chmod() calls.