Annotation of wikisrc/tutorials/bus_space_tutorial.mdwn, revision 1.16
1.9 mspo 1: [[!toc ]]
1.1 mspo 2:
1.3 mspo 3: ## Introduction
1.1 mspo 4:
5: ### Why was this tutorial created?
6:
7: - Introductory-level documentation is scarce
8: - Writing device drivers is often considered black magic
1.13 ryoon 9: - Reading the man pages won't give you the big picture
1.1 mspo 10: - BSD systems are always in need of new drivers
11: - Device drivers are fun
12:
1.13 ryoon 13: ### What won't be covered here?
1.1 mspo 14:
1.13 ryoon 15: We don't have much time, so several ~~advanced~~ topics were omitted:
1.1 mspo 16:
17: - Interrupt handling
18: - Direct Memory Access and the bus\_dma framework
19: - Power management
20: - Driver detachment
21: - Drivers as kernel modules
22: - Examples for buses other than PCI
23: - Pretty much everything else...
24:
25: However, once you finish this tutorial, you should be able to pursue
26: this knowledge yourself.
27:
28: ### What is a driver anyway?
29:
30: - The interface between user space and hardware, implemented as a part
31: of the kernel
32: - The NetBSD drivers are written mostly in C
33: - Sometimes they have machine dependent assembler parts, but this is a
34: rare case
35:
36: ### What do you need to write a driver?
37:
38: - C programming skills
39: - Hardware documentation (or the ability to reverse engineer the
40: hardware)
41: - A reference driver implementation will help but is not essential
42: - A NetBSD installation and kernel source, or a cross-build
43: environment (the latter is usually preferred for development of
44: drivers)
45: - A lot of time, coffee and patience
46:
47: ### Why is writing the device drivers considered difficult?
48:
1.13 ryoon 49: - It's not as difficult as you may expect, in fact during this
50: tutorial we'll prove that it's quite easy
1.1 mspo 51: - You need to think on a very low level
52: - Good understanding of computer architecture is a must
1.13 ryoon 53: - Often documentation is the main problem - writing the driver is not
54: possible if you don't understand how the device works
1.1 mspo 55: - No access to documentation (uncooperative hardware vendors,
56: vendors out of business)
57: - Documentation is incomplete or plain wrong
1.13 ryoon 58: - Reverse engineering can solve these problems but it's a very
1.1 mspo 59: time consuming process
60:
1.4 mspo 61: ## The NetBSD driver model
1.1 mspo 62:
63: ### The NetBSD kernel basics
64:
65: - NetBSD has a classic monolithic UNIX-like kernel - all drivers are
66: running in the same address space
67: - Thanks to the above, communication between drivers and other kernel
68: layers is simple
69: - However, it also means that one badly written driver can affect the
70: whole kernel
71: - Numerous in-kernel frameworks standardise the way drivers are
72: written (bus\_space, autoconf, etc.)
73:
74: ### The NetBSD source directory structure
75:
1.13 ryoon 76: - We'll only cover parts interesting for a device driver programmer
1.1 mspo 77: - src/sys/
78: - kernel source directory
79: - src/sys/dev/
80: - machine-independent device drivers
81: - src/sys/arch/
82: - port-specific or architecture-specific parts (such as the
83: low-level system initialisation procedures or machine-dependent
84: drivers)
1.6 mspo 85: - src/sys/arch/$PORTNAME/conf/
1.1 mspo 86: - kernel configuration files for a given port
87:
88: ### Kernel autoconfiguration framework - autoconf(9)
89:
90: - Autoconfiguration is the process of matching hardware devices with
91: an appropriate device driver
92: - The kernel message buffer (dmesg) contains information about
93: autoconfiguration of devices
94: - driver0 at bus0: Foo hardware
95: - Instance 0 of the driver has attached to instance 0 of the
96: particular bus
97: - Such messages often carry additional bus-specific information
98: about the exact location of the device (like the device and
99: function number on the PCI bus)
100: - driver0: some message
101: - Additional information about the driver state or device
102: configuration
103:
104: ### Autoconfiguration as seen in the dmesg
105:
106: NetBSD 6.99.12 (GENERIC) #7: Fri Oct 5 18:43:21 CEST 2012
107: rkujawa@saiko.local:/Users/rkujawa/netbsd-eurobsdcon2012/src/sys/arch/cobalt/compile/obj/GENERIC
108: Cobalt Qube 2
109: total memory = 32768 KB
110: avail memory = 27380 KB
111: mainbus0 (root)
112: com0 at mainbus0 addr 0x1c800000 level 3: ns16550a, working fifo
113: com0: console
114: cpu0 at mainbus0: QED RM5200 CPU (0x28a0) Rev. 10.0 with built-in FPU Rev. 1.0
115: cpu0: 48 TLB entries, 256MB max page size
116: cpu0: 32KB/32B 2-way set-associative L1 instruction cache
117: cpu0: 32KB/32B 2-way set-associative write-back L1 data cache
118: mcclock0 at mainbus0 addr 0x10000070: mc146818 compatible time-of-day clock
119: panel0 at mainbus0 addr 0x1f000000
120: gt0 at mainbus0 addr 0x14000000
121: pci0 at gt0
122: pchb0 at pci0 dev 0 function 0: Galileo GT-64011 System Controller, rev 1
123: pcib0 at pci0 dev 9 function 0
124: pcib0: VIA Technologies VT82C586 PCI-ISA Bridge, rev 57
125: viaide0 at pci0 dev 9 function 1
126: viaide0: VIA Technologies VT82C586 (Apollo VP) ATA33 controller
127: viaide0: primary channel interrupting at irq 14
128: atabus0 at viaide0 channel 0
129: viaide0: secondary channel interrupting at irq 15
130: atabus1 at viaide0 channel 1
131: wd0 at atabus0 drive 0
132: wd0: <netbsd-cobalt.img>
133: wd0: 750 MB, 1524 cyl, 16 head, 63 sec, 512 bytes/sect x 1536192 sectors
134:
135: ### The bus\_space(9) framework
136:
1.13 ryoon 137: - "The goal of the bus\_space functions is to allow a single driver
1.1 mspo 138: source file to manipulate a set of devices on different system
139: architectures, and to allow a single driver object file to
140: manipulate a set of devices on multiple bus types on a single
1.13 ryoon 141: architecture."
1.1 mspo 142: - Provides a set of functions implementing common operations on the
143: bus like mapping, reading, writing, copying, etc.
144: - The bus\_space(9) is implemented at the machine-dependent level
1.13 ryoon 145: (typically it's a part of architecture-specific code), but all
1.6 mspo 146: implementations present the same interface
147: > At least they should, some functions are missing on less popular ports
1.1 mspo 148:
149: ### Machine independent drivers
150:
151: - If possible drivers should work on any hardware platform
152: - High quality, machine-independent (MI) drivers are an important
153: factor that adds to NetBSD portability
154: - Some drivers are completely MI, some have MD or bus dependent
155: attachments and some are completely MD
156: - A driver for a typical PCI card will be completely MI
157: - A driver for the components of a SoC will usually be completely
158: MD
159: - The bus\_space abstraction helps to achieve portability,
160: transparently handling endianness issues and hiding bus
161: implementation details from the device driver
162: - Even if we have MI drivers, writing the drivers is always
163: significant part of effort needed to port NetBSD to new hardware
164:
1.4 mspo 165: ## Example driver from scratch
1.1 mspo 166:
167: ### Development environment
168:
169: - Out of scope of this course, but very well documented
170: - Cross compiling is an easy task with the build.sh script
171: - Described in [Part V of the NetBSD
172: Guide](http://www.netbsd.org/docs/guide/en/part-compile.html)
173: - Check out the NetBSD sources
1.6 mspo 174: - $ build.sh -m cobalt tools
1.1 mspo 175: will build compiler, assembler, linker, etc. for cobalt port
1.6 mspo 176: - $ build.sh -m cobalt kernel=GENERIC
1.1 mspo 177: will build the GENERIC kernel for cobalt
1.13 ryoon 178: - Call build.sh with a -u parameter to update (won't rebuilding
1.1 mspo 179: everything)
180: - build.sh
181: is calling nbconfig and nbmake tools, no magic involved
182:
183: ### Quick introduction to GXemul
184:
185: - A framework for full-system computer architecture emulation,
186: excellent for educational purposes
187: - Capable of emulating several real machines supported by NetBSD
1.13 ryoon 188: - We'll emulate a [Cobalt](http://en.wikipedia.org/wiki/Cobalt_Qube),
1.1 mspo 189: MIPS-based micro server with PCI bus
1.13 ryoon 190: - I've modified GXemul and implemented an emulation of an additional
1.1 mspo 191: PCI device
192: - It will be used to show (almost) a real-life example of the driver
193: development process
194:
195: ### Our hardware - functional description
196:
197: - Business applications often use arithmetic operations like addition
198: - Fake Cards Inc. responded to market needs and created a new product,
199: Advanced Addition Accelerator
200: - Pointy Haired Bosses will certainly buy it to accelerate their
1.13 ryoon 201: business applications, so let's create a driver for NetBSD!
1.1 mspo 202:
203: ### Our hardware - technical details
204:
205: - Overview
206: - Implemented as a PCI device
207: - Arithmetic unit capable of addition of two numbers
1.6 mspo 208: - Four registers in the PCI memory space
209: > Only three of these registers are of any importance for us at this moment
1.1 mspo 210: - PCI configuration space
211: - Identified by the PCI vendor ID 0xfabc and product ID 0x0001
212: - Base Address Register 0x10 used to configure the engine address
213: - 4 x 32-bit registers = 16 bytes
214: - Other configuration registers irrelevant
215:
216: ### Our hardware - technical details (memory mapped register set)
217:
218: - Advanced Addition Acceleration registers
219:
1.10 ryoon 220: [[!table data="""
221: Register Name |Offset |Description
222: COMMAND |0x4 |Register used to issue commands to the engine
223: DATA |0x8 |Register used to load data to internal engine registers
224: RESULT |0xC |Register used to store the result of arithmetic operation
225: """]]
1.1 mspo 226:
227: - COMMAND register
228:
1.10 ryoon 229: [[!table data="""
230: Bit |R/W |Description
231: 0 |W |Execute ADD operation on values loaded into internal register A and B
232: 1 |R/W |Select internal register A for access through DATA register
233: 2 |R/W |Select internal register B for access through DATA register
234: """]]
1.1 mspo 235:
236: - Selecting internal register A and B at the same time will lead to
237: undefined behaviour
238:
239: ### Our hardware - technical details (memory mapped register set)
240:
241: - DATA register
242:
1.10 ryoon 243: [[!table data="""
244: Bit |R/W |Description
245: 0:31 |R/W |Read/write the value in internal engine register
246: """]]
1.1 mspo 247:
248: - RESULT register
249:
1.10 ryoon 250: [[!table data="""
251: Bit |R/W |Description
252: 0:31 |R |Holds the result of last ADD operation
253: """]]
1.1 mspo 254:
255: ### Our hardware - technical details (operation algorithm)
256:
257: - Select the internal register A for access (write 0x2 into COMMAND
258: register)
259: - Write the first number into DATA register
260: - Select the internal register B for access (write 0x4 into COMMAND
261: register)
262: - Write the second number into DATA register
263: - Issue the ADD operation (write 0x1 into COMMAND register)
264: - Read the result from RESULT register
265:
266: ### Adding a new driver to the NetBSD kernel
267:
1.13 ryoon 268: - We'll discuss the steps needed to add a new MI PCI device driver to
1.1 mspo 269: the NetBSD kernel
270: - Add the vendor and device ID to the database of PCI IDs
271: - Create a set of the driver source files in
1.6 mspo 272: src/sys/dev/$BUSNAME/
273: - Add the new driver to src/sys/dev/$BUSNAME/$BUSNAME.files file
274: - Add the new driver to DEVNAMES file
275: > Required if you are NetBSD developer, optional otherwise.
1.1 mspo 276:
277: ### Modifying the PCI device database
278:
279: unmatched vendor 0xfabc product 0x0001 (Co-processor
280: processor, revision 0x01) at pci0 dev 12 function 0
281: not configured
282:
283: - The kernel does not know anything about this vendor and device
284: - Add it to the PCI device database - src/sys/dev/pci/pcidevs
285: - vendor VENDORNAME 0xVENDORID Long Vendor Name
286: - product VENDORNAME PRODUCTNAME 0xPRODUCTID Long Product Name
287: - To regenerate pcidevs\*.h run awk -f devlist2h.awk pcidevs or
1.13 ryoon 288: Makefile.pcidevs if you're on NetBSD
1.1 mspo 289:
290: ### Modifying the PCI device database - example
291:
292: --- pcidevs 29 Sep 2012 10:26:14 -0000 1.1139
293: +++ pcidevs 5 Oct 2012 08:52:59 -0000
294: @@ -669,6 +669,7 @@
295: vendor CHRYSALIS 0xcafe Chrysalis-ITS
296: vendor MIDDLE_DIGITAL 0xdeaf Middle Digital
297: vendor ARC 0xedd8 ARC Logic
298: +vendor FAKECARDS 0xfabc Fake Cards
299: vendor INVALID 0xffff INVALID VENDOR ID
300:
301: /*
302: @@ -2120,6 +2121,9 @@
303: /* Eumitcom products */
304: product EUMITCOM WL11000P 0x1100 WL11000P PCI WaveLAN/IEEE 802.11
305:
306: +/* FakeCards products */
307: +product FAKECARDS AAA 0x0001 Advanced Addition Accelerator
308: +
309: /* O2 Micro */
310: product O2MICRO 00F7 0x00f7 Integrated OHCI IEEE 1394 Host Controller
311: product O2MICRO OZ6729 0x6729 OZ6729 PCI-PCMCIA Bridge
312:
313: ### Modifying the PCI device database - example
314:
315: Fake Cards Advanced Addition Accelerator (Co-processor
316: processor, revision 0x01) at pci0 dev 12 function 0
317: not configured
318:
319: - Now the kernel knows the vendor and product ID
1.13 ryoon 320: - But there's still no driver for this device
1.1 mspo 321:
322: ### Adding the new PCI driver
323:
324: - Choose a name - short, easy to remember, avoid numbers
1.5 mspo 325: - faa looks like a good name, but you can choose any name you like
1.1 mspo 326: - Create a set of new files in src/sys/dev/pci
327: - faa.c
328: - main driver code
329: - faareg.h
1.6 mspo 330: - register definitions
331: > Might not exist if the driver is only a simple passthrough from a specific bus to another MI driver.
1.1 mspo 332: - faavar.h
333: - driver structures and functions used in other parts of the
1.6 mspo 334: kernel
335: > Omitted if not needed.
1.1 mspo 336: - Modify driver definitions
337: - src/sys/dev/pci/files.pci
338: - src/sys/dev/DEVNAMES
339: - Configure the kernel to use the newly added driver -
1.6 mspo 340: src/sys/arch/$PORTNAME/conf/GENERIC
1.1 mspo 341:
342: ### Adding the new PCI driver - main driver
343:
344: - Kernel includes are at the beginning, followed by machine-specific
345: and bus-specific includes
346: - Should also include faareg.h and faavar.h files
347: - A minimal driver needs just two functions
348: - faa\_match
349: (or faa\_probe for some buses)
350: - faa\_attach
351: - The CFATTACH\_DECL\_NEW macro plugs the above functions into
352: autoconf(9) mechanism
353:
354: ### Adding the new PCI driver - main driver
355:
356: - static int faa\_match(device\_t parent, cfdata\_t match, void
357: \*aux);
358: - Check if the driver should attach to a given device (for example
359: in case of PCI bus, it will be used to check vendor and product
360: ID)
361: - parent
1.13 ryoon 362: - pointer to parent's driver device structure
1.1 mspo 363: - match
364: - pointer to autoconf(9) details structure
365: - aux
366: - despite the name the most important argument, usually contains
367: bus-specific structure describing device details
368:
369: - static void faa\_attach(device\_t parent, device\_t self, void
370: \*aux);
371: - Attach the driver to a given device
372: - parent
373: - same as with match function
374: - self
1.13 ryoon 375: - pointer to driver's device structure
1.1 mspo 376: - aux
377: - same as with match function
378: - See definitions of these functions in the
1.16 ! kim 379: [[!template id=man name="driver" section="9"]]
1.1 mspo 380: man page.
381:
1.13 ryoon 382: ### Adding the new PCI driver - main driver cont'd
1.1 mspo 383:
384: - CFATTACH\_DECL\_NEW(faa, sizeof(struct faa\_softc), faa\_match,
385: faa\_attach, NULL, NULL);
386: - driver name
1.13 ryoon 387: - size of softc structure containing state of driver's instance
1.1 mspo 388: - match/probe function
389: - attach function
390: - detach function
391: - activate function
1.13 ryoon 392: - The "\_NEW" name is unfortunate
1.1 mspo 393: - Pass NULL for unimplemented functions
1.13 ryoon 394: - We won't cover detach and activate now, as they are not needed for a
1.1 mspo 395: simple driver
396:
397: ### Adding the new PCI driver - main driver example
398:
399: - src/sys/dev/pci/faa.c
400:
401: <!-- -->
402:
403: #include <sys/cdefs.h>
1.16 ! kim 404: __KERNEL_RCSID(0, "$NetBSD: bus_space_tutorial.mdwn,v 1.15 2020/09/08 21:05:59 kim Exp $");
1.1 mspo 405: #include <sys/param.h>
406: #include <sys/device.h>
407: #include <dev/pci/pcivar.h>
408: #include <dev/pci/pcidevs.h>
409: #include <dev/pci/faareg.h>
410: #include <dev/pci/faavar.h>
411:
412: static int faa_match(device_t, cfdata_t, void *);
413: static void faa_attach(device_t, device_t, void *);
414:
415: CFATTACH_DECL_NEW(faa, sizeof(struct faa_softc),
416: faa_match, faa_attach, NULL, NULL);
417:
418: static int
419: faa_match(device_t parent, cfdata_t match, void *aux)
420: {
421: return 0;
422: }
423:
424: static void
425: faa_attach(device_t parent, device_t self, void *aux)
426: {
427: }
428:
429: ### Adding the new PCI driver - auxiliary includes
430:
431: - src/sys/dev/pci/faareg.h
432:
433: <!-- -->
434:
435: #ifndef FAAREG_H
436: #define FAAREG_H
437: /*
438: * Registers are defined using preprocessor:
439: * #define FAA_REGNAME 0x0
440: * We'll add them later, let's leave it empty for now.
441: */
442: #endif /* FAAREG_H */
443:
444: - src/sys/dev/pci/faavar.h
445:
446: <!-- -->
447:
448: #ifndef FAAVAR_H
449: #define FAAVAR_H
450:
451: /* sc_dev is an absolute minimum, we'll add more later */
452: struct faa_softc {
453: device_t sc_dev;
454: };
455: #endif /* FAAVAR_H */
456:
457: ### Adding the new PCI driver - registering the driver (courtesy)
458:
459: - src/sys/dev/DEVNAMES
460:
461: <!-- -->
462:
463: --- DEVNAMES 1 Sep 2012 11:19:58 -0000 1.279
464: +++ DEVNAMES 6 Oct 2012 19:59:06 -0000
465: @@ -436,6 +436,7 @@
466: ex MI
467: exphy MI
468: ezload MI Attribute
469: +faa MI
470: fb luna68k
471: fb news68k
472: fb newsmips
473:
474: ### Adding the new PCI driver - registering the driver
475:
476: - See config(5)
477: - src/sys/dev/pci/files.pci
478:
479: <!-- -->
480:
481: --- pci/files.pci 2 Aug 2012 00:17:44 -0000 1.360
482: +++ pci/files.pci 6 Oct 2012 19:59:10 -0000
483: @@ -1122,3 +1122,9 @@
484: device tdvfb: wsemuldisplaydev, rasops8, vcons, videomode
485: attach tdvfb at pci
486: file dev/pci/tdvfb.c tdvfb
487: +
488: +# FakeCards Advanced Addition Accelerator
489: +device faa
490: +attach faa at pci
491: +file dev/pci/faa.c faa
492: +
493:
494: ### Adding the new PCI driver to the kernel configuration
495:
496: - src/sys/arch/cobalt/conf/GENERIC
497:
498: <!-- -->
499:
500: --- GENERIC 10 Mar 2012 21:51:50 -0000 1.134
501: +++ GENERIC 6 Oct 2012 20:12:37 -0000
502: @@ -302,6 +302,9 @@
503: #fms* at pci? dev ? function ? # Forte Media FM801
504: #sv* at pci? dev ? function ? # S3 SonicVibes
505:
506: +# Fake Cards Advanced Addition Accelerator
507: +faa* at pci? dev ? function ?
508: +
509: # Audio support
510: #audio* at audiobus?
511:
512: - The above definition means that an instance of faa may be attached
513: to any PCI bus, any device, any function
514: - The exact position of the rule in the configuration file is not
515: important in this case
516: - See
1.16 ! kim 517: [[!template id=man name="config" section="5"]]
1.1 mspo 518: for a description of the device definition language
519:
520: ### Adding the new PCI driver - example
521:
522: - The driver should compile now
1.13 ryoon 523: - The driver's match function will check if the driver is able to work
1.1 mspo 524: with a given device
525: - Since it is not implemented, the kernel will not attach the driver
526:
527: ### Matching the PCI device
528:
529: - Modify the faa\_match function to match the specified PCI device
530: - Use PCI\_VENDOR and PCI\_PRODUCT macros to obtain the IDs
531:
532: <!-- -->
533:
534: static int
535: faa_match(device_t parent, cfdata_t match, void *aux)
536: {
537: const struct pci_attach_args *pa = (const struct pci_attach_args *)aux;
538:
539: if ((PCI_VENDOR(pa->pa_id) == PCI_VENDOR_FAKECARDS)
540: && (PCI_PRODUCT(pa->pa_id) == PCI_PRODUCT_FAKECARDS_AAA))
541: return 1;
542:
543: return 0;
544: }
545:
546: ### Attaching to the PCI device
547:
548: faa0 at pci0 dev 12 function 0
549:
550: - The driver has successfully matched and attached to the PCI device
551: but still is not doing anything useful
1.13 ryoon 552: - Let's fill the attach function and actually program the hardware
1.1 mspo 553:
554: ### Variable types used with bus\_space
555:
556: - bus\_space\_tag\_t
557:
1.13 ryoon 558: - type used to describe a particular bus, usually passed to the
1.10 ryoon 559: driver from MI bus structures
1.1 mspo 560:
561: - bus\_space\_handle\_t
562:
1.13 ryoon 563: - used to describe a mapped range of bus space, usually created with
1.10 ryoon 564: the bus\_space\_map() function
1.1 mspo 565:
566: - bus\_addr\_t
567:
1.13 ryoon 568: - address on the bus
1.1 mspo 569:
570: - bus\_size\_t
571:
1.13 ryoon 572: - an amount of space on the bus
1.1 mspo 573:
574: - Contents of these types are MD, so avoid modifying from within the
1.6 mspo 575: driver
1.13 ryoon 576: > although you'll often have to use bus\_size\_t
1.1 mspo 577:
1.13 ryoon 578: ### Why do we need to "map" the resources?
1.1 mspo 579:
1.13 ryoon 580: - "The bus space must be mapped before it can be used, and should be
581: unmapped when it is no longer needed"
582: - It's a machine-dependent process but it's also conveniently hidden
1.1 mspo 583: from the programmer by the bus\_space framework
584:
585: ### Mapping the hardware resources
586:
587: - The generic bus\_space(9) way to map space
588:
589: <!-- -->
590:
591: bus_space_map(bus_space_tag_t space, bus_addr_t address,
592: bus_size_t size, int flags, bus_space_handle_t *handlep);
593:
594: - bus\_space\_map
595:
1.13 ryoon 596: - creates a mapping from the physical address to a kernel virtual
1.1 mspo 597: address
598:
599: - space
600:
1.13 ryoon 601: - represents the bus on which the mapping will be created
1.1 mspo 602:
603: - address
604:
1.13 ryoon 605: - typically represents the physical address for which a mapping will
1.10 ryoon 606: be created
1.1 mspo 607:
608: - size
609:
1.13 ryoon 610: - describes the amount of bus space to be mapped
1.1 mspo 611:
612: - handlep
613:
1.13 ryoon 614: - pointer to mapped space (filled after successful mapping)
1.1 mspo 615:
616: - Separate space and address
617:
618: ### Mapping the hardware resources
619:
620: - The PCI-specific way to map space
621:
622: <!-- -->
623:
624: pci_mapreg_map(const struct pci_attach_args *pa, int reg, pcireg_t type,
625: int busflags, bus_space_tag_t *tagp, bus_space_handle_t *handlep,
626: bus_addr_t *basep, bus_size_t *sizep);
627:
628: - pci\_mapreg\_map
629:
1.13 ryoon 630: - creates mapping from physical address present in specified BAR
1.1 mspo 631: register to kernel virtual address
632:
633: - pa
634:
1.13 ryoon 635: - struct describing PCI attachment details (passed through aux)
1.1 mspo 636:
637: - reg
638:
1.13 ryoon 639: - BAR register number
1.1 mspo 640:
641: - type
642:
1.13 ryoon 643: - Select mapping type (I/O, memory)
1.1 mspo 644:
645: - busflags
646:
1.14 ryoon 647: - Passed to bus_space_map flags argument
1.1 mspo 648:
649: - tagp
650:
1.13 ryoon 651: - pointer to bus_space_tag
1.1 mspo 652:
653: - handlep
654:
1.13 ryoon 655: - pointer to a mapped space
1.1 mspo 656:
657: - basep
658:
1.13 ryoon 659: - address of a mapped space
1.1 mspo 660:
661: - sizep
662:
1.13 ryoon 663: - size of mapped space (equivalent to BAR size)
1.1 mspo 664:
665: - The last four parameters are filled after successful mapping
666:
667: ### Mapping the registers using BAR - adding auxiliary includes
668:
669: - src/sys/dev/pci/faareg.h
670:
671: <!-- -->
672:
673: #define FAA_MMREG_BAR 0x10
674:
675: - src/sys/dev/pci/faavar.h
676:
677: <!-- -->
678:
679: struct faa_softc {
680: device_t sc_dev;
681:
682: bus_space_tag_t sc_regt;
683: bus_space_handle_t sc_regh;
684: bus_addr_t sc_reg_pa;
685:
686: };
687:
688: ### Mapping the registers using BAR - main driver code
689:
690: - src/sys/dev/pci/faa.c
691:
692: <!-- -->
693:
694: static void
695: faa_attach(device_t parent, device_t self, void *aux)
696: {
697: struct faa_softc *sc = device_private(self);
698: const struct pci_attach_args *pa = aux;
699:
700: sc->sc_dev = self;
701:
702: pci_aprint_devinfo(pa, NULL);
703:
704: if (pci_mapreg_map(pa, FAA_MMREG_BAR, PCI_MAPREG_TYPE_MEM, 0,
705: &sc->sc_regt, &sc->sc_regh, &sc->sc_reg_pa, 0) != 0 ) {
706: aprint_error_dev(sc->sc_dev, "can't map the BAR\n");
707: return;
708: }
709:
710: aprint_normal_dev(sc->sc_dev, "regs at 0x%08x\n", (uint32_t) sc->sc_reg_pa);
711: }
712:
713: ### Accessing the hardware registers
714:
715: - The bus\_space\_read\_ and bus\_space\_write\_ functions are basic
716: methods of reading and writing the hardware registers
717:
718: - uintX\_t bus\_space\_read\_X(bus\_space\_tag\_t space,
719: bus\_space\_handle\_t handle, bus\_size\_t offset);
720:
721: - void bus\_space\_write\_X(bus\_space\_tag\_t space,
722: bus\_space\_handle\_t handle, bus\_size\_t offset, uintX\_t value);
723:
724: - space
725:
726: - tag describing the bus
727:
728: - handle
729:
730: - describes the exact location on the bus where read/write
731: should occur, this handle is obtained by bus\_space\_map
732:
733: - offset
734:
735: - offset from handle location
736:
737: - The read function returns the data read from the specified
738: location, while write has an argument value which should be
739: filled with data to be written
740:
741: ### Variants of bus\_space\_read and bus\_space\_write
742:
1.10 ryoon 743: [[!table data="""
744: Data |Read function |Write function
1.14 ryoon 745: 8-bit |bus_space_read_1 |bus_space_write_1
1.10 ryoon 746: 16-bit |bus_space_read_2 |bus_space_write_2
747: 32-bit |bus_space_read_4 |bus_space_write_4
748: 64-bit |bus_space_read_8 |bus_space_write_8
749: """]]
1.1 mspo 750:
751: - There are many more variants of read and write functions and they
752: are useful in certain situations, see the
1.16 ! kim 753: [[!template id=man name="bus_space" section="9"]]
1.1 mspo 754: man page
755:
756: ### Accessing the hardware registers - example
757:
758: - Create a function that will write a value into the DATA register of
759: our device, then read it back and check if the value is the same as
760: written
761:
762: - Define the DATA register in the driver
763:
764: - src/sys/dev/pci/faareg.h
765:
766: <!-- -->
767:
768: #define FAA_DATA 0x8
769: #define FAA_COMMAND 0x4
770: #define FAA_COMMAND_STORE_A __BIT(1)
771:
772: - Define the new function in main driver code
773:
774: - static bool faa\_check(struct faa\_softc \*sc);
775:
776: ### Accessing the hardware registers - example
777:
778: - src/sys/dev/pci/faa.c
779:
780: <!-- -->
781:
782: static void
783: faa_attach(device_t parent, device_t self, void *aux)
784: {
785: /* ... */
786: if (!faa_check(sc)) {
787: aprint_error_dev(sc->sc_dev, "hardware not responding\n");
788: return;
789: }
790: }
791:
792: static bool
793: faa_check(struct faa_softc *sc)
794: {
795: uint32_t testval = 0xff11ee22;
796: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_COMMAND, FAA_COMMAND_STORE_A);
797: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_DATA, testval);
798: if (bus_space_read_4(sc->sc_regt, sc->sc_regh, FAA_DATA) == testval)
799: return true;
800:
801: return false;
802: }
803:
804: ### Accessing the hardware registers - running the example
805:
806: - Update the kernel binary and run it again
807:
808: - Check the GXemul log
809:
810: <!-- -->
811:
812: [ faa: COMMAND register (0x4) WRITE value 0x2 ]
813: [ faa: DATA register (0x8) WRITE value 0xff11ee22 ]
814: [ faa: DATA register (0x8) READ value 0xff11ee22 ]
815:
816: - GXemul will conveniently display all accesses to our device
817:
818: - The faa driver still does attach without error, which means that the
819: check function is working properly
820:
821: <!-- -->
822:
823: faa0 at pci0 dev 12 function 0: Fake Cards Advanced Addition Accelerator (rev. 0x01)
824: faa0: registers at 0x10110000
825:
826: ### Implementing addition using the hardware
827:
828: - The basic principle of device operation should be laid out in the
829: data sheet
830:
831: - We need to implement an algorithm based on this description
832:
833: - Writing such an algorithm is often not needed, since the NetBSD
834: kernel already has frameworks for common device types (such as
835: atabus/wd for IDE and SATA hard disk controllers, wsdisplay/wscons
836: for frame buffers, etc.)
837:
838: ### Implementing addition using the hardware
839:
840: - Define all registers
841:
842: - src/sys/dev/pci/faareg.h
843:
844: <!-- -->
845:
846: #define FAA_STATUS 0x0
847: #define FAA_COMMAND 0x4
848: #define FAA_COMMAND_ADD __BIT(0)
849: #define FAA_COMMAND_STORE_A __BIT(1)
850: #define FAA_COMMAND_STORE_B __BIT(2)
851: #define FAA_DATA 0x8
852: #define FAA_RESULT 0xC
853:
854: ### Implementing addition using the hardware
855:
856: - Add a new function to the main driver code
857:
858: - src/sys/dev/pci/faa.c
859:
860: <!-- -->
861:
862: static void
863: faa_attach(device_t parent, device_t self, void *aux)
864: {
865: /* ... */
866: aprint_normal_dev(sc->sc_dev, "just checking: 1 + 2 = %d\n", faa_add(sc, 1, 2));
867: }
868:
869: static uint32_t
870: faa_add(struct faa_softc *sc, uint32_t a, uint32_t b)
871: {
872: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_COMMAND, FAA_COMMAND_STORE_A);
873: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_DATA, a);
874: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_COMMAND, FAA_COMMAND_STORE_B);
875: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_DATA, b);
876: bus_space_write_4(sc->sc_regt, sc->sc_regh, FAA_COMMAND, FAA_COMMAND_ADD);
877: return bus_space_read_4(sc->sc_regt, sc->sc_regh, FAA_RESULT);
878: }
879:
880: ### Implementing addition using the hardware - running the example
881:
882: - Update the kernel binary and run it again
883:
884: - Check GXemul log
885:
886: <!-- -->
887:
888: [ faa: COMMAND register (0x4) WRITE value 0x2 ]
889: [ faa: DATA register (0x8) WRITE value 0x1 ]
890: [ faa: COMMAND register (0x4) WRITE value 0x4 ]
891: [ faa: DATA register (0x8) WRITE value 0x2 ]
892: [ faa: COMMAND register (0x4) WRITE value 0x1 ]
893: [ faa: RESULT register (0xC) READ value 0x3 ]
894:
895: - Looks like it worked!
896:
897: <!-- -->
898:
899: faa0 at pci0 dev 12 function 0: Fake Cards Advanced Addition Accelerator (rev. 0x01)
900: faa0: registers at 0x10110000
901: faa0: just checking: 1 + 2 = 3
902:
1.3 mspo 903: ## Interacting with userspace
1.1 mspo 904:
905: ### The kernel-user space interface
906:
907: - Now that the core functionality of the kernel driver is working, it
908: should be exposed to user space
909: - The interface between kernel driver and userspace can be designed in
910: many different ways
911: - The classic UNIX way of interfacing between the kernel and user
912: space is a device file
913: - Even when using device files there is no single interfacing method
914: that fits all use cases
1.13 ryoon 915: - It's up to the programmer to define the communication protocol
1.1 mspo 916:
917: ### Device files
918:
1.13 ryoon 919: - crw-r-- 1 root wheel 101, 1 Aug 12 21:53 /dev/file
1.1 mspo 920: - The kernel identifies which driver should service the request to
921: this file by using major and minor numbers (101 and 1 in the example
922: above)
923: - The major number identifies the driver
924: - The minor number usually identifies the driver instance, although
925: the driver is free to use it in any other way
926: - In NetBSD device files are created statically
927: - By the MAKEDEV script during installation or boot
928: - Manually by using the mknod utility
929:
930: ### Operations on device files
931:
1.16 ! kim 932: - [[!template id=man name="open" section="2"]]
1.1 mspo 933: and
1.16 ! kim 934: [[!template id=man name="close" section="2"]]
! 935: - [[!template id=man name="read" section="2"]]
1.1 mspo 936: and
1.16 ! kim 937: [[!template id=man name="write" section="2"]]
! 938: - [[!template id=man name="ioctl" section="2"]]
! 939: - [[!template id=man name="poll" section="2"]]
! 940: - [[!template id=man name="mmap" section="2"]]
1.13 ryoon 941: - and more...
1.1 mspo 942: - Any mix of the above system calls might be used to interface between
943: the kernel and user space
1.13 ryoon 944: - We'll implement an ioctl(2)-based communication mechanism
1.1 mspo 945:
946: ### Adding cdevsw
947:
948: - cdevsw
949:
950: is used to decide which operation on the character device file calls
951: which driver function
952:
953: - Not all calls have to be implemented, although some device layers
954: define a set of calls that a driver must implement
955:
956: - For example disk drivers must implement open, close, read, write and
957: ioctl
958:
959: - src/sys/dev/pci/faa.c
960:
961: <!-- -->
962:
963: dev_type_open(faaopen);
964: dev_type_close(faaclose);
965: dev_type_ioctl(faaioctl);
966:
967: const struct cdevsw faa_cdevsw = {
968: faaopen, faaclose, noread, nowrite, faaioctl,
969: nostop, notty, nopoll, nommap, nokqfilter, D_OTHER
970: };
971:
972: ### Prototyping the cdevsw operations
973:
974: - The dev\_type\* macros are used to prototype the functions passed to
975: cdevsw
976: - Pass no followed by a function name to the appropriate cdevsw field
977: if it is not implemented
1.13 ryoon 978: - There's also bdevsw for block devices, but we won't use it in this
1.1 mspo 979: example
980: - The last member of the cdevsw structure defines the device flags,
981: originally it was used to define the device type (still used for
982: disks, tape drives and ttys, for other devices pass D\_OTHER)
983:
984: ### Implemeting the cdevsw operations - open / close
985:
986: - src/sys/dev/pci/faa.c
987:
988: <!-- -->
989:
990: int
991: faaopen(dev_t dev, int flags, int mode, struct lwp *l)
992: {
993: struct faa_softc *sc;
994: sc = device_lookup_private(&faa_cd, minor(dev));
995:
996: if (sc == NULL)
997: return ENXIO;
998: if (sc->sc_flags & FAA_OPEN)
999: return EBUSY;
1000:
1001: sc->sc_flags |= FAA_OPEN;
1002: return 0;
1003: }
1004: int
1005: faaclose(dev_t dev, int flag, int mode, struct lwp *l)
1006: {
1007: struct faa_softc *sc;
1008: sc = device_lookup_private(&faa_cd, minor(dev));
1009:
1010: if (sc->sc_flags & FAA_OPEN)
1011: sc->sc_flags =~ FAA_OPEN;
1012:
1013: return 0;
1014: }
1015:
1016: ### Defining the ioctls
1017:
1018: - ioctl(2)
1019:
1020: can be used to call kernel-level functions and exchange data between
1021: the kernel and user space
1022:
1023: - The classic way of passing data is by using structures, their
1024: definitions are shared between the kernel and user space code
1025: - The driver might support more than one ioctl, the \_IO\* macros are
1026: used to define the operation and associated structure used to
1027: exchange data
1028:
1029: - \_IO
1030:
1031: - just a kernel function call, no data exchange
1032:
1033: - \_IOR
1034:
1035: - kernel function call and data pass from kernel to user space
1036:
1037: - \_IOW
1038:
1039: - kernel function call and data pass from user space to kernel
1040:
1041: - \_IOWR
1042:
1043: - kernel function call and data exchange in both directions
1044:
1045: - \#define DRIVERIO\_IOCTLNAME \_IOXXX(group, ioctl\_number, data
1046: structure)
1047:
1.3 mspo 1048: ## Using ioctls
1.1 mspo 1049:
1050: ### Defining the ioctls
1051:
1052: - src/sys/dev/pci/faaio.h
1053:
1054: <!-- -->
1055:
1056: #include <sys/ioccom.h>
1057:
1058: #define FAAIO_ADD _IOWR(0, 1, struct faaio_add)
1059:
1060: struct faaio_add {
1061: uint32_t a;
1062: uint32_t b;
1063: uint32_t *result;
1064: };
1065:
1066: - In the above example the ioctl group is not defined (0), but a
1067: single letter identifier could appear as first argument to \_IOWR
1068:
1069: ### Implemeting the cdevsw operations - ioctl
1070:
1071: - src/sys/dev/pci/faa.c
1072:
1073: <!-- -->
1074:
1075: int
1076: faaioctl(dev_t dev, u_long cmd, void *data, int flag, struct lwp *l)
1077: {
1078: struct faa_softc *sc = device_lookup_private(&faa_cd, minor(dev));
1079: int err;
1080:
1081: switch (cmd) {
1082: case FAAIO_ADD:
1083: err = faaioctl_add(sc, (struct faaio_add *) data);
1084: break;
1085: default:
1086: err = EINVAL;
1087: break;
1088: }
1089: return(err);
1090: }
1091: static int
1092: faaioctl_add(struct faa_softc *sc, struct faaio_add *data)
1093: {
1094: uint32_t result; int err;
1095:
1096: aprint_normal_dev(sc->sc_dev, "got ioctl with a %d, b %d\n",
1097: data->a, data->b);
1098:
1099: result = faa_add(sc, data->a, data->b);
1100: err = copyout(&result, data->result, sizeof(uint32_t));
1101: return err;
1102: }
1103:
1104: ### Using copyout to pass data to userspace
1105:
1106: - The copy(9) functions are used to copy kernel space data from/to
1107: user space
1108: - copyout(kernel\_address, user space\_address, size);
1109: - Actually on Cobalt we could just do data-\>result = faa\_add();
1110: instead of calling the copyout function, but that is a bad idea
1111: - Some architectures (such as sparc64) have totally separate kernel
1112: and user address spaces $ \implies $ user space addresses are
1113: meaningless in the kernel
1114:
1115: ### Defining device major number
1116:
1117: - Device major numbers for hardware drivers are usually defined in a
1.6 mspo 1118: per-port manner
1.13 ryoon 1119: > It's also possible to define a major in a machine-independent way in src/sys/conf/majors
1.6 mspo 1120: - src/sys/arch/$PORTNAME/conf/majors.$PORTNAME
1.1 mspo 1121: - src/sys/arch/cobalt/conf/majors.cobalt
1122: - The following defines a new character device file called /dev/faa\*
1123: with major number 101, but only if the faa driver is included in the
1124: kernel (last argument)
1125: - device-major faa char 101 faa
1126:
1127: ### Creating the device node
1128:
1129: - The mknod utility can be used to create the device file manually
1130: - The driver name can be specified instead of the major number - it
1131: will be automatically resolved into the correct major number
1132: - mknod name [b | c] [major | driver] minor
1133: - mknod /dev/faa0 c faa 0
1134: - Created successfully
1.13 ryoon 1135: - crw-r-r- 1 root wheel 101, 0 Oct 8 2012 /dev/faa0
1.1 mspo 1136:
1137: ### An example user space program
1138:
1139: - The example program will open the device file and call ioctl(2) on
1140: it
1141: - As simple as possible, just to show how communication is done
1142: - Using ioctls from the user space
1143: - Open the device file with O\_RDWR
1144: - Call ioctl(2) with the operation number and structure as
1145: parameters
1146:
1147: ### An example user space program - source
1148:
1149: void add(int, uint32_t, uint32_t);
1150:
1151: static const char* faa_device = "/dev/faa0";
1152:
1153: int
1154: main(int argc, char *argv[])
1155: {
1156: int devfd;
1157:
1158: if (argc != 3) {
1159: printf("usage: %s a b\n", argv[0]);
1160: return 1;
1161: }
1162: if ( (devfd = open(faa_device, O_RDWR)) == -1) {
1163: perror("can't open device file");
1164: return 1;
1165: }
1166:
1167: add(devfd, atoi(argv[1]), atoi(argv[2]));
1168:
1169: close(devfd);
1170: return 0;
1171: }
1172:
1173: ### An example user space program - source
1174:
1175: void
1176: add(int devfd, uint32_t a, uint32_t b)
1177: {
1178: struct faaio_add faaio;
1179: uint32_t result = 0;
1180:
1181: faaio.result = &result;
1182: faaio.a = a;
1183: faaio.b = b;
1184:
1185: if (ioctl(devfd, FAAIO_ADD, &faaio) == -1) {
1186: perror("ioctl failed");
1187: }
1188: printf("%d\n", result);
1189: }
1190:
1191: ### An example user space program - running it
1192:
1193: # make
1194: cc -o aaa_add aaa_add.c
1195: # ./aaa_add 3 7
1196: faa0: got ioctl with a 3, b 7
1197: 10
1198:
1199: - The program is successfully accessing the faa driver through the
1200: ioctl
1201: - The faa0:... line is a kernel message, normally only seen on the
1202: console terminal
1203:
1.3 mspo 1204: ## A few tips
1.1 mspo 1205:
1206: ### Avoiding common pitfalls
1207:
1208: - Always free resources allocated in the match or probe functions
1.13 ryoon 1209: - Always use bus\_space methods, don't access the hardware using a
1.1 mspo 1210: pointer dereference
1211: - If possible test on more than one hardware architecture, some bugs
1212: may surface
1.13 ryoon 1213: - Don't reinvent the wheel, try to use existing kernel frameworks as
1.1 mspo 1214: much as possible
1215: - Use copy(9) (or uiomove(9) or store(9)/fetch(9)) to move data
1216: between the kernel and user space
1217:
1218: ### Basic driver debugging
1219:
1220: - Use aprint\_debug to print debug-level messages on console and log
1221: them (enabled by passing AB\_DEBUG from the boot loader)
1222: - Use the built-in DDB debugger
1223: - Enabled by the kernel option DDB
1224: - A kernel panic will start DDB if the DDB\_ONPANIC=1 kernel
1225: option is specified or the ddb.onpanic sysctl is set to 1.
1226: - Run \# sysctl -w kern.panic\_now=1 to trigger a panic manually
1227: (DIAGNOSTIC option)
1228: - Remote debugging is possible on some ports
1229: - With KGDB through the serial port
1230: - With IPKDB through the network
1231:
1.3 mspo 1232: ## Summary
1.1 mspo 1233:
1234: ### Further reading
1235:
1236: - Documentation, articles:
1237: - [A Machine-Independent DMA Framework for NetBSD, Jason R.
1238: Thorpe](http://www.netbsd.org/docs/kernel/bus_dma.pdf)
1239: - [Writing Drivers for NetBSD, Jochen
1240: Kunz](ftp://ftp.netbsd.org/pub/NetBSD/misc/ddwg/NetBSD-driver_writing-1.0.1e.pdf)
1241: - [NetBSD Documentation: Writing a pseudo
1242: device](http://www.netbsd.org/docs/kernel/pseudo/)
1.16 ! kim 1243: - [[!template id=man name="autoconf" section="9"]],
! 1244: [[!template id=man name="bus_space" section="9"]]
! 1245: [[!template id=man name="bus_dma" section="9"]]
! 1246: [[!template id=man name="driver" section="9"]],
! 1247: [[!template id=man name="pci" section="9"]]
1.1 mspo 1248: man pages, etc.
1249:
1250: - Example source code of drivers:
1251: - tdvfb
1252:
1.4 mspo 1253: voodoofb are fairly good frame buffer driver examples with
1.1 mspo 1254: documentation publicly available.
1255:
1256: - etsec
1257:
1258: is a nice example of a more complicated network interface driver
1259:
1260: ### Get the source code
1261:
1262: - Download the source code and materials for this tutorial
1263: - <https://github.com/rkujawa/busspace-tutorial>
1264: - <https://github.com/rkujawa/gxemul-tutorial>
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to bus_space_tutorial.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / tutorials