Having trouble viewing this e-mail? View it in your browser.
ClearText from Kansas Fire & Rescue Training Institute
ClearText 2010, No. 5 November 30, 2010

National Certification: Part 2—Test Design and Structure

In the last edition of ClearText, we explained accreditation, certification and the role of national standards. This edition will explain how items are selected for an exam and what drives the system. If you were engaged in the learning process during your class and understand the processes of exam selection and preparation given in this edition, you should have everything you need to pass the exam. The only thing left is to actually do the final preparation (covered in the next edition of ClearText)!

What’s on the Test?

Each requirement of the standard has an equal weight in the testing process. Everyone involved from the student to the program director can build a case that some of the knowledge and skills required in the standard may be more important than others. In the "real world," it is much more important for a firefighter to have good SCBA knowledge and skills than know how to answer a fire station telephone. In certification testing, both of those skills have the same weight. Accreditation criteria require that we must test one as frequently and thoroughly as the other…and they are both worth the same number of points!

We must have a process established through which ALL requirements of the standard (and level) are tested. If we do not test all requirements, we must have a random process of selecting written and skills exam components. Most standards include too many knowledge and skills components to test within a reasonable time; so we use a random process of selection.

Written exams are selected using a computer program that searches the written exam database of questions and selects (at random) the number of questions we specify. Each written exam database is required to be at least twice as big as the exam that we administer (100 question exam = 200+ question database). Most of our exam databases are 3–500% larger than the exam we administer. We are required to have multiple versions of the exam and to limit the use of any single version to reduce the possibility of someone seeing the same version of the exam on retests. We typically generate new versions of an exam every month or two.

Each exam question must meet specific criteria to meet validity and reliability standards. Each question must be constructed using recognized test construction methods, referenced to an appropriate requirement of the standard, and to a reading text (IFSTA manual, etc.). The question, standard and reference form a triangle much like the fire triangle in that if any element is missing or incorrect, the question is invalid (see Figure 1). These requirements are scrutinized for every question in the database. If the question doesn't meet these criteria, it doesn't make it into the database.

Figure 1

The Institute adds a fourth side to this triangle making it a tetrahedron. This fourth side is the course curriculum or lesson plan. The lesson plan is added to the process to ensure that not only does the question meet the accreditation criteria, it gives further credibility to the system by ensuring that what a student is taught is also what is being tested. This link explains the Institute's insistence that instructors teach the lesson plan without skipping or changing the material (see Figure 2).

Figure 2

Skills Testing

Strict criteria govern this process, too. We must test all skills required by the standard or use a random selection process. If we use a random selection process, we must prove that the process tests a large enough sampling of the required skills to provide reasonable evidence that the candidate has met the requirements of the standard. In other words, they need to be able to perform all skills required by the standard. If we cannot provide that evidence, we must test every requirement. This means that our "sample" must include a statistically significant number of the required skills. Fire Fighter I includes 119 individual skills, which are far too many to test within a reasonable time. Therefore, we use a random sample, where the specific list of skills to be tested at any single site must be kept confidential until the time of the exam.

Some standards require the candidate to perform other skills such as writing letters (Fire Officer I), communicating verbally (Fire Officer, Instructor), teaching lessons, and other skills that can't be tested through traditional approaches. In these cases, interviews, boards of review, and videotaped presentations are used. Each of these has stringent guidelines, and the best preparation is to simply know how to perform to the criteria (e.g., write a letter, make a presentation, etc.). When these guidelines are used, the Institute publishes the criteria in the Study Guide. In these cases, you can practice, too. Review the criteria from the Study Guide, and practice writing or making presentations. The worst time to do a skill for the first time is during that exam; there is simply no room for error!

The entire testing process is designed to evaluate the requirements of the standard with no bias or tricks. It is designed in a straight-forward manner giving you access to everything you need. We want you to succeed. We can't help you take the test, but we can help you learn and prepare. In the next edition of ClearText, we will focus on how to prepare for certification exams.

Next Issue

National Certification: Part 3—Preparing for Certification Exams

Have a topic you want to see addressed? Send your idea to KUFIRE@ku.edu!

Add kuce2@ku.edu to your address book or safe list to ensure that our messages are delivered properly.
Unsubscribe: If you no longer wish to receive e-mail please unsubscribe.