Steps in Assessment Development Second in the series on Development of a Bilingual Test for Spanish-English Children, this presentation is a general overview of the process for developing a new assessment test. Presentation Video
Free
Presentation Video  |   November 01, 2013
Steps in Assessment Development
Author Affiliations & Notes
  • Elizabeth D. Peña
    University of Texas at Austin
  • Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Co-Presenters: Elizabeth D. Peña, University of Texas at Austin; Aquiles Iglesias, Temple University; Vera F. Gutierrez-Clellen, San Diego State University; Brian A. Goldstein, La Salle University; and Lisa M. Bedore, University of Texas at Austin.
    Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Co-Presenters: Elizabeth D. Peña, University of Texas at Austin; Aquiles Iglesias, Temple University; Vera F. Gutierrez-Clellen, San Diego State University; Brian A. Goldstein, La Salle University; and Lisa M. Bedore, University of Texas at Austin. ×
  • Disclosure: All of the above-listed authors/co-presenters benefit financially from royalty payments from the Bilingual English-Spanish Assessment (BESA.).
    Disclosure: All of the above-listed authors/co-presenters benefit financially from royalty payments from the Bilingual English-Spanish Assessment (BESA.).×
Article Information
Cultural & Linguistic Diversity / Research Issues, Methods & Evidence-Based Practice / Clinical Practice Research / Diagnostics, Screening and Assessment Research
Presentation Video   |   November 01, 2013
Steps in Assessment Development
CREd Library, November 2013, doi:10.1044/cred-pvd-c13007
CREd Library, November 2013, doi:10.1044/cred-pvd-c13007

The following is a transcript of the presentation video, edited for clarity. Presentation slides are available for download via the PDF button in the toolbar.

Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Videos in this series are:

  1. Challenges in Assessing Bilingual Populations

    (Elizabeth D. Peña, University of Texas at Austin)

  2. Steps in Test Development

    (Elizabeth D. Peña, University of Texas at Austin)

  3. Bilingual Phonology Assessment Design

    (Brian A. Goldstein, La Salle University)

  4. Bilingual Pragmatics Assessment Design

    (Aquiles Iglesias, Temple University)

  5. Bilingual Semantics Assessment Design

    (Lisa M. Bedore, University of Texas at Austin)

  6. Bilingual Morphosyntax Assessment Design

    (Vera F. Gutierrez-Clellen)

Plan the Test
What are the steps in test development?
Well, first you plan the test. What are the domains that you want to address?
In our case, we looked at semantics, morphosyntax, phonology, and pragmatics.
Write Items for Each Domain
Then what we did is we wrote items for each of these domains. We wrote probably three or four times the items that we would eventually want.
A clinical test is one that is pretty efficient. You want to be able to give a test, and have each subtest take maybe 20 minutes, right? You don't want to do a three- or four-hour test. But to start, you have to start out with a three or a four hour test to figure out which items are your best items for this population. That's basically what we did, and each person is going to talk about how we did that for each domain in particular.
Administer to a Small Sample
Then you administer all the items to a small sample. Usually 50 kids. If you're developing something like the GRE, you would administer that to several hundred. We're not doing the GRE, so 50 or so kids was a good sample size to start with.
Conduct Item Analysis
Then you conduct item analysis. And through that item analysis, you start throwing out the items that don't work.
Administer Revised Test and Cross-Validate.
Then, we take this shorter, revised test. And we administer it to another sample. A bigger sample.
Then we continue to do this, doing item analysis and cross-validation as we start to step through all these steps and reduce the number of items that we have in each of the different domains.
Item Difficulty
I'm going to tell you a little bit about item difficulty. Item difficulty is the proportion of a population that got an item correct. Those are p values. And p values go from 0 to 1. If they are close to 0, that means nobody got that item right. So it's way too hard, so that doesn't tell you anything, so you get rid of it.
If everybody got that item right, then it's way too easy. So you might include one or two so you don't totally bum out the kids, but you don't want too many really easy items, because that's not going to tell you anything either.
Then the items that you select depends, of course, on the purpose of your test. So we use these same principles for item difficulty and apply it for different kids at different levels of exposure to two language as well as for kids with and without impairments that we were trying to identify.
Item Discrimination
The next thing you do is look at item discrimination. Item discrimination is the comparison of the percentage of typical kids who got the item right, and the percentage of children with impairments who got that item right. We subtract and look at the differences between those two. There we're going to get a difference of, again, 0 to 1. And the greater the difference, the more sensitive that item is to the impairment you're trying to identify.
If you can build your test based on those items, items that are sensitive to that impairment, but don't have big discrimination values for level of exposure, for example, or age of first language exposure, then you have a better chance of having a measure that is going to help you make distinctions between your clinical and non-clinical group.
References for this Series
Allen, M. & Yen, W. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth.
Allen, M. & Yen, W. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth.×
Alt, M., Meyers, C. & Figueroa, C. (2013). Factors that influence fast mapping in children exposed to Spanish and English. Journal of Speech, Language, and Hearing Research, 56(4), 1237–1248 [Article]
Alt, M., Meyers, C. & Figueroa, C. (2013). Factors that influence fast mapping in children exposed to Spanish and English. Journal of Speech, Language, and Hearing Research, 56(4), 1237–1248 [Article] ×
Alt, M. & Suddarth, R. (2012). Learning novel words: Detail and vulnerability of initial representations for children with specific language impairment and typically developing peers. Journal of Communication Disorders, 45(2), 84–97 [Article] [PubMed]
Alt, M. & Suddarth, R. (2012). Learning novel words: Detail and vulnerability of initial representations for children with specific language impairment and typically developing peers. Journal of Communication Disorders, 45(2), 84–97 [Article] [PubMed]×
Anderson, R. T. (2001). Lexical morphology and verb use in child first language loss: A preliminary case study investigation. International Journal of Bilingualism, 5(4), 377–401 [Article]
Anderson, R. T. (2001). Lexical morphology and verb use in child first language loss: A preliminary case study investigation. International Journal of Bilingualism, 5(4), 377–401 [Article] ×
Bedore, L. M., Peña, E. D., Summers, C. L., Boerger, K. M., Resendiz, M. D., Greene, K., Bohman, T. M. & Gillam, R. B. (2012). The measure matters: Language dominance profiles across measures in Spanish/English bilingual children. Bilingualism: Language and Cognition, 15(3), 616–629 [Article]
Bedore, L. M., Peña, E. D., Summers, C. L., Boerger, K. M., Resendiz, M. D., Greene, K., Bohman, T. M. & Gillam, R. B. (2012). The measure matters: Language dominance profiles across measures in Spanish/English bilingual children. Bilingualism: Language and Cognition, 15(3), 616–629 [Article] ×
Bedore, L. M., Peña, E. D., Gillam, R. B. & Ho, T. (2010). Language sample measures and language ability in Spanish-English bilingual kindergarteners. Journal of Communication Disorders, 43(6), 498–510 [Article] [PubMed]
Bedore, L. M., Peña, E. D., Gillam, R. B. & Ho, T. (2010). Language sample measures and language ability in Spanish-English bilingual kindergarteners. Journal of Communication Disorders, 43(6), 498–510 [Article] [PubMed]×
Bishop, D. V. (1998). Development of the children's communication checklist (CCC): A method for assessing qualitative aspects of communicative impairment in children. Journal of Child Psychology and Psychiatry, 39(6), 879–891 [Article] [PubMed]
Bishop, D. V. (1998). Development of the children's communication checklist (CCC): A method for assessing qualitative aspects of communicative impairment in children. Journal of Child Psychology and Psychiatry, 39(6), 879–891 [Article] [PubMed]×
Bonifacio, S., Girolametto, L., Bulligan, M., Callegari, M., Vignola, S. & Zocconi, E. (2007). Assertive and responsive conversational skills of Italian-speaking late talkers. International Journal of Language & Communication Disorders, 42(5), 607–623 [Article]
Bonifacio, S., Girolametto, L., Bulligan, M., Callegari, M., Vignola, S. & Zocconi, E. (2007). Assertive and responsive conversational skills of Italian-speaking late talkers. International Journal of Language & Communication Disorders, 42(5), 607–623 [Article] ×
Brice, A. & Montgomery, J. (1996). Adolescent pragmatic skills: A comparison of Latino students in English as a second language and speech and language programs. Language, Speech, and Hearing Services in Schools, 27(1), 68–81 [Article]
Brice, A. & Montgomery, J. (1996). Adolescent pragmatic skills: A comparison of Latino students in English as a second language and speech and language programs. Language, Speech, and Hearing Services in Schools, 27(1), 68–81 [Article] ×
Cotton, E. & Sharp, J. (1988). Spanish in the Americas. Washington, DC: Georgetown University Press.
Cotton, E. & Sharp, J. (1988). Spanish in the Americas. Washington, DC: Georgetown University Press.×
Carrow, E. (1974). Austin Spanish articulation test. Austin, TX: Learning Concepts.
Carrow, E. (1974). Austin Spanish articulation test. Austin, TX: Learning Concepts.×
Hammond, R. (2001). The Sounds of Spanish: Analysis and application (with special reference to American English). Somerville, MA: Cascadilla Press.
Hammond, R. (2001). The Sounds of Spanish: Analysis and application (with special reference to American English). Somerville, MA: Cascadilla Press.×
Goldstein, B. (2007). Measuring phonological skills in bilingual children: Methodology and clinical applications. In Centeno J., Obler L. & Anderson R. (Eds.). Studying Communication Disorders In Spanish Speakers: Theoretical, Research, & Clinical Aspects. Clevedon, UK: Multilingual Matters.
Goldstein, B. (2007). Measuring phonological skills in bilingual children: Methodology and clinical applications. In Centeno J., Obler L. & Anderson R. (Eds.). Studying Communication Disorders In Spanish Speakers: Theoretical, Research, & Clinical Aspects. Clevedon, UK: Multilingual Matters.×
Goldstein, B. & McLeod, S. (2012). Typical and atypical multilingual speech acquisition. In McLeod S. & Goldstein B. (Eds.). Multilingual aspects of speech sound disorders in children. Clevedon, UK: Multilingual Matters.
Goldstein, B. & McLeod, S. (2012). Typical and atypical multilingual speech acquisition. In McLeod S. & Goldstein B. (Eds.). Multilingual aspects of speech sound disorders in children. Clevedon, UK: Multilingual Matters.×
Goldstein, B. & Gildersleeve-Neumann, C. (2012). Phonological development and disorders. In Goldstein B. (Ed.). Bilingual language development and disorders in Spanish-English speakers (2nd edition). Baltimore: Brookes.
Goldstein, B. & Gildersleeve-Neumann, C. (2012). Phonological development and disorders. In Goldstein B. (Ed.). Bilingual language development and disorders in Spanish-English speakers (2nd edition). Baltimore: Brookes.×
Gray, S. (2004). Word learning by preschoolers with specific language impairment: Predictors and poor learners. Journal of Speech, Language, and Hearing Research, 47(5), 1117–1132 [Article]
Gray, S. (2004). Word learning by preschoolers with specific language impairment: Predictors and poor learners. Journal of Speech, Language, and Hearing Research, 47(5), 1117–1132 [Article] ×
Gray, S. (2005). Word learning by preschoolers with specific language impairment: Effect of phonological or semantic cues. Journal of Speech, Language, and Hearing Research, 48(6), 1452–1467 [Article]
Gray, S. (2005). Word learning by preschoolers with specific language impairment: Effect of phonological or semantic cues. Journal of Speech, Language, and Hearing Research, 48(6), 1452–1467 [Article] ×
Gutiérrez-Clellen, V. F., Restrepo, M. A. & Simón-Cereijido, G. (2006). Evaluating the discriminant accuracy of a grammatical measure with Spanish-speaking children. Journal of Speech, Language, and Hearing Research, 49(6), 1209–1223 [Article]
Gutiérrez-Clellen, V. F., Restrepo, M. A. & Simón-Cereijido, G. (2006). Evaluating the discriminant accuracy of a grammatical measure with Spanish-speaking children. Journal of Speech, Language, and Hearing Research, 49(6), 1209–1223 [Article] ×
Gutiérrez-Clellen, V. F. & Simón-Cereijido, G. (2007). Evaluating the discriminant accuracy of a grammatical measure with Latino English-speaking children. Journal of Speech, Language, and Hearing Research, 50(4), 968–981 [Article]
Gutiérrez-Clellen, V. F. & Simón-Cereijido, G. (2007). Evaluating the discriminant accuracy of a grammatical measure with Latino English-speaking children. Journal of Speech, Language, and Hearing Research, 50(4), 968–981 [Article] ×
Hodson, B. (1986). Assessment of phonological processes-Spanish. San Diego: Los Amigos Research Associates.
Hodson, B. (1986). Assessment of phonological processes-Spanish. San Diego: Los Amigos Research Associates.×
Jacobson, P. F. & Schwartz, R. G. (2005). English past tense use in bilingual children with language impairment. American Journal of Speech-Language Pathology, 14(4), 313–323 [Article] [PubMed]
Jacobson, P. F. & Schwartz, R. G. (2005). English past tense use in bilingual children with language impairment. American Journal of Speech-Language Pathology, 14(4), 313–323 [Article] [PubMed]×
Kohnert, K. J. & Bates, E. (2002). Balancing bilinguals ii: Lexical comprehension and cognitive processing in children learning Spanish and English. Journal of Speech, Language, and Hearing Research, 45(2), 347–359 [Article]
Kohnert, K. J. & Bates, E. (2002). Balancing bilinguals ii: Lexical comprehension and cognitive processing in children learning Spanish and English. Journal of Speech, Language, and Hearing Research, 45(2), 347–359 [Article] ×
Leonard, L. B., Eyer, J. A., Bedore, L. M. & Grela, B. G. (1997). Three accounts of the grammatical morpheme difficulties of English-speaking children with specific language impairment. Journal of Speech, Language, and Hearing Research, 40(4), 741–753 [Article]
Leonard, L. B., Eyer, J. A., Bedore, L. M. & Grela, B. G. (1997). Three accounts of the grammatical morpheme difficulties of English-speaking children with specific language impairment. Journal of Speech, Language, and Hearing Research, 40(4), 741–753 [Article] ×
Mcgregor, K. K., Newman, R. M., Reilly, R. M. & Capone, N. C. (2002). Semantic representation and naming in children with specific language impairment. Journal of Speech, Language, and Hearing Research, 45(5), 998–1014 [Article]
Mcgregor, K. K., Newman, R. M., Reilly, R. M. & Capone, N. C. (2002). Semantic representation and naming in children with specific language impairment. Journal of Speech, Language, and Hearing Research, 45(5), 998–1014 [Article] ×
Mason, M., Smith, M. & Hinshaw, M. (1976). Medida Española de articulación (Measurement of Spanish Articulation). San Ysidro, CA: San Ysidro School District.
Mason, M., Smith, M. & Hinshaw, M. (1976). Medida Española de articulación (Measurement of Spanish Articulation). San Ysidro, CA: San Ysidro School District.×
Mattes, L. (1985). Spanish articulation measures. Oceanside, CA: Academic Communication Associates.
Mattes, L. (1985). Spanish articulation measures. Oceanside, CA: Academic Communication Associates.×
Melgar de Gonzalez, M. (1976). Como detectar al niño con problemas del habla [Identifying the child with speech problems]. Mexico City: Trillas.
Melgar de Gonzalez, M. (1976). Como detectar al niño con problemas del habla [Identifying the child with speech problems]. Mexico City: Trillas.×
Paul, R. & Norbury, C.F. (2012). Language disorders from Infancy through adolescence. St. Louis, MO: Elsevier.
Paul, R. & Norbury, C.F. (2012). Language disorders from Infancy through adolescence. St. Louis, MO: Elsevier.×
Peña, E.D., Gutíerrez-Clellen, V.F., Iglesias, A., Goldstein, B. & Bedore, L.M. (2014). Bilingual English Spanish Assessment (BESA). AR-Clinical Publications.
Peña, E.D., Gutíerrez-Clellen, V.F., Iglesias, A., Goldstein, B. & Bedore, L.M. (2014). Bilingual English Spanish Assessment (BESA). AR-Clinical Publications.×
Restrepo, M. A. & Kruth, K. (2000). Grammatical characteristics of a Spanish-English bilingual child with specific language impairment. Communication Disorders Quarterly, 21(2), 66–76 [Article]
Restrepo, M. A. & Kruth, K. (2000). Grammatical characteristics of a Spanish-English bilingual child with specific language impairment. Communication Disorders Quarterly, 21(2), 66–76 [Article] ×
Rice, M. L. & Wexler, K. (1996). Toward tense as a clinical marker of specific language impairment in English-speaking children. Journal of Speech, Language, and Hearing Research, 39(6), 1239–1257 [Article]
Rice, M. L. & Wexler, K. (1996). Toward tense as a clinical marker of specific language impairment in English-speaking children. Journal of Speech, Language, and Hearing Research, 39(6), 1239–1257 [Article] ×
Sheng, L., Peña, E. D., Bedore, L. M. & Fiestas, C. E. (2012). Semantic deficits in Spanish-English bilingual children with language impairment. Journal of Speech, Language, and Hearing Research, 55(1), 1–15 [Article]
Sheng, L., Peña, E. D., Bedore, L. M. & Fiestas, C. E. (2012). Semantic deficits in Spanish-English bilingual children with language impairment. Journal of Speech, Language, and Hearing Research, 55(1), 1–15 [Article] ×
Toronto, A. (1977). Southwest Spanish articulation test. Oceanside, CA: National Education Laboratory Publishers, Inc.
Toronto, A. (1977). Southwest Spanish articulation test. Oceanside, CA: National Education Laboratory Publishers, Inc.×
Vermeer, A. & Shohov, S.P. (2004). Exploring the lexicon: Quantitative and qualitative aspects of children's L1/L2 word knowledge. In Advances in Psychology Research. Hauppauge, NY: Nova Science Publishers.
Vermeer, A. & Shohov, S.P. (2004). Exploring the lexicon: Quantitative and qualitative aspects of children's L1/L2 word knowledge. In Advances in Psychology Research. Hauppauge, NY: Nova Science Publishers.×