The Importance of Narrative Assessments in Speech Language Pathology (Revised)

As SLPs we routinely administer a variety of testing batteries in order to assess our students’ speech-language abilities. Grammar, syntax, vocabulary, and sentence formulation get frequent and thorough attention. But how about narrative production? Does it get its fair share of attention when the clinicians are looking to determine the extent of the child’s language deficits? I was so curious about what the clinicians across the country were doing that in 2013, I created a survey and posted a link to it in several SLP-related FB groups.  I wanted to find out how many SLPs were performing narrative assessments, in which settings, and with which populations.  From those who were performing these assessments, I wanted to know what type of assessments were they using and how they were recording and documenting their findings.   Since the purpose of this survey was non-research based (I wasn’t planning on submitting a research manuscript with my findings), I only analyzed the first 100 responses (the rest were very similar in nature) which came my way, in order to get the general flavor of current trends among clinicians, when it came to narrative assessments. Here’s a brief overview of my [limited] findings.

Of the first 100 clinician’s whose responses were analyzed, the two largest groups were composed of SLPs who had been practicing for over 15 years (39%) as well as those who had been practicing from 5-10 years (18%).

Of course it was not a surprise that 76% of these practitioners were based in school setting followed by 23% of practitioners in private practice. Out of them only about 74% of clinicians had ever assessed narratives. These numbers went further on decrease when asked how often these narrative assessments took place: 40% of clinicians stated that they’ve done it rarely with <25% of students, followed by 28% who stated that they performed narrative assessments only some of the time with <50% of students.

When asked if the clinicians favored standardized instruments, non-standardized instruments, or both, the responses were split. Out of those SLPs who performed narrative assessments (see above), 52% favored standardized instruments while a whopping 68% favored non-standardized instruments, indicating an overlap in what instruments were used by clinicians.

However, it was when I asked the question: ‘What type of narrative assessments have you administered?’ that I received the most interesting responses. More than 50% of clinicians preferred the Test of Narrative Language (TNL-2 is now available). Another 27% preferred non-specific dynamic assessments and yet another 27% performed a wide variety of other assessments, which included but not limited to: informal language samples analyses (e.g., One Frog Too Many), Peter and the Cat Narrative Assessment, MindWing concepts’ Story Grammar Marker (SGM), Developmental Sentence Score (DSS), Michelle Garcia Winner’s Dynamic Assessment Protocol, informal story retelling analysis via SALT software,  Assessment of Preschool Narrative Skills (McCabe and Rollins, 1994), The Edmonton Narrative Norms Instrument  (ENNI), as well as a numerous other instruments.

One thing which became immediately apparent by reviewing the survey results is that SLPs who used non-standardized instruments clearly used an impressively wide variety of assessments. Another was that some of the SLP-listed instruments did not actually assess narrative skills.  For example some survey respondents stated that they used the Test of Word Finding in Discourse (TWFD). However, this test actually assesses conversational skills and not narrative skills. A few responders stated that they used “responses on the definitions subtest of the Word Test – 2” (WORD-3 is now available), as a narrative measure, however, those responses can at best count as an informal assessment of spontaneous sentence formulation vs. a true narrative assessment.

So what actually constitutes a true narrative assessment and why are narrative assessments so important?

Well, in order for an assessment to count as a narrative assessment, an SLP must listen to some kind of prompted/unprompted story recounting some form of an event (instead of engaging in a back-and-forth conversational exchange with the child or listening to him/her formulate a brief response based on presented questions).

There are several different types of narratives which can be assessed:

• Personal
• Fictional  
• Expository 
• Persuasive 

Out of these narrative types, accounts of personal experience, as well as the retelling of fictional stories, are used most frequently by researchers to assess younger children’s narrative competence (whereas expository and persuasive narratives are more useful for adolescent assessment purposes). Here are just some of the reasons why these are preferred.

Personal narratives constitute a large section of functional discourse commonly found in the spontaneous conversation of typically developing preschool and school-aged children. They are sensitive to narrative competence (Bliss & McCabe 2012). They require the speaker to organize an account or a recount of past experiences. They provide effective ways to tax the system due to unplanned discourse (Hadley, 1998; Hughes, 2001). Although they are natural to tell, they also require that the storyteller makes the story interesting to listeners so not to lose their attention (Hughes, 2001).  Finally, they are important for connecting with family and friends as well as for literacy acquisition (Bliss & McCabe 2012).

Similarly, fictional narratives tap into the child’s knowledge of organizational framework and are sensitive to language impairment (McCabe & Rosenthal-Rollins, 1994).  They have been routinely been identified as one of the single best predictors of future academic success (Bishop & Edmundson, 1987; Feegans & Appelbaum 1986). Poor discourse and narrative abilities place children at risk for learning and literacy-related difficulties including reading problems (McCabe & Rosenthal-Rollins, 1994).

Narrative analyses help to distinguish children with language impairment from their typically developing peers (Allen et al 2012).  Furthermore, language produced during story retelling is positively related to bilingual reading achievement (Miller et al, 2006).

Narrative retelling bridges the gap between oral and written language and is important for appropriate reading and writing development (Snow et al, 1998). It provides insights into child’s verbal expression since it taps into multiple language features and organizational abilities simultaneously (Hoffman, 2009; McCabe & Bliss, 2003; Ukrainetz, 2006). This is not the case with standardized testing. Many children may evidence appropriate performance on standardized testing but still display significant difficulties in the area of narrative production. However, unless SLPs assess narratives in children specifically, they run the risk of overlooking weaknesses in this area of functioning.

Finally, assessment of narratives is very important because poor storytelling skills are indicative of social pragmatic language impairment. In fact, part of the diagnostic criteria of social pragmatic communication disorder in the DSM-V™ is “difficulties following rules for conversation and storytelling”.

So what are some ways in which SLPs can assess narratives?

When it comes to narrative assessment SLPs have many options available to them (as illustrated above). Depending on the age and type of the population they are working with (e.g., school-age vs. preschool) they can choose from a variety of standardized, non-standardized, or dynamic assessments.   However, when choosing the assessment it is very important to ensure that it examines both microstructure and macrostructure domains of storytelling.

When analyzing microstructure we are looking at the child’s grammar, syntax and vocabulary competence (content and form). In contrast, when we examine macrostructure we are looking at the child’s use of language. Is the story organized and cohesive? (Justice et al, 2006) Does it contain the story grammar elements commensurate with their age such as setting, initiating event, internal response, attempt, consequences, and reaction (Stein & Glenn, 1979)?

For school children 5;00-11;11 years of age, the standardized assessment, Test of Narrative Language-2 TNL-2 (2017) examines the microstructure and macrostructure elements, as well as measures the child’s ability to answer literal and inferential comprehension questions.   It costs $201. I have not had the opportunity to review TNL-2. However, the previous version of TNL  had notable limitations.  It was more effective at identifying narrative deficits in severely language impaired children vs. children with more subtler degree of impairment.  The orginal TNL, test-retest reliabilities were not separately calculated for different ages. As per Baxter & Van Lingen, 2005, the test-retest data was based on a small, non-representative group and was not strong enough for clinical decision-making (p. 1041). It also did not equally represent all tested age-groups and SES levels. For example, the 5-year old group (n=83) was small and almost ½ of the number of the other groups and “upper-income” children were overrepresented in the sample (Baxter & Van Lingen, 2005, p. 1041). As many other standardized tests, TNL favored children with “mainstream” experience and is not recommended for use with multicultural/low SES children from “low print”/”low talk” environments. (Hayward et al 2008).

According to the developers, the TNL-2 has been improved in three major ways:

1. New, more colorful pictures
2. More inferential comprehension questions
3. Consistent scoring format

They also increased the number of items on the comprehension tasks and added more inferential questions as well as more questions for the youngest, low-functioning and the oldest, high-functioning children.  Supposedly scoring for the oral narratives is now similar across the three types of stories which makes  the TNL-2 easier to score, and more reliable (as per authors).

To continue, if you are working with preschool/kindergarten aged children, then you might want to consider either the Narrative Assessment Protocol (NAP) by Pence, Justice, & Gosse (2007) or the Narrative Language Measures (NLM) by Petersen & Spencer (2012) both of which are available online for free.

Let’s begin with NAP. Developed for children 3-6 years of age, it uses Mercer Mayer’s (1969) ‘Frog, Where are You?’ book to analyze the child’s story microstructure (e.g., sentence and phrase structures, modifiers, nouns and verbs). According to the manual, the macrostructure domain of the NAP is currently under development. However, since the most recent manual was last updated in August 2008, I am not quite certain how up-to-date this information really is. To continue NAP is fairly easy to use. The child is first shown a book then asked to tell a story using pictures as prompts. One of the advantages of NAP is that it does not require narrative transcription and manual utterance tallying, since the examiner scores the narrative as the story is being told. However, the lack of macrostructure analysis is a big disadvantage in my eyes, which is why I personally do not use this assessment when I analyze the narratives of preschool children.

Next up is the NLM.  Referred to by its authors as the free RTI focused assessment tool, the NLM has two developed versions: NLM:P and NLM:K for preschool and kindergarten children respectively. NLM: Listening assesses a student’s ability to comprehend and produce complex language that is reflective of written language. Students are asked to listen to a brief narrative, and then retell that narrative. —Narrative retells incorporate listening comprehension, cognition, memory, and expressive language skills. Scoring takes place in real-time. Additional sections of the NLM allow for the examination of story grammar comprehension and inferential word learning or word use. A personal story generation can also be elicited to monitor a student’s expressive language.

NLM For Preschool: is a set of 3 FREE RTI focused assessment tools for preschool children which measures various dimensions of narrative language. It contains:

Test of Narrative Retell (TNR)

• Read a model story
• Ask for a retell

Test of Story Comprehension (TSC)

• Read story
• Ask factual and inferential questions
• For younger children
• Extremely limited language
• Less demanding than TNR

Test of Personal Generation (TPG)

• Tell a model story
• Ask child to tell own thematically related personal story

NLM for School-Aged Children K-3rd grades is based on Stein and Glenn (1979) story grammar (see below) and consists of 2 FREE tests:

• Test of Narrative Retell (TNR: SA)
• Test of Personal Generation (TPG SA)

Both use the same 25 short stories that have a consistent structure, length, and language complexity and highlight events that young children are likely to experience in their daily lives in US (e.g., getting hurt, losing something). Both —include real‐time scoring forms.

NLM administration time approx 2–5 min per subtest. TNR and TSC can be scored in real time while the child is narrating.  Scoring rubrics contain 0–2 or 0–3 point ratings and 2 critical subscales story grammar and language complexity. NLM limitations include child’s lack of exposure to testing, cultural considerations, lack of norms or cut scores to help identify at risk children. Furthermore,  single subtest administration should be avoided, and if infrequently probed (seasonal) the administration of all 3 subtests is recommended for validity purposes.

Personally, I prefer informal narrative assessments to the standardized ones.  There are many advantages to informal assessments.  They provide more detailed information regarding microstructural and macrostructural elements as well as child’s thought processes and socio-emotional functioning. They are based on least biased assessment principle  (Goldstein, 2006), which means that you can make adaptations/modifications if you are using a standardized protocol, you can take into account cultural considerations, and of course spontaneous language sample measures that can supplement and clarify diagnostic information from standardized assessments (Rojas & Iglesias, 2009).

Here’s how I perform mine. I use the books recommended by the SALT Software website because it offers convenient suggestions by age, ranging from preschool through 6^th grade. I typically read the story to the child first then cover up the words and ask the child to retell the story based on pictures alone. I read the story first because: “the model narrative presents the events, plot structure, and words that the narrator is to retell, which allows more reliable scoring than a generated story that can go in many directions” (Allen et al, 2012, p. 207).

As the child is retelling the story I digitally record his/her narrative for transcription and thorough analysis.  During story telling, I use the prompts such as: ‘What else can you tell me?’ and ‘Can you tell me more?’ to elicit additional information, however, I try not to prompt the child excessively; otherwise the child is producing prompted responses vs. telling me a spontaneous story.

So what am I looking at when I am performing my narrative analysis?

Well, in the words of Pookins from Helen Lester’s ‘Pookins Gets Her Way’: “Lots!”

• Sequencing Ability
  • Is the story order appropriate or jumbled up?
•  Working Memory
  •  Is the child using enough details?
  • Are these details relevant to the story?
• Grammar
  • Does the sentence structure contains errors such as run-on sentences or incorrectly produced words
  • Is the child using temporal markers and cohesive ties to connect the story together
  • Is the child’s sentence complexity adequate for their age?
• Vocabulary
  • Is it immature or age-level?
  • Does the child have word retrieval issues or is s/he maintaining lexical fluency?
• Pragmatics
  • Is there story cohesion and coherence?
  • Is there adequate use of anaphoric references (pronouns identifying characters)?
• Perspective Taking
  • Does the child have appropriate insight into character’s feelings, beliefs, thoughts?

I evaluate these areas using the guidelines set forth in Hedberg & Westby, 1993 “Analyzing Storytelling Skills” book, which offers a comprehensive guide to narrative analysis.  After I finish, I include the transcription and a detailed write-up in the body of my language report, so parents and professionals could see it and understand the nature of the child’s errors/weaknesses. Another reason why I include the transcription in the body of my report is because it allows parents and professionals to compare past results with future gains and to see in which specific areas (e.g., story complexity, sentence structure, perspective taking, use of details etc) progress was made following therapeutic intervention.

Because storytelling encompasses a number of higher-level language and cognitive skills (Paul et al, 1996), I believe that it should be an integral component of every language evaluation since no other assessment yields as much information about the child’s language abilities as the narrative one.

Now that I’ve told you how I perform narrative assessments, I’d love to know how you perform them and what formal/informal instruments do you use?

Helpful Smart Speech Therapy Resources: 

• Narrative Assessment Bundle
• Narrative Assessments of Preschool and School Aged Children
• Vocabulary Development: Working With Disadvantaged Populations
• Understanding Complex Sentences

 References:

1. Allen, M,  Ukrainetz, T & Carswell, A (2012) The narrative language performance of three types of at-risk first-grade readers. Language, Speech, and Hearing Services in Schools, 43(2), 205-221.
2. Baxter, A. & Van Lingen, G. (2005). Test review of the Test of Narrative Language. In Spies, R.A. and B. S. Plake (Eds.), The sixteenth mental measurement yearbook (pp. 1040-1045). Lincoln, NE: Buros Institute of Mental Measurements.
3. Bishop, D. V. M., & Edmundson, A. (1987). Language impaired 4-year-olds: Distinguishing transient from persistent impairment. Journal of Speech and Hearing Disorders, 52, 156–173.
4. Bliss, L. & McCabe, A (2012, Oct) Personal Narratives: Assessment and Intervention. Perspectives on Language Learning and Education. 19:130-138
5. Feegans, L.,& Appelbaum, M (1986). Validation of language subtypes in learning disabled children. Journal of Educational Psychology, 78, 358–364.
6. Goldstein, B. A. (2006). Clinical implications of research on language development and disorders in bilingual children. Topics in Language Disorders, 26(4), 305–321.
7. Hayward, D. V., Stewart, G. E., Phillips, L. M., Norris, S. P., & Lovell, M. A. (2008). Test review: Test of narrative language (TNL). Language, Phonological Awareness, and Reading Test Directory (pp. 1-12). Edmonton, AB: Canadian Centre for Research on Literacy. Retrieved 9/2/13 from http://www.ualberta.ca/~lphillip/documents/The%20Test%20of%20Narrative%20Language%20(TNL).doc
8. Hedberg, N. & Westby, C. (1993). Analyzing story-telling skills: Theory to Practice. Tucson, AZ: Communication Skill Builders.
9. Hoffman, L. M. (2009a). Narrative language intervention intensity and dosage: Telling the whole story. Topics in Language Disorders, 29, 329–343.
10. Hughes, D., (2001, Oct) Assessment of Narrative Abilities in Preschool and School-Age Children. Perspectives in Language Learning and Education.  pp. 7-11.
11. Justice, L. M., Bowles, R. P., Kaderavek, J. N., Ukrainetz, T. A., Eisenberg, S. L., & Gillam, R. B. (2006). The index of narrative microstructure: A clinical tool for analyzing school-age children’s narrative performance. American Journal of Speech-Language Pathology, 15, 177–191.
12. McCabe, A., & Rollins, P. R. (1994). Assessment of preschool narrative skills. American Journal of Speech-Language Pathology, 3(1), 45–56
13. Miller, J et al (2006). Oral language and reading in bilingual children. Learning Disabilities Research and Practice, 21, 30–43.
14. Paul R, Hernandez R, Taylor L, Johnson K. (1996) Narrative development in late talkers: early school age. Journal of Speech and Hearing Research, 39(6):1295–1303
15. Rojas, R.  & Iglesias, A. (2009, March 03). Making a Case for Language Sampling : Assessment and Intervention With (Spanish-English) Second Language Learners. The ASHA Leader.
16. Stein, N., & Glenn, C. (1979). An analysis of story comprehension in elementary school children. In R. O. Freedle (Ed.), New directions in discourse processing (Vol. 2, pp. 53-120). Norwood, NJ: Ablex.
17. Ukrainetz, T. A. (2006b). Teaching narrative structure: Coherence, cohesion, and captivation. In T. A. Ukrainetz (Ed.), Contextualized language intervention: Scaffolding PreK–12 literacy achievement (pp. 195–246). Austin, TX: Pro-Ed.