Assessing Literacy in a Digital World: Validating a Scenario-Based Reading-to-Write Assessment

ISBN-13: 9789811957147 / Angielski / Twarda / 2022 / 185 str.

Yumei Zhang

Opis

This book illustrates the latest developments in literacy and language assessment in the digital context, and subsequently presents a rigorous validation study on a newly proposed form of assessment (scenario-based assessment, SBA) that seeks to respond to the contextual change of literacy activities. It combines theories and innovative practices in both the literacy and language assessment sectors.The empirical validation study on SBA, presented here, can help readers understand how digital scenarios can be realized in assessment practices with the aid of computer technology, and how the scenario settings in the digital context can affect EFL learners’ reading-to-writing performance. In this way, it can facilitate the reconceptualization of L2 literacy in the digital context. Moreover, the evidence and critical examination presented here can offer readers more comprehensive insights into the value or validity of a given innovative approach before it is adopted in their contexts.   

Spis treści

Chapter 1 Introduction.- 1.1. Background.- 1.2 Purpose.- 1.3 Significance.- 1.4 Layout of the book.- Chapter 2 Literature Review.- 2.1 Changes of literacy contexts wrought by information technology.- 2.2 Scenario-based assessment.- 2.2.1 Definition and key features.- 2.2.2 Main concerns toward SBA.- 2.2.3 Previous research on SBA.- 2.3 Validity theories and validation methods.- 2.3.1 The evolution of validity theories and validation frameworks.- 2.3.2 Argument-based approach to validation.- 2.3.2.1 Scoring.- 2.3.2.2 Generalization.- 2.3.2.3 Extrapolation.- 2.3.2.4 Theory-based interpretations.- 2.4 Reading-to-write literate practice.- 2.4.1 Theoretical frameworks.- 2.4.2 Previous studies on reading-to-write ability.- 2.4.3 Argumentation as reading-to-write practice.- 2.5 Summary of the literature review.- Chapter 3 Research Methodology.- 3.1 Research questions.- 3.2 Theoretical framework.- 3.3 Participants.- 3.3.1 Experts.- 3.3.2 Student participants.- 3.3.3 Raters.- 3.4 Instruments.- 3.4.1 The SBRtW and RtW tasks.- 3.4.2 Expert judgement guideline.- 3.4.3 Stimulated recalls.- 3.4.4 Questionnaire.- 3.5 Procedures.- 3.5.1 Stage 1: Stimulated recalls.- 3.5.2 Stage 2: A quasi-experiment.- 3.5.3 Stage 3: The final assessment.- 3.6 Data analyses.- 3.6.1 Analyses of the stimulated recall data.- 3.6.2 Preliminary statistical analyses.- 3.6.3 Repeated measures ANOVA and paired sample t-tests.- 3.6.4 Analyses based on generalizability theory.- 3.6.5 Many-facet Rasch measurement.- Chapter 4 The Development of the SBRtW Assessment.- 4.1 Development procedure and guidelines.- 4.2 Purpose and potential use of the test.- 4.3 The scenarios and the features.- 4.4 Topics.- 4.5 Reading texts.- 4.6 Item types and formats.- 4.7 Scoring rubrics for the writing tasks.- 4.7.1 Bottom-up extraction of the holistic rubric and analytic criteria.- 4.7.2 Rubric refinement informed by theories and previous rubrics.- 4.7.3 Revision based on expert judgement and trial scaling.- Chapter 5 Results.- 5.1 Appraisal of the scoring inference.- 5.1.1 The appropriateness of the score rule.- 5.1.2 Rating accuracy and rater agreement.- 5.2 Appraisal of the generalization inference.- 5.2.1 The representativeness of the observation sample.- 5.2.2 Generalizability and bias analyses.- 5.3 Appraisal of extrapolation inference and theory-based interpretations.- 5.3.1 The authenticity of the SBRtW assessment tasks.- 5.3.2 The underlying processes in the SBRtW assessment.- 5.3.3 The usefulness of the SBA features in facilitating performance.- 5.3.4 The relationship between reading and writing.- 5.3.5 The progressiveness across grade levels.- 5.4 Summary of the results.- Chapter 6 Discussion.- 6.1 Validity argument for the soring inference.- 6.1.1 The importance of scoring rubric to construct validity.- 6.1.2 The role of scoring reliability in validity.- 6.2 Validity argument for the generalization inference.- 6.2.1 Task sampling for generalization.- 6.2.2 The effect of topical issues.- 6.3 Validity argument for extrapolation and theory-based interpretations.- 6.3.1 Task authenticity for validity.- 6.3.2 Underlying processes as an indication of attributes at play.- 6.3.3 The effect of scenario-based features on writing performance.- 6.3.4 The role of reading comprehension in writing performance.- 6.3.5 Variance of writing performance across grade levels.- Chapter 7 Conclusions.- 7.1 Summary of the main findings.- 7.2 Implications.- 7.3 Limitations and suggestions for future research.- References.- Appendices.- Appendix 1 The guidelines for expert judgments.- Appendix 2 The protocol for stimulated recall procedure (English Version).- Appendix 3 The questionnaire (Form 1 as an example).- Appendix 4 Informed consent.- Appendix 5 An example of the SBRtW assessment.- Appendix 6 The holistic scale for the writing.- Appendix 7 The analytic rating scale for writing

O autorze

Dr. Yumei Zhang is a lecturer and Chair of the English Education Department at Beijing International Studies University, and an applied linguistics researcher with substantial experience in second language acquisition and language education. Dr. Zhang’s research interests include second language acquisition, language teaching and assessment. In addition, she has extensive teaching experience at both the high school and university level.