Language learning, teaching, and testing with MERLIN

Here, we present several use cases for the MERLIN corpus. The scenarios relate to teaching practice, but also to the design of teaching materials as well as to the planning of courses and textbooks. Testers will find some practical examples in the last section.

MERLIN for teachers and material writers

Integrate MERLIN texts into the classroom 

Objective: Your students assess their own L2 writing competence with relation to CEFR levels and thus better understand where they are in the language learning process.

Scenario: Choose MERLIN texts that match the CEFR level of your learners or a specific test tasks that fits thematically with the course content.

Your learners ...

  • discuss strengths and weaknesses of an example text from the MERLIN corpus and compare it with the target hypothesis.
  • work on a MERLIN task and compare their own productions with the MERLIN texts.
  • evaluate each other's text productions with the help of the MERLIN rating grid (recommended from level B1).

And here's how:

  1. Download a suitable test task.
  2. Download the Merlin corpus and choose ↘ meta_ltext_THs (learner texts including target hypotheses).
  3. Search for texts with this task [Entrance page↘ Filter texts with the file manager].

Assess the learning progress of your learners

Objective: Compare your group with learners at the same or different competence levels

Scenario: Use the MERLIN texts to check ...

  • what difficulties other learners at the same competence level are having
  • where your learners are in comparison to learners with the same or a different L1.

And here's how:

  1. Download the Merlin corpus and choose meta_ltext_THs (learner texts including target hypotheses).
  2. Search in your file manager for texts at a specific level, e.g. B1, and with a specific mother tongue, e.g. Russian [↘ Entrance page↘ Filter texts with the file manager].

hint bulb Learners' proficiency often comes in a profile, so that a learner might be more successful in grammar than, for example, in vocabulary. You can use MERLIN to explore different aspects of learners' communicative L2 competence, e.g. vocabulary range/control, grammatical accuracy, coherence/cohesion, at different CEFR levels.

Examine features, irregularities and errors in your learners' texts 

Objective: Check incorrect use, overuse or underuse of words and structures  

Scenarios:

  • Errors in the use of a particular structure appear repeatedly in your learners' texts, e.g. in Czech mít rád is being used with the infinitive of a verb instead of an accusative object. You want to examine whether this phenomenon is typical and in which contexts it occurs.
  • You suggest that there is a correlation between the overuse or inappropriate use of a structure/lexical unit and the learners' mother tongue. For example, you observe the overuse of schon (already, yet) in written German texts of learners with Polish as their L1. You want to verify your assumption.

And here's how:

  1. Open the ANNIS search.
  2. Enter the word or lemma you are looking for ["schon"].
  3. Choose the corpus [Czech, German, Italian] and click  Search.
  4. To search only texts written by Polish L1 speakers, enter additionally into the search field: & meta::_author_L1="Polish" .

hint bulb For further explanations see Search and help.

Find examples for teaching materials in the MERLIN corpus

Objective: Explore crucial aspects of language learning, such as learners' use of collocations, verbal aspect, and mood, etc. and find suitable examples for your own materials.

Scenarios:

  • You want to illustrate the command of word order in the German main clause. 
  • You search for examples of incorrect use of a certain structure to add usage notes to your materials or suggestions to avoid its overuse or underuse. 
  • Your search for typical errors related to the use of a certain word or structure to identify practice focusses and find examples sentences.
  • You want to analyze and compare texts with regard to certain phenomena of learner language.

And here's how:

  1. Review in the Search and help in section 2 the list of annotated learner language features for the ones that interest you.
  2. Open the ANNIS search to search for those features, e.g. G_Refl_type. Proceed as described here Search and help.
  3. Now view the search results. Under full text you can explore the feature in context and copy phrases and sentences from the learner's text.

MERLIN for curriculum design and course planning

General overview of the text productions at a certain CEFR level 

Objective / Scenario: You want to get a general overview of the text productions at a certain CEFR level.

And here's how: To get a general impression of e.g. B1 texts, sort or filter the MERLIN texts by files that received a B1 rating (fair average). Proceed as described on the Entrance page under Filter texts with the file manager.

Identify relevant milestones in learner language

Objective: Identify relevant milestones of the L2 or typical errors (in relation to the levels of the CEFR) and facilitate decisions about the selection and sequencing of content in the curriculum and syllabus.

Scenarios

  • You want to examine which phenomena of learner language are especially relevant to the addressed CEFR level of the textbook or to the course level. You could do so by exploring MERLIN texts for phenomena that learners typically implement incorrectly at a certain CEFR level, but do apparently master at the next higher level.
  • In turn, you might want to verify whether learners master a phenomenon or typical learner "problem" already from a certain level. It thus has not to be focused any longer.
  • You would like to check which phenomena should be taken up again and again across different levels, e.g. by exploring which feature/error occurs at all proficiency levels in the MERLIN corpus.

And here's how:

  1. Use the ANNIS search and the   Query Builder to obtain information on the frequency of a feature/error for a distinct group of texts (e.g. writtten productions that received a B1 rating). Proceed as described under Search and help.
  2. After launching the query (↘ Search) you can see the number of the query results below the search window, e.g. 1460 matches in 244 documents.
  3. To get an impression of the frequency of phenomena within a given feature category, e.g. all features related to grammar, use the   Frequency Analysis in ANNIS as described under Search and help   4 Retrieve statistical information.

MERLIN for language testing

Using the CEFR systematically in test development 

Most European language tests are (or claim to be) related to the CEFR. While the Council of Europe provides numerous helpful materials, there is not yet much empirical data (i.e. CEFR-related language samples) to support the test development process, especially for languages other than English (an overview of the available reference level descriptions can be found on the web pages of the Council of Euope; for English see www.englishprofile.org). 

We believe that MERLIN data ...

  • can help to enhance transparency and quality in CEFR-related test construction for Czech, German, and Italian.
  • can be used for benchmarking purposes.
  • can foster empirically based development of assessment materials.

Furthermore, MERLIN data lends itself to the empirical validation of the CEFR scales (see MERLIN for research) and might be helpful for empirically based rating scale construction.

Adjusting the understanding of CEFR levels among raters

Objective: Create a common understanding of the CEFR levels in your institution 

Scenarios: Practise rating procedures of written L2 productions with your colleagues with the help of the MERLIN tasks and texts and compare your ratings with those of the MERLIN team.

  • Extract a random sample of written tests on a specific task, e.g. "Neujahrsbrief an einen Freund schreiben" and have your colleagues re-rate the texts using the MERLIN rating grid
  • Discuss your results and compare them to MERLIN ratings.

And here's how:

How to find learner texts for a specific task.

  1. Select the appropriate task in the task overview and download it.
  2. Download the Merlin corpus and choose  meta_ltext_THs (learner texts including target hypotheses).
  3. Search the file manager for texts with this task [↘ Entrance page↘ Filter texts with the file manager].

Links

Council of Europe (2011). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Council of Europe 2001
Council of Europe (2018). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion volume with new descriptors. Council of Europe 2018
Platform of resources and references for plurilingual and intercultural education
The English profile