This is an old revision of the document!

1.5 Sub-corpora

Based on the annotation of the languages per chat, different sub-corpora were created. Sub-corpora were built for the needs of the project members. Hopefully, they will be usefull for other research questions as well.

The following basic considerations were applied when creating the sub-corpora:

Definitions for sub-corpora

• Each chat was to be assigned to only one language-sub-corpus. As mentioned in the section languages per chat, each chat has an annotation for languages that occur in 100 and more messages (meta::lang_100_and_more). However, there are chats with e.g. more than 100 messages in French and in German. In that case, the language that provides most chats is used for the chat to be assigned to a language-sub-corpus. Example: if a chat is built up of 150 messages in French and 120 messages in German, it appears in the main corpus (WUS) as well as in the French sub-corpora (WUS_FRA and WUS_FRA_DEMOG) but not in any of the German sub-corpora.
• Additionally, we differentiate between chats where we have demographic information for all participants and those where we do not. In the former case, the sub-corpus gets the extension _DEMOG.
• Where additional tasks were performed on individual chats (e.g. normalization or part-of-speech tagging) we created additional sub-corpora per language.

Main sub-corpora

These rules result in the following main corpora:

• WUS: All data, i.e. the whole corpus
• WUS_DEU: All data where non-dialectal German provides the most messages
• WUS_DEU_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.
• WUS_FRA: All data where French provides the most messages
• WUS_FRA_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.
• WUS_GSW: All data where dialectal German provides the most messages
• WUS_GSW_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.
• WUS_ITA: All data where Italian provides the most messages
• WUS_ITA_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.
• WUS_ROH: All data where Romansh provides the most messages
• WUS_ROH_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.

Smaller corpora

Next to these main corpora, there are some smaller corpora:

• WUS_SMALL: chats that are either smaller than 100 messages or where the languages cannot be clearly distinguished
• WUS_SMALL_DEMOG: a subgroup thereof where we have the permission from all communication partners to use their texts.
• WUSdemographics: only demographic data per person. This corpus is much faster if you want to look up demographic data only.
• WUS_ARGDROP and WUS_ARGDROP_language: corpora, for which argument drop has been manually annotated.

More information about the subcorpora

The individual sub-corpora are well documented in terms of size etc. within the browsing tool. Check the according section for more information.