Text and Data Mining

The American National Corpus (ANC) project is creating a massive electronic collection of American English, including texts of all genres and transcripts of spoken data produced from 1990 onward. The ANC will provide the most comprehensive picture of American English ever created, and will serve as a resource for education, linguistic and lexicographic research, and technology development.

The ANC has so far released 22 million words of American English, which is available from the Linguistic Data Consortium--please consult the LDC Catalog entry. The ANC has also released an "Open" portion of the full ANC consisting of approximately 15 million words, which is freely available for download. All ANC and OANC data include annotations for word and sentence boundaries, part of speech (4 tagsets), and noun and verb chunks. Parts of the corpus are annotated for additional linguistic features.

The British National Corpus (BNC) is a one hundred million word corpus of British English, both spoken and written. The BNC Online service allows you to search this corpus in a variety of ways and download citations from the corpus, using a computer connected to the Internet.

Home

The Bulgarian „Brown” Corpus is compiled in conformity with the methodology elaborated at the Brown university (Brown university, Providence, Rhode Island, USA) and used in the compilation of the well-known Brown Corpus of Standard American English. The Bulgarian „Brown” Corpus consists of 500 text samples distributed in 15 categories from two types of texts - fiction and informative prose. Samples' length is set at 2 000 words with the precise number of words varying, as the adopted methodology envisages respecting sentence boundaries. The corpus amounts to 1 001 286 words. The samples are excerpts from texts created or published for the first time in the period 1990-2005, the main part dated after 2000.

The corpus is supplied with the relevant documentation. Finally, checks for wrong replacement of Cyrillic letters with Latin characters, as well as spelling and punctuation checking were performed. The first version of the corpus was compiled in 2001-2002. Some of the principles underlying the Brown Corpus such as originality of the texts, recentness of creation, etc. had to be disregarded in order to provide sufficient coverage of all categories. The experience gained in creating first version along with the significant increase of publications made electronically available in the period 2002-2005 afforded and greatly facilitated the compilation of the second version of the corpus.

The Buckeye Corpus of conversational speech contains high-quality recordings from 40 speakers in Columbus OH conversing freely with an interviewer. The speech has been orthographically transcribed and phonetically labeled.

Bulgarian National Corpus (BulNC) is a large representative corpus of Bulgarian, publicly available for research and educational purposes. A web-based search application provides access to the corpus. BulNC is constantly enlarged and developed. The aim is to ensure representativeness and balance of the data by including texts from different modality (written and spoken), various time periods, domains and genres.

Assessing Language Development through Learner Corpora

While learner corpora, as with their native-speaker counterparts, have been used to compile dictionaries and grammars for learners based on typical problems and errors of usage found in learner data, they can also be used by teachers and learners themselves to observe and assess their own and one another's performance in a more relaxed and more anonymous way, without the strains of real-time monitoring. Even relatively small corpora consisting of 10,000 words can yield a massive amount of usage data. Such data can then be compared with, for example, native-speaker corpora to reveal what learners are not using, as well as how correctly or appropriately they are using the target language. Moreover, learner corpora can be diagnostic, tracking the progress of learners over the course of a semester, a year or a longer time period. Thus, learner corpora represent concrete empirical evidence of language development.

The Corpus of Contemporary American English (COCA) is the largest freely-available corpus of English, and the only large and balanced corpus of American English. It is much more than a simple online dictionary. The corpus was created by Mark Davies of Brigham Young University, and it is used by tens of thousands of users every month (linguists, teachers, translators, and other researchers). COCA is also related to other large corpora that we have created.

The corpus contains more than 425 million words of text and is equally divided among spoken, fiction, popular magazines, newspapers, and academic texts. It includes 20 million words each year from 1990-2011 and the corpus is also updated once or twice a year (the most recent texts are from March 2011). Because of its design, it is perhaps the only corpus of English that is suitable for looking at current, ongoing changes in the language (see the 2011 article in Literary and Linguistic Computing).

100,000 word file

The Corpus of Czech verse (CCV) is a lemmatized, phonetically, morphologically, metrically and strophically annotated corpus.*
Each lexical unit is provided with information about its basic word form (lemma), phonetic transcription and grammatical categories; each verse line is provided with information about its type of metre (iamb, trochee, etc.), length (n-foot), type of the end of a line (masculine, feminine, etc.) and the metrical pattern. (Currently, only syllabotonic verse lines are annotated in terms of metrics.) On higher levels rhyme pairs are annotated, or n-some and fixed forms (sonnet, rondel, etc.). In the metrical and strophical description it is possible to search by means of Database of Czech Metres; the lemmatization level is partly accessible through Frequency lists; rhyme pairs can be searched in the application Gunstick

The Czech National Corpus (CNC) is an academic project focusing on building a large electronic corpus of mainly written Czech. Institute of the Czech National Corpus (ICNC), Faculty of Arts, Charles University in Prague has been in charge of the CNC, its expansion, development and other related activities, particularly those associated with teaching and advancing the field of the corpus linguistics.

From Brigham Young University.
The Corpus of Global Web-Based English (GloWbE) is composed of 1.9 billion words from 1.8 million web pages in 20 different English-speaking countries. The corpus was created by Mark Davies of Brigham Young University, and it was released in 2013.

GloWbE (pronounced like "globe") is related to other large corpora that we have created, including the 450 million word Corpus of Contemporary American English (COCA) and the 400 million word Corpus of Historical American English (COHA). Together, these three corpora allow researchers to examine variation in English -- by dialect, genre, and over time -- in ways that are not possible with any other large corpora of English.

The International Corpus of English (ICE) began in 1990 with the primary aim of collecting material for comparative studies of English worldwide. Twenty research teams around the world are preparing electronic corpora of their own national or regional variety of English. Each ICE corpus consists of one million words of spoken and written English produced after 1989. For most participating countries, the ICE project is stimulating the first systematic investigation of the national variety. To ensure compatability among the component corpora, each team is following a common corpus design, as well as a common scheme for grammatical annotation.

The Text Creation Partnership creates standardized, accurate XML/SGML encoded electronic text editions of early print books. We transcribe and mark up the text from the millions of page images in ProQuest's Early English Books Online, Gale Cengage's Eighteenth Century Collections Online, and Readex's Evans Early American Imprints.

This work, and the resulting text files, are jointly funded and owned by more than 150 libraries worldwide. All of the TCP's work will be released the public domain for anyone to use.

See Access links to all categories of the database.

This website allows you to quickly and easily search more than 100 million words of text of American English from 1923 to the present, as found in TIME magazine. You can see how words, phrases and grammatical constructions have increased or decreased in frequency and see how words have changed meaning over time.

For over half a century, the UCLA Phonetics Laboratory has collected recordings of hundreds of languages from around the world, providing source materials for phonetic and phonological research, of value to scholars, speakers of the languages, and language learners alike. The materials on this site comprise audio recordings illustrating phonetic structures from over 200 languages with phonetic transcriptions, plus scans of original field notes where relevant.