You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Achim Ruopp 2dd960ba69 Merge branch 'master' into dev 5 years ago
Results Results for ev2 changed for 100 cycle run 7 years ago
baseline More descriptive path names for extraction folders 5 years ago
candidates 100% speedup 7 years ago
crawlertest added usage 7 years ago
dicts Merge branch 'dev' of into dev 5 years ago
docalign_task check correct language identification 6 years ago
docaligner paper updates 6 years ago
html_convert Merge changes from MTM 2015 5 years ago
merge/metadata utils for metadata generation 7 years ago
metadata Updating meta-data building instructions 5 years ago
monolingual Adding monolingual data links 5 years ago
.gitignore ignore more files 7 years ago Adding hardware and OS requirements 5 years ago License changed to Apache License Version 2.0 6 years ago Adding link to installation instructions 5 years ago added page download option to metadata server 7 years ago
common_crawl_process.png Updating high-level process chart 5 years ago The results: document mapping and script for extracting info from ILSP Crawler 7 years ago
requirements.txt updated requirements 7 years ago


Collecting data for machine translation training from CommonCrawl is a two-phase process illustrated in the following diagram:

CommonCrawl process diagram


Hardware requirements and installation instructions can be found here.

Phase 1: Language annotation, building a meta-data file and monolingual data extraction

The first phase detects the languages of the web pages contained in the crawl and other meta-data. A meta-data file is built from this analysis.

The metadata documentation describes phase 1 step-by-step.

With data from this phase monolingual data for language model training can be extracted. The data for most of the CommonCrawl crawls and many languages can be found on:

Phase 2: Extracting parallel data and optional cleaning

In the second phase the meta-data collected in phase 1 is used to extract parallel data from CommonCrawl data based on URL pattern matching. Phase 2 is documented step-by-step in the baseline documentation

For the language pairs en↔de, en↔fr, en↔es, en↔it, en↔pt, en↔nl and en↔ru matched URL data for CommonCrawl 2015_32 is available for data extraction in release 0.1.0