DOCENT -- DOCUMENT-LEVEL LOCAL SEARCH DECODER FOR PHRASE-BASED SMT

Christian Hardmeier 26 September 2012

Sebastian Schleussner 13 July 2016

Docent is a decoder for phrase-based Statistical Machine Translation (SMT). Unlike most existing SMT decoders, it treats complete documents, rather than single sentences, as translation units and permits the inclusion of features with cross-sentence dependencies to facilitate the development of discourse-level models for SMT. Docent implements the local search decoding approach described by Hardmeier et al. (EMNLP 2012).

If you publish something that uses or refers to this work, please cite the following paper:

@inproceedings{Hardmeier:2012a, Author = {Hardmeier, Christian and Nivre, Joakim and Tiedemann, J"{o}rg}, Booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning}, Month = {July}, Pages = {1179--1190}, Publisher = {Association for Computational Linguistics}, Title = {Document-Wide Decoding for Phrase-Based Statistical Machine Translation}, Address = {Jeju Island, Korea}, Year = {2012}}

Requests and comments about this software can be addressed to [email protected]

DOCENT LICENSE

All code included in docent, except for the contents of the 'external' directory, is copyright 2012 by Christian Hardmeier. All rights reserved.

Docent is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Docent is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Docent. If not, see http://www.gnu.org/licenses/.

SNOWBALL STEMMER LICENSE

Docent contains code from the Snowball stemmer library downloaded from http://snowball.tartarus.org/ (in external/libstemmer_c). This is what the Snowball stemmer website says about its license (as of 4 Sep 2012):

"All the software given out on this Snowball site is covered by the BSD License (see http://www.opensource.org/licenses/bsd-license.html ), with Copyright (c) 2001, Dr Martin Porter, and (for the Java developments) Copyright (c) 2002, Richard Boulton.

Essentially, all this means is that you can do what you like with the code, except claim another Copyright for it, or claim that it is issued under a different license. The software is also issued without warranties, which means that if anyone suffers through its use, they cannot come back and sue you. You also have to alert anyone to whom you give the Snowball software to the fact that it is covered by the BSD license.

We have not bothered to insert the licensing arrangement into the text of the Snowball software."

BUILD INSTRUCTIONS

See file doc/Build.txt

USAGE

See also file doc/Usage.txt

1. File input formats

Docent accepts the NIST-XML and the MMAX formats for document input. Output is always produced in the NIST-XML format. MMAX input requires you to provide both a NIST-XML file and an MMAX directory, since the NIST-XML input file is used as a template for the output. NIST-XML output without MMAX can be used for models that process unannotated plain text input. MMAX input can be used to provide additional annotations, e.g. if the input has been annotated for coreference with a tool like BART (http://www.bart-coref.org/).

2. Decoder configuration

The decoder uses an XML configuration file format. There are two example configuration files in tests/config. The tag specifies the initialisation of the random number generation. When it is empty (), a random seed value is generated and logged at the beginning of the decoder run. In order to rerun a specific sequence of operations (for debugging), a seed value can be provided as shown in one of the example configuration files.

The only phrase table format currently supported is the binary phrase table format of moses (generated with processPhraseTable). Note that Docent, unlike moses, doesn't have a parameter to enforce a limit on the number of translations for a given phrase that are loaded from the phrase table. We recommend that you filter your phrase tables with the filter-pt tool from the Moses distribution and a setting like "-n 30" before using them with Docent. Otherwise you may experience very poor performance.

Language models should be in KenLM's probing hash format (other formats supported by KenLM can potentially be used as well).

There is some more documentation about the configuration file format on Docent's Wiki page: https://github.com/chardmeier/docent/wiki/Docent-Configuration

3. Invoking the decoder

There are three main docent binaries:

docent for decoding a corpus on a single machine without producing intermediate output

lcurve-docent for producing learning curves

mpi-docent for decoding a corpus on an MPI cluster

I recommend that you use lcurve-docent for experimentation.

Usage: lcurve-docent {-n input.xml|-m input.mmaxdir input.xml} config.xml outputstem

Use the -n or -m switch to provide NIST-XML input only or NIST-XML input combined with MMAX annotations, respectively. The file config.xml contains the decoder configuration.

The decoder produces output files for the initial search state and intermediate states whenever the step number is equal to a power of two, starting at 256. Output files are named outstem.000000000.xml outstem.000000256.xml etc.

4. Extending the decoder

To implement new feature functions, start with one of the existing. SentenceParityModel.cpp contains a very simple proof-of-concept feature function that promotes length parity per document (i.e. all sentences in a document should consistently have either odd or even length). If you're planning to implement something more complex, NgramModel.cpp or SemanticSpaceLanguageModel.cpp may be good places to start.

Good luck!