[读论文] Copying Mechanism in Sequence-to-Sequence

Paper Today:

'Incorporating Copying Mechanism in Sequence-to-Sequence Learning'

This paper develops a model called COPYNET which performs well in an important mechanism called 'copy mechanism'.

In human language communication, there are many situations that we will use 'copy mechanism', such as in a dialogue:

In order to make machine generate such dialogue, there are two things to do.

• First, to identify what should be copied.

• Second, to decide where the copy part should be addressed.

Currently there are some popular models like seq2seq, and adding Attention Mechanism to seq2seq.
COPYNET is also an encoder-decoder model, but a different strategy in neural network based models.
RNN and Attention Mechanism requires more 'understanding', but COPYNET requires high 'literal fidelity'.

There are mainly 3 improvements in the decoder part.
Prediction:
Based on the mix of two probabilistic modes, generate mode and copy mode, the model can pick the proper subsentence and generate some OOV words.

State Update:
There's a minor change that they designed a selective read for copy mode, which enables the model to notice the location information.

Reading M:
This model can get a hybrid of content based addressing and location based addressing.

In the experiment, this model did very well in tasks like text summarization.