Dependency parsing

Because the shared tasks of CoNLL-2006 and CoNLL-2007 focused on data-driven dependency parsing, it has recently been extensively studied in parsing research. The aim of dependency pars- ing is to compute a tree structure of a sentence where nodes are words, and edges re- present the relations among words. Figure 1 shows a dependency tree for the sentence “IL-8 recognizes and activates CXCR1.” An advantage of dependency parsing is that dependency trees are a reasonable approxima- tion of the semantics of sentences, and are readily usable in NLP applications. Furthermore, the effi- ciency of popular approaches to dependency pars- ing compare favorable with those of phrase struc- ture parsing or deep parsing. While a number of ap- proaches have been proposed for dependency pars- ing, this paper focuses on two typical methods.

Figure 1: CoNLL-X dependency tree

MST McDonald and Pereira (2006)’s dependency parser,http://sourceforge.net/projects/mstparser based on the Eisner algorithm for projective dependency parsing (Eisner, 1996) with the second- order factorization. KSDEP Sagae and Tsujii (2007)’s dependency parser,http://www.cs.cmu.edu/~sagae/parser/ based on a probabilistic shift-reduce al- gorithm extended by the pseudo-projective parsing technique (Nivre and Nilsson, 2005).

Phrase structure parsing

Owing largely to the Penn Treebank, the mainstream of data-driven parsing research has been dedicated to the phrase structure parsing. These parsers output Penn Treebank-style phrase structure trees, although function tags and empty categories are stripped off (Figure 2). While most of the state-of-the-art parsers are based on probabilistic CFGs, the parameteriza- tion of the probabilistic model of each parser varies. In this work, we chose the following four parsers.

Figure 2: Penn Treebank-style phrase structure tree

NO-RERANK Charniak (2000)’s parser, based on a lexicalized PCFG model of phrase structure trees.http://bllip.cs.brown.edu/resources.shtml The probabilities of CFG rules are parameterized on carefully hand-tuned extensive information such as lexical heads and symbols of ancestor/sibling nodes. RERANK Charniak and Johnson (2005)’s rerank- ing parser. The reranker of this parser receives n- bestWe set n=50 in this paper. parse results from NO-RERANK, and selects the most likely result by using a maximum entropy model with manually engineered features. BERKELEY Berkley's parser (Petrov and Klein, 2007).http://nlp.cs.berkeley.edu/Main.html#Parsing The parameterization of this parse is op- timized automatically by assigning latent variables to each nonterminal node and estimating the param- eters of the latent variables by the EM algorithm (Matsuzaki et al., 2005). STANFORD Stanford’s unlexicalized parser (Klein and Manning, 2003).http://nlp.stanford.edu/software/lex-parser.shtml Unlike NO-RERANK, proba- bilities are not parameterized on lexical heads.

Deep parsing

Recent research developments have allowed for ef- ficient and robust deep parsing of real-world texts (Kaplan et al., 2004; Clark and Curran, 2004; Miyao and Tsujii, 2008). While deep parsers compute theory-specific syntactic/semantic structures, pred- icate argument structures (PAS) are often used in parser evaluation and applications. PAS is a graph structure that represents syntactic/semantic relations among words (Figure 3). The concept is therefore similar to CoNLL dependencies, though PAS ex- presses deeper relations, and may include reentrant structures. In this work, we chose the two versions of the Enju parser (Miyao and Tsujii, 2008).

Figure 3: Predicate argument structure

ENJU The HPSG parser that consists of an HPSG grammar extracted from the Penn Treebank, and a maximum entropy model trained with an HPSG treebank derived from the Penn Treebank.http://www-tsujii.is.s.u-tokyo.ac.jp/enju/ ENJU-GENIA The HPSG parser adapted to biomedical texts, by the method of Hara et al. (2007). Because this parser is trained with both WSJ and GENIA, we compare it parsers that are retrained with GENIA (see section 3.3).

PPI extraction is an NLP task to identify protein pairs that are mentioned as interacting in biomedical papers. Because the number of biomedical papers is growing rapidly, it is impossible for biomedical re- searchers to read all papers relevant to their research; thus, there is an emerging need for reliable IE tech- nologies, such as PPI identification.

Figure 4 shows two sentences that include pro- tein names: the former sentence mentions a protein interaction, while the latter does not. Given a pro- tein pair, PPI extraction is a task of binary classi- fication; for example, <IL-8, CXCR1> is a positive example, and <RBP, TTR> is a negative example. Recent studies on PPI extraction demonstrated that dependency relations between target proteins are ef- fective features for machine learning classifiers (Ka- trenko and Adriaans, 2006; Erkan et al., 2007; Sætre et al., 2007). For the protein pair IL-8 and CXCR1 in Figure 4, a dependency parser outputs a depen- dency tree shown in Figure 1. From this dependency tree, we can extract a dependency path shown in Fig- ure 5, which appears to be a strong clue in knowing that these proteins are mentioned as interacting.

Figure 4: Sentences including protein names

Figure 5: Dependency path

We follow the PPI extraction method of Sætre et al. (2007), which is based on SVMs with SubSet Tree Kernels (Collins and Duffy, 2002;Moschitti, 2006), while using different parsers and parse rep- resentations. Two types of features are incorporated in the classifier. The first is bag-of-words features, which are regarded as a strong baseline for IE sys- tems. Lemmas of words before, between and after the pair of target proteins are included, and the linear kernel is used for these features. These features are commonly included in all of the models. Filtering by a stop-word list is not applied because this setting made the scores higher than Sætre et al. (2007)’s set- ting. The other type of feature is syntactic features. For dependency-based parse representations, a de- pendency path is encoded as a flat tree as depicted in Figure 6 (prefix “r” denotes reverse relations). Be- cause a tree kernel measures the similarity of trees by counting common subtrees, it is expected that the system finds effective subsequences of dependency paths. For the PTB representation, we directly en- code phrase structure trees.

Figure 6: Tree representation of a dependency path

Conversion of parse representations

It is widely believed that the choice of representa- tion format for parser output may greatly affect the performance of applications, although this has not been extensively investigated. We should therefore evaluate the parser performance in multiple parse representations. In this paper, we create multiple parse representations by converting each parser’s de- fault output into other representations when possi- ble. This experiment can also be considered to be a comparative evaluation of parse representations, thus providing an indication for selecting an appro- priate parse representation for similar IE tasks.

Figure 7 shows our scheme for representation conversion. This paper focuses on five representa- tions as described below.

Figure 7: Conversion of parse representations

CoNLL The dependency tree format used in the 2006 and 2007 CoNLL shared tasks on dependency parsing. This is a representation format supported by several data-driven dependency parsers. This repre- sentation is also obtained from Penn Treebank-style trees by applying constituent-to-dependency conver- sionhttp://nlp.cs.lth.se/pennconverter/ (Johansson and Nugues, 2007). It should be noted, however, that this conversion cannot work perfectly with automatic parsing, because the con- version program relies on function tags and empty categories of the original Penn Treebank. PTB Penn Treebank-style phrase structure trees without function tags and empty nodes. This is the default output format for phrase structure parsers. We also create this representation by converting ENJU’s output by tree structure matching, although this conversion is not perfect because forms of PTB and ENJU output are not necessarily compatible. HD Dependency trees of syntactic heads (Fig- ure 8). This representation is obtained by convert- ing PTB trees. We first determine lexical heads of nonterminal nodes by using Bikel’s implementation of Collins’ head detection algorithmhttp://www.cis.upenn.edu/~dbikel/software.html (Bikel, 2004; Collins, 1997). We then convert lexicalized trees into dependencies between lexical heads. SD The Stanford dependency format (Figure 9). This format was originally proposed for extracting dependency relations useful for practical applica- tions (de Marneffe et al., 2006). A program to con- vert PTB is attached to the Stanford parser. Although the concept looks similar to CoNLL, this representa- tion does not necessarily form a tree structure, and is designed to express more fine-grained relations such as apposition. Research groups for biomedical NLP recently adopted this representation for corpus anno- tation (Pyysalo et al., 2007a) and parser evaluation (Clegg and Shepherd, 2007; Pyysalo et al., 2007b). PAS Predicate-argument structures. This is the de- fault output format for ENJU and ENJU-GENIA.

Figure 8: Head dependencies

Figure 9: Stanford dependencies

Although only CoNLL is available for depen- dency parsers, we can create four representations for the phrase structure parsers, and five for the deep parsers. Dotted arrows in Figure 7 indicate imper- fect conversion, in which the conversion inherently introduces errors, and may decrease the accuracy. We should therefore take caution when comparing the results obtained by imperfect conversion. We also measure the accuracy obtained by the ensem- ble of two parsers/representations. This experiment indicates the differences and overlaps of information conveyed by a parser or a parse representation.

Domain portability and parser retraining

Since the domain of our target text is different from WSJ, our experiments also highlight the domain portability of parsers. We run two versions of each parser in order to investigate the two types of domain portability. First, we run the original parsers trained with WSJSome of the parser packages include parsing models trained with extended data, but we used the models trained with WSJ section 2-21 of the Penn Treebank. (39832 sentences). The results in this setting indicate the domain portability of the original parsers. Next, we run parsers re-trained with GE- NIA The domains of GENIA and AImed are not exactly the same, because they are collected independently. (8127 sentences), which is a Penn Treebank- style treebank of biomedical paper abstracts. Accu-racy improvements in this setting indicate the pos- sibility of domain adaptation, and the portability of the training methods of the parsers. Since the parsers listed in Section 2 have programs for the training with a Penn Treebank-style treebank, we use those programs as-is. Default parameter settings are used for this parser re-training.

In preliminary experiments, we found that de- pendency parsers attain higher dependency accuracy when trained only with GENIA. We therefore only input GENIA as the training data for the retraining of dependency parsers. For the other parsers, we in- put the concatenation of WSJ and GENIA for the retraining, while the reranker of RERANK was not re- trained due to its cost. Since the parsers other than NO-RERANK and RERANK require an external POS tagger, a WSJ-trained POS tagger is used with WSJ- trained parsers, and geniatagger (Tsuruoka et al., 2005) is used with GENIA-retrained parsers.

Experiment settings

In the following experiments, we used AImed (Bunescu and Mooney, 2004), which is a popular corpus for the evaluation of PPI extraction systems. The corpus consists of 225 biomedical paper ab- stracts (1970 sentences), which are sentence-split, tokenized, and annotated with proteins and PPIs. We use gold protein annotations given in the cor- pus. Multi-word protein names are concatenated and treated as single words. The accuracy is mea- sured by abstract-wise 10-fold cross validation and the one-answer-per-occurrence criterion (Giuliano et al., 2006). A threshold for SVMs is moved to adjust the balance of precision and recall, and the maximum f-scores are reported for each setting.

Comparison of accuracy improvements

Tables 1 and 2 show the accuracy obtained by using the output of each parser in each parse representa- tion. The row “baseline” indicates the accuracy ob- tained with bag-of-words features. Table 3 shows the time for parsing the entire AImed corpus, and Table 4 shows the time required for 10-fold cross validation with GENIA-retrained parsers.

Table 1: Accuracy on the PPI task with WSJ-trained parsers (precision/recall/f-score)

Table 2: Accuracy on the PPI task with GENIA-retrained parsers (precision/recall/f-score)

Table 3: Parsing time (sec.)

Table 4: Evaluation time (sec.)

When using the original WSJ-trained parsers (Ta- ble 1), all parsers achieved almost the same level of accuracy — a significantly better result than the baseline. To the extent of our knowledge, this is the first result that proves that dependency parsing, phrase structure parsing, and deep parsing perform equally well in a real application. Among these parsers, RERANK performed slightly better than the other parsers, although the difference in the f-score is small, while it requires much higher parsing cost.

When the parsers are retrained with GENIA (Ta- ble 2), the accuracy increases significantly, demon- strating that the WSJ-trained parsers are not suffi- ciently domain-independent, and that domain adap- tation is effective. It is an important observation that the improvements by domain adaptation are larger than the differences among the parsers in the pre- vious experiment. Nevertheless, not all parsers had their performance improved upon retraining. Parser retraining yielded only slight improvements for RERANK, BERKELEY, and STANFORD, while larger improvements were observed for MST, KSDEP, NO- RERANK, and ENJU. Such results indicate the dif- ferences in the portability of training methods. A large improvement from ENJU to ENJU-GENIA shows the effectiveness of the specifically designed do- main adaptation method, suggesting that the other parsers might also benefit from more sophisticated approaches for domain adaptation.

While the accuracy level of PPI extraction is the similar for the different parsers, parsing speed differs significantly. The dependency parsers are much faster than the other parsers, while the phrase structure parsers are relatively slower, and the deep parsers are in between. It is noteworthy that the dependency parsers achieved comparable accuracy with the other parsers, while they are more efficient.

The experimental results also demonstrate that PTB is significantly worse than the other representa- tions with respect to cost for training/testing and contributions to accuracy improvements. The con- version from PTB to dependency-based representa- tions is therefore desirable for this task, although it is possible that better results might be obtained with PTB if a different feature extraction mechanism is used. Dependency-based representations are com- petitive, while CoNLL seems superior to HD and SD in spite of the imperfect conversion fromPTB to CoNLL. This might be a reason for the high per- formances of the dependency parsers that directly compute CoNLL dependencies. The results for ENJU- CoNLL and ENJU-PAS show that PAS contributes to a larger accuracy improvement, although this does not necessarily mean the superiority of PAS, because two imperfect conversions, i.e., PAS-to-PTB and PTB-to- CoNLL, are applied for creating CoNLL.

Parser ensemble results

Table 5 shows the accuracy obtained with ensembles of two parsers/representations (except the PTB for- mat). Bracketed figures denote improvements from the accuracy with a single parser/representation. The results show that the task accuracy significantly improves by parser/representation ensemble. Inter- estingly, the accuracy improvements are observed even for ensembles of different representations from the same parser. This indicates that a single parse representation is insufficient for expressing the true potential of a parser. Effectiveness of the parser en- semble is also attested by the fact that it resulted in larger improvements. Further investigation of the sources of these improvements will illustrate the ad- vantages and disadvantages of these parsers and rep- resentations, leading us to better parsing models and a better design for parse representations.

Table 5: Results of parser/representation ensemble (f-score)

Comparison with previous results on PPI extraction

PPI extraction experiments on AImed have been re- ported repeatedly, although the figures cannot be compared directly because of the differences in data preprocessing and the number of target protein pairs (Sætre et al., 2007). Table 6 compares our best re- sult with previously reported accuracy figures. Giu- liano et al. (2006) and Mitsumori et al. (2006)) do not rely on syntactic parsing, while the former ap- plied SVMs with kernels on surface strings and the latter is similar to our baseline method. Bunescu and Mooney (2005) applied SVMs with subsequence kernels to the same task, although they provided only a precision-recall graph, and its f-score is around 50. Since we did not run experiments on protein-pair-wise cross validation, our system can- not be compared directly to the results reported by Erkan et al. (2007) and Katrenko and Adriaans (2006), while Sætre et al. (2007) presented better re- sults than theirs in the same evaluation criterion.