JTextPro: A Java-based Text Processing Toolkit

Xuan-Hieu Phan (pxhieu at gmail dot com), Graduate School of Information Sciences, Tohoku University

JTextPro: A Java-based Text Processing toolkit that currently includes important processing steps for natural language/text processing as follows:

Sentence boundary detection (using maximum entropy classifier train on WSJ corpus)
Word tokenization (following Penn conventions)
Part-of-speech tagging (using Conditional Random Fields - CRFTagger), and
Phrase chunking (also using Conditional Random Fields - CRFChunker).

This project aims at integrating essential processing steps for developing higher-level applications in natural language processing, text/web data mining, and information extraction. This tool frees developers from time-consuming preprocessing work in order to focus on developing their own features. In the future, we continue to improve the performance of the toolkit and update new features while keeping the source code well-organized and easy to incorporating into other applications.

Download

Related links:

FlexCRFs: Flexible Conditional Random Fields
GibbsLDA++: A C/C++ and Gibbs Sampling-based Implementation of Latent Dirichlet Allocation (LDA)
CRFTagger: CRF English POS Tagger
CRFChunker: CRF English Phrase Chunker
JWebPro: A Java-based Web Processing Toolkit
JVnSegmenter: A Java-based Vietnamese Word Segmentation Tool

Researches using this tool for running experiments should include the following citation:

Xuan-Hieu Phan, "JTextPro: A Java-based Text Processing Toolkit", http://jtextpro.sourceforge.net/, 2006.

We would like to thank professor Tu-Bao Ho for providing us Penn Treebank data for training the POS tagging and chunking models. We would also like to thank Sourceforge.net for hosting this project.

Last updated: March 24, 2007