Ryan Schram Yawahigu ana amwahao Ol rot bilong laip bilong mi (or, Curriculum vitae)

Wiki mining: An idea for an automatic topic identification system for Dokuwiki

I have always wanted a system that would automatically identify important words in a Dokuwiki wiki article and add tags to the page with these words. My latest idea is to create a system of plugins that can do this for me. The system would consist of an action plugin that adds to the indexer web bug and stores the keywords as page metadata and a render plugin that converts stored keywords into tags, and then adds the tags to the page (perhaps with an existing plugin for manually adding tags to articles). I am not sure how to implement this myself and so I’m asking for help from anyone who can help me develop the design for this system and create the plugins to implement it.

The problem: It’s easier to add to a wiki than to consolidate and extend what’s there

I have been using Dokuwiki (DW) to run a small wiki for my class notes, lecture outlines, and other presentations for several years. It’s great to be able to go back to old lecture notes and reuse them in other lectures or expand them into longer guides to a topic. I find it it is very easy to add new articles, but I find it is harder to pull together what I have accumulated. DW provides many different ways to do this, for instance, by adding tags to articles so that articles on the same topic can be grouped together.

The problem with this and similar methods of organizing material in a wiki is that it requires one to first decide on how to organize the material and then manually add information (e.g. tags, wiki links) to the pages. It would be great if the software itself could analyze the information and automatically suggest categories or other relationships among articles. These might need to be edited later, but if a system automatically found the topics of an article were, and then what other articles cover these topics, that would lower the barrier to making edits in the wiki to consolidate the materials. It would be like a prompt to action. Once you finish one article, the software would immediately suggest another edit to make to connect that article to other articles.

To make this happen, one would need to add a plugin to DW to do some kind of text mining on articles to find their topics. DW already has a full text index for its search engine, so a lot of the raw data that one needs for text mining is already stored for each article. The plugin I’d like to create would extend some of the existing indexing functions.

Part 1: Finding keywords by generating a TF*IDF vector for a page during indexing

DW pages include a web bug that triggers several background jobs when a page is loaded, including updating the full text index. When the bug adds new index entries, it triggers the event INDEXER_PAGE_ADD. At this point, I would like a plugin to tokenize the page content and for each word, calculate the TF*IDF value, and then store the term-to-TF*IDF associative array as a metadata entry. Based on my limited understanding of DW, this might go like this:

The terms in this array with the highest TF*IDF values are, we hope, the most important words in the article and can be used as key words. I believe that the existing tokenize functions used by the DW indexer filter stop words, but in fact stop words don’t need to be filtered. These words are common in both the document and in all documents, so they even if the TF is high, they will always have a low TF*IDF value.

I have not really given a thought to efficiency. For instance, when INDEXER_PAGE_ADD is triggered, the page is already going to be tokenized and the term and raw term frequency mappings are stored temporarily. I don’t know how to make use of that, so I foresee doing two tokenizations. This is an area where I could use help on design.

Now when the full text index is updated, each page will also have an array of its terms TF*IDF values which can potentially be used in a variety of ways. One way to use this would be to sort through this term-to-TF*IDF array, find a certain number of the highest weighted terms (say, the top five), and list these as wikilinks to pages where one could gather a list of other pages with inbound links to the same term. I have thought less about this part of the system. I think one could create a render plugin for a new markup element, e.g. ~~KEYWORDS~~ which would return key words with links, much like tag plugins create links for manually entered tags for a page.

Another application of TF*IDF vectors

The new metadata term-to-TFIDF array could be used to find “similar” articles. In text mining, a document can be represented as a vector of its terms’ TF*IDF. Each of the TF*IDF values is a coordinate in a multidimensional space where each term is a dimension that goes from low importance to high importance. Documents with terms that have the same TFIDF weight in each document are located at the same point on that dimension. Hence, the angle between these the document vectors is also a measure of how close they are, that is, how many important words they have in common, or perhaps more precisely how close in importance their common terms are. This can be represented as the cosine of the angle between two document vectors.

If each page has a TF*IDF vector stored in metadata, then another plugin could calculate its cosine similarity with other articles’ vectors, rank them by their similarity, and store all or part of this sorted list of articles. This could then be accessed by another render plugin to display a list of similar articles in the wiki.

If you have any suggestions about how I should pursue this, or people I should ask for help, please let me know.