9 Cool NLTK Functions You Did Not Know Exist


NLTK is a great library for natural language processing with support for all the commonly used functions like stemming, lemmatization, stopwords, POS, etc. But this is just the tip of the iceberg, in fact, there are many useful functions that you probably did not know about in NLTK. In this article, we will go through such NLTK functions like Concordance, Similar, Generate, Dispersion Plot, etc.

So let us get started.

1. Concordance

NLTK concordance is a useful function to search every occurrence of a particular word in the context and also display the context around the search keyword. The function only prints the output, does not allows the user to store it for further preprocessing.

For example, in the below example of concordance function, we try to find all the occurrences of the word “magic”. You can notice that it is surrounded by the text between which it occurs. This is quite helpful not only to search the occurrence of a word or phrase but also to have a quick view of its context.

In [1]:
with open("concordance.txt") as f:
    text = f.read()
In [2]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :
Displaying 12 of 12 matches:
erning body known as the Ministry of Magic and subjugate all wizards and Muggle
ties are invited to attend exclusive magic schools that teach the necessary ski
orating objects and wildlife such as magic wands , magic plants , potions , spe
s and wildlife such as magic wands , magic plants , potions , spells , flying b
f Witchcraft and Wizardry , a famous magic school in Scotland that educates you
es which are well above the level of magic generally executed by people his age
 recent activities , the Ministry of Magic and many others in the magical world
ys to defend themselves against dark magic . [ 18 ] Hermione and Ron form `` Du
t a dark corridor in the Ministry of Magic , followed by a burning desire to le
ortured and races to the Ministry of Magic . He and his friends face off agains
ed Death Eaters ) at the Ministry of Magic . Although the timely arrival of mem
nd gained control of the Ministry of Magic . Harry , Ron and Hermione drop out

2. Generate

Generate function generates some random text in the various styles. To do this, we type the name of the text followed by the term generate. For the first time, this command will take a longer time to run, as it gathers statistics about word sequences. We will get different output text each time we run the generate function.

In [3]:
import matplotlib.pyplot as plt
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :
in 2012 , a sport in the Ministry of Magic . , large , happy , but
because of the battle , including fantasy , drama , coming of age ,
and finds an old potions textbook filled with many annotations and
recommendations signed by a mysterious writer titled ; `` the Half-
Blood Prince '' . a new Defence Against the Dark Arts . Harry and
Ginny 's belongings . the books have sold more than one school year .
The first book concludes with Harry 's history , and Voldemort to be
one of which was the diary and

3. Count

Count function returns the total count of a given word in the text. In the below example, we can see that count of the word “Harry” is 85 in the sample text.

In [4]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :

4. Collocation_list

A collocation is a sequence of words that occur together unusually often. Collocation_list function returns a list of collocation words with the default size of 2.

In the below example, we can see that “Harry Potter” appears in the collocation list, which is easy to see why. Similarly, “Arts Teacher” is also a collocation example below.

In [5]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :
[('Harry', 'Potter'),
 ('Dark', 'Arts'),
 ('Wizarding', 'World'),
 ('wizarding', 'world'),
 ('Arts', 'teacher'),
 ('Lord', 'Voldemort'),
 ('Half-Blood', 'Prince'),
 ('new', 'Defence'),
 ('Death', 'Eaters'),
 ('Sirius', 'Black'),
 ('non-magical', 'people'),
 ('Deathly', 'Hallows'),
 ('Peter', 'Pettigrew'),
 ('device', 'called'),
 ('million', 'copies'),
 ('Hogwarts', 'School'),
 ('Marvolo', 'Riddle'),
 ('Tom', 'Marvolo'),
 ('United', 'States'),
 ('exists', 'parallel')]

5. Dispersion Plot

A dispersion plot in NLTK is used to visualize the position and number of occurrences of the words in a text corpus.

In the below example of dispersion plot, each stripe represents an instance of a word, and each row represents the entire text. Here we have visualized the occurrence of the words “Harry”, “Ron”, “Death”, and “magic”.

In [6]:
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :
dispersion plot

6. Similar

The Similar function in NLTK takes an input word and returns other words that appear in a similar range of contexts in the text.

For example, whenever the words Hogwarts, ron, and witchcraft occur, they will be surrounded by the same context that is “of Hogwarts and”, “of ron and” and “of witchcraft and”.

In [7]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :
hogwarts ron witchcraft age death adolescence keys terror
parentage fire

7. Common_context

The function common_contexts allows us to examine just the contexts that are shared by two or more words. For example, the words like Hogwarts, magic, and ron are usually surrounded by context “of_and”.

In [8]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
text1.common_contexts(['hogwarts', 'magic','ron'])
[Out] :

8. Index

Index function returns the first index of the word in the text. Remember that the first token starts from the 0th index.

In our below example, the word “Iron” occurs for the first time at the 37th index.

In [9]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)
[Out] :

9. Vocab

Vocab function in NLTK returns the total vocab of the text that is the count of all the unique words present in the text.

In the example below, the Vocab function returns the count of 1129 our text corpus.

In [10]:
import nltk
tokens = nltk.word_tokenize(text)
text1 = nltk.Text(tokens)


Reference – NLTK Documentation

  • Afham Fardeen

    This is Afham Fardeen, who loves the field of Machine Learning and enjoys reading and writing on it. The idea of enabling a machine to learn strikes me.

    View all posts

Follow Us

Leave a Reply

Your email address will not be published. Required fields are marked *