Data Strategy

July 9, 2007

Writing a statistical spelling corrector

Filed under: Information Retrieval — chucklam @ 2:42 pm

I was working on a presentation today and happened to come across an article by Peter Norvig (Director of Research at Google) on how to build a statistical spelling corrector: http://norvig.com/spell-correct.html.

The basic idea is rather straightforward and Peter does a good job of explaining it, even including a sample program in Python. It would be interesting if Peter had dug deeper into how Google uses query logs to build its language and error models for its search term suggestion feature (i.e., “Did you mean:”), although he steered clear of that. He only described a generic spelling corrector that is trained from generic text documents that one can find off the Web. I suppose that makes sense for a tutorial article since none of us have access to Google’s query log.

Alternatively, or if you just don’t care to build your own corrector, there’s a Yahoo Spelling Suggestion API that you can use. It’s free if you’re making less than 5,000 queries per day.

Advertisements

2 Comments »

  1. Spelling corrector that learns from query logs

    I had previously written a post pointing to a Peter Norvig (Director of Research at Google) article on how to write a statistical spelling corrector. My post noted that Peter’s article didn’t explain how spelling suggestions by search engin…

    Trackback by Data Strategy — September 6, 2007 @ 1:59 am

  2. […] Filed under: Information Retrieval — chucklam @ 1:59 am I had previously written a post pointing to a Peter Norvig (Director of Research at Google) article on how to write a statistical […]

    Pingback by Spelling corrector that learns from query logs « Data Strategy — September 6, 2007 @ 1:59 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: