Pages

Jul 8, 2013

Text Mining: Named Entity Detection

An interesting task of text mining is detecting entities in the text. Such entities could be a person, a company, a product, or a location. Since an entity is associated with a special name, it is also called Named Entity. For example, the following text contains 3 named entities:
       Apple has hired Paul Deneve as vice president, reporting to CEO Tim Cook.
The first term “Apple” indicates a company, and the second and third are persons.

Named entity detection (NER) is an important component in  social media analysis. It helps us to understand user sentiment on specific products. NER is also important for product search for E-commerce companies. It helps us to understand user search query related to certain products.

To map each name to an entity, one solution is using a dictionary of special names. Unfortunately, this approach has two serious problems. The first problem is that our dictionary is not complete. New companies are created and new products are sold every day. It is hard to keep track all the new names. The second problem is the ambiguity of associating a name to an entity. The following example illustrates this:
As Washington politicians argue about the budget reform, it is a good time to look back at George Washington’s time.
In this text, the first mention of “Washington” refers to a city, while the second mention refers to a person. The distinction of these two entities comes from their context.

To resolve ambiguity in entity mapping,  we can create certain rules to utilize the context. For example, we can create the following rules:
  1. When ‘Washington’ is followed by ‘politician’, then it refers to a city.
  2. When ‘Washington’ is preceded by ‘in’, then it refers to a city.
  3. When ‘Washington’ is preceded by ‘George’, then it refers to a person.
But such rules could be too many. For example, each of the following phrases would generate a different rule: “Washington mentality”, “Washington atmosphere”, “Washington debate” as well as “Washington biography” and “Washington example”. The richness of natural language makes the number of rules exploding and still susceptible to exceptions.

Instead of manually creating rules, we can apply machine learning. The advantage of machine learning is that it creates patterns automatically from examples. No rule needs to be manually written by humans. The machine learning algorithm takes a set of training examples, and chunk out its own model (that is comparable to rules). If we get new training data, we can re-train the machine learning algorithm and generate a new model quickly.

How does the machine learning approach work? I will discuss it in the next post. 

55 comments:

  1. Thank you for this post. Very interesting.

    ReplyDelete
  2. I think it is amazing that the search engine will try to interpret and answer my question. It sounds like program that was playing jeopardy. What was it called? Watson?
    As the technology grows it will reduce the really odd search results that pop up. To see more info please visit http://essayhogwarts.com/dissertation-proposal/.

    ReplyDelete
  3. Interesting post. I will wait for your next post on machine learning.

    ReplyDelete
  4. After I originally left a comment I seem to have clicked on the -Notify me when new comments are added- checkbox aand now whenever a comment is added I recieve 4 emaols with the same comment.
    Perhaps there iis a means you are able to remove me from that service?
    Many thanks!

    Also visit my homepage ... bristle hair btush - -

    ReplyDelete
  5. Sometimes single mothers and couples may encounter such situations wherein childhood
    becomes impossible for them. Try to think of
    what areas in your neighborhood could use a little bit
    of sprucing up - As you try to decide what services are needed, you
    can actively advertise and promote it to try to get others involved.

    Closely linked to this strategy is the technology strategy.


    Feel free to surf to my blog post; Adoption Network
    Law Center webblog - -

    ReplyDelete
  6. This article has covered the topic quite well. Very informational and interesting. Thanks for sharing this knowledge with us.

    new year, new year images, new year wallpaper, new year quotes, new year wishes, new year sms, new year greetings, whatsapp status

    ReplyDelete
  7. We are offering website services & web design services for your business website also offering E-commerce website....

    ReplyDelete
  8. Hello. Now You can share mother's day quotes,images & messages.
    Mother's day quotes
    Mothers day messages
    Happy mother's day messages

    ReplyDelete
  9. In the particular article You might understand how for getting pay per click campaign started building net applications

    ReplyDelete
  10. The war between humans, orcs and elves continues earn to die . Lead your race through a series of epic battles, using your crossbow to fend off foes and sending out units to destroy castleshappy wheels . Researching and upgrading wisely will be crucial to your success! There are 5 ages total and each one will bring you new units to train to fight in the war for you cause.
    earn to die 2
    Whatever you do, don’t neglect your home base because you cannot repair it and once it is destroyed, you lose! Age of War is the first game of the series and really sets the tone for the Age of War games . Also try out the Age of Defense series as it is pretty similar.
    In this game, you start at the cavern men’s age, then evolvetank trouble ! There is a total of 5 ages, each with its units and turrets. Take control of 16 different units and 15 different turrets to defend your base and destroy your enemy.
    The goal of the game also differs depending on the level. In most levels the goal is to reach a finish line or to collect tokens. Many levels feature alternate or nonexistent goals for the player. The game controls are shown just under gold miner. Movement mechanisms primarily include acceleration and tilting controls. cubefield
    It consists of a total of 17 levels and the challenge you face in each level increases as you go up. unfair mario The game basically has a red ball that has to be moved across the various obstacles in its path to the goal. slitherio

    ReplyDelete

  11. Great! Thanks for sharing the information. That is very helpful for increasing my knowledge in this fiel
    Red Ball | | duck life | Slitherio
    Red Ball 2 | Red Ball 3 | Red Ball 4

    ReplyDelete
  12. good Preserve in head if you carry out the guidelines in Click Here you with interesting content material. nice.

    ReplyDelete
  13. good all the add-ons you’d prefer, hence by using a build will sportsdevil not working It is usually compatible with multiple operating devices and great.

    ReplyDelete
  14. good creative way we need to present. In simple way official website link your google android right now !! We give you the formal VivaVideo Download Free nice.

    ReplyDelete

  15. Wonderful blog! I found it while searching on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Many thanks.
    2048 game | five nights at freddy's 4 | five nights at freddy's 3 | fireboy and watergirl | fireboy and watergirl 4||red ball | age of war

    ReplyDelete
  16. Can one jailbreak iOS 10 / iOS 10.0.2 / 10.0.1? If not, what is latest on iOS 10 / 10.0.2 jailbreak status for iPhone, iPad and iPod touch devices? you can get answer of this Question as now you can get iOS 10 jailbreak.

    ReplyDelete
  17. good you can hook up your cell to your Computer either Home windows or Xender PC Download You can hook up your clever mobile phone to Computer applying nice.

    ReplyDelete