Dec 7, 2012

The basics of Image Mining

In online stores, listings of products normally come with pictures. For people who purchase dresses, the difference in images would make a lot difference. A good product search engine would be able to match search keywords with listing text and images combined.

How do we process images in search engine? Here are the steps:
First, we segment the image into the foreground and background. Below are examples of segmentation done by a computer algorithm, with the top row as original picture and bottom row the segmented images with the background whitened out.
The segmentation algorithm is based on GrabCut, which is essentially creating a rectangular bounding box around the foreground object, and then removing all the pixels outside the bounding box.

After the segmentation is done, we extract the photographic and object feature from an image.
Photographic features include:
  • Aspect Ratio (image height divided by its width)
  • Brightness 
  • Contrast 
  • Colorfulness
Object features include:
  • Color histogram
  • Texture histogram
  • Shape features
An image will be represented as a vector of these features.
In product search engine, the image vector of a listing is combined with text vector to represent the listing.