Pages

Jan 20, 2014

Statistics and Data Mining

Probability theory is the core of data mining. In the early day of data mining, people count the frequent pattern of those who buy diaper also buy beer. This is essentially deriving a conditional probability buying beer given buying diaper. In mathematical term, it is P(buy beer | buy diaper).

Here is example: Suppose 5% of the shoppers buy diaper, and only 2% of shoppers buy both diaper and beer, what the likelihood of some buys diaper after buying beer?  The answer is a conditional probability.
The so-called “association rule” in data mining can be much clearly understood through these statistics terms. 

In machine learning, the core concept is predicting whether a data sample belong to a certain population (class). For example, we want to predict whether a person is a potential buyer, or a credit card transaction is fraudulent. Such prediction is always associated with a confidence score between 0 and 1. When the confidence score is 1, we are 100% sure about the prediction. Generating such confidence score requires statistical inference.

Statistics is essential when evaluating a data mining model. We use control group or conduct A/B test. For example, if we test with 1000 people in treatment group (Group A) vs. 1000 people in control group (Group B) and find Group A has better performance, is this result conclusive? In other words, is it statistically significant? That has to be answered by statistic knowledge.

In data collection phrase, statistics helps us to decide the size of training data (sample), and whether it is representative. Not to mention, some current popular machine learning methods such as logistic regression were originally developed in statistics community.

As mathematics provides foundation for physics, statistics has now become a foundation for machine learning. Its importance will become more prominent overtime in data mining.

30 comments:

  1. very informative. Big data analysis can be also applied into our SEO job. By analyzing the key words that users used to search the site, we can know what they are interested in and how to keep them interesting. BOC Sciences achieved a great progress with this method.

    ReplyDelete
  2. very informative. Big data analysis can be also applied into our SEO job. By analyzing the key words that users used to search the site, we can know what they are interested in and how to keep them interesting. BOC Sciences achieved a great progress with this method.

    ReplyDelete
  3. Do not worry jamu agar tahan lama di ranjang if your penis an erection or not. cara mengatasi ejakulasi dini pada pria The more you are anxious and cara membuat penis kuat dan tahan lama think about it would make more cara pemesanan spirulina difficult the erection occurs or even erection cara pemesanan cordyceps plus capsule that had been hard even suddenly flaccid again. obat kuat tradisional Do not be nervous and remained relaxed and confident with your erection. All you have to understand that, a ptia not always get an erection in any occasion.

    ReplyDelete
  4. Thanks for sharing this interesting and useful things with all of us. this is what I am looking for.
    CP

    ReplyDelete
  5. I found a lot of entertaining products in your weblog, especially the discussion. free itune codes

    ReplyDelete

  6. this is great atikel ,, i like it,, thank for u sharing info for all people Situs Agen Judi Piala Eropa 2016 Terpercaya

    ReplyDelete
  7. Want to watch the most important match of Euro Cup 2016, the big final live from your home,just check out Euro Cup Final Live Streaming

    ReplyDelete
  8. If you love cats, you must check out my blog National Cat Day
    which is solely dedicated to cats. Learn everything you want to know about cats and stay updated. Enjoy with your pets on Cat Day and make her feel special.

    ReplyDelete
  9. Want to know about Veterans Day,which is a day to celebrate the great officers of the US army,just checkout my blog Veterans Day 2016 and stay updated.

    ReplyDelete
  10. Backlinks Affordable link building service and high pr backlinks service with express delivery!
    dofollow Backlinks Affordable link building service and high pr backlinks service with express delivery!
    High PR Backlinks Affordable link building service and high pr backlinks service with express delivery!

    ReplyDelete
  11. London Escorts For those of you seeking a VIP escort who can provide something a little bit out of the ordinary, you have come to the right place. 000 London Escorts is an Elite Central London Escorts Agency providing high class escort courtesans in Central London.

    ReplyDelete
  12. German Sex Videos The best daily updated Amateurs Porn Videos,sexiest girls on the net.

    ReplyDelete
  13. tight pussy Tight Pussy Video's, Tight Pussy Porn, Tight Pussy Hentai Cartoon Porn, Free Tight Pussy Cam Shows, Latino Tight Pussy, Tight Pussy Cartoon Video's, And Everything Porn Where There Is A Cute Tight Pussy Involved :)

    ReplyDelete
  14. Valentines Week

    Valentine Week Days List 2017

    Valentine Week 2017
    ---------------------------------------------------------------------------------------------


    Happy Promise day Wishes

    Happy Promise day Wishes for boyfriend

    Happy Promise day Wishes for husband

    --------------------------------------------------------------------------------------------
    Happy Hug day Wishes

    Happy Hug day Wishes for boyfriend


    Happy Hug day Wishes for husband

    ---------------------------------------------------------------------------------------------

    Happy Valentine day Wishes



    Happy Valentine day Wishes for boyfriend



    Happy Valentine day Wishes for husband

    ReplyDelete
  15. gusto Free Porn Movies at Gusto Porn. A huge amount of porn industry content just a click away. Hi-def video quality, images and pictures from sex lovers around the globe. All types of adult categories, there is something for everyone that likes xxx or loves porno. Enjoy!

    ReplyDelete
  16. indmms.com Watch free Desi Xxx leaked porn full videos online on your desktop or mobile phone. Free full length XXX movies. Share your own xxx and leaked porn video on www.txxx.xxx ...

    ReplyDelete
  17. SnapFuck Women you know are sharing their naked pictures at www.snapfuck.co . And even more hot girls are looking for casual sex and discreet affairs. Registration is 100% Free.

    ReplyDelete
  18. free cams Free sex Chat. Watch Best Porn Cams and enjoy free live sex shows with sexy adult cams girls. ​Ffree sex webcams, live adult cams, free sex webcams, adult webcams, free live porn.

    ReplyDelete
  19. how to cure acne fast What exactly helps for acne & pimples? Discover how to get rid of acne and pimples quickly and permanently. For the time “after” acne, pimples certain observations made suggest to stay away from certain remedies or cosmetic recommendation, to avoid collateral harm such as scars etc. ​Everyone’s skin varies in structure and composition so everyone’s reaction to any treatment methods are likely to be completely different. It depends on the hardness of the outbreak as well as other mitigating circumstances. ​Therefore, how to treat acne and pimples? ​Our information and facts will assist you to fight them!

    ReplyDelete
  20. VR Porn Galaxy See the hottest pornstars like they were in the same room as you in Virtual Reality.

    ReplyDelete
  21. NYC Escorts New York Escorts - The Best NYC Escorts and Independent New York City Escort available in Manhattan. Indulge with the best Independent VIP Ebony Escort in NYC. The ultimate Elite New York Escort Service.

    ReplyDelete
  22. HQ Japanese blowjobs Tsunamiblowjob.club offers you unlimited Japanese blowjobs. Uncencored and only in HQ 1080p for highest pleasures. You find also great Japanese lesbian inside. We have now MAXI PROMO for only 1 USD / month. Come and check it out.

    ReplyDelete
  23. big black cock Watch free Big Black Cock porn videos on supportyourking. Select from the best full length Big Black Cock XXX movies to play. supportyourking.com updates hourly!

    ReplyDelete