<div><img alt="" src="https://sai977.github.io/images/quora.png" title="Image: https://sai977.github.io/images/quora.png"></div><h2>Quora question pair similarity</h2><h3>Introduction</h3><span>Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world.<br>Over 100 million people visit Quora every month, so it's no surprise that many people ask similarly worded questions. Multiple questions with the same intent can cause seekers to spend more time finding the best answer to their question and make writers feel they need to answer multiple versions of the same question. Quora values canonical questions because they provide a better experience to active seekers and writers, and offer more value to both of these groups in the long term.</span><h3>Problem Statement</h3><ul><li>Identify which questions asked on Quora are duplicates of questions that have already been asked.</li><li>This could be useful to instantly provide answers to questions that have already been answered.</li><li>We are tasked with predicting whether a pair of questions are duplicates or not.</li></ul><h6>Objectives<ol><li>The cost of misclassification can be very high.</li><li>You would want a probability of a pair of questions to be duplicated so that you can choose any threshold of choice.</li><li>No strict latency concerns.</li><li>Interpretability is partially important.</li></ol></h6><h3>Type of Machine Learning Problem</h3>It is a binary classification problem, for a given pair of questions we need to predict if they are duplicate or not.<h6>Performance metric<ul><li>log-loss</li><li>Binary Confusion Matrix</li></ul></h6><h6>Objective and Constraints<ol><li>Minimize RMSE.</li><li>Try to provide some interpretability.</li></ol></h6><h6>Preprocessing<ul><li>Converting / Merging whole data to required format: u_i, m_j, r_ij</li><li>Checking for NaN values, Removing Duplicates</li><li>Basic Statistics (#Ratings, #Users, and #Movies)</li><li>Splitting data into Train and Test(80:20)</li></ul></h6><h6>Exploratory Data Analysis<ul><li>is_duplicate</li><div><img width="450" alt="" src="https://sai977.github.io/images/is_duplicate.png" title="Image: https://sai977.github.io/images/is_duplicate.png"><li>Plot representing unique and repeated questions</li><div><img width="450" alt="" src="https://sai977.github.io/images/Plot%20representing%20unique%20and%20repeated%20questions.png" title="Image: https://sai977.github.io/images/Plot%20representing%20unique%20and%20repeated%20questions.png"><li>Log-Histogram of question appearance counts</li><div><img width="450" alt="" src="https://sai977.github.io/images/Log-Histogram%20of%20question%20appearance%20counts.png"><li>Word Cloud for Duplicate Question pairs</li><div><img width="450" alt="" src="https://sai977.github.io/images/Word%20Cloud%20for%20Duplicate%20Question%20pairs.png"><li>Word Cloud for non-Duplicate Question pairs:</li><div><img width="450" alt="" src="https://sai977.github.io/images/Word%20Cloud%20for%20non-Duplicate%20Question%20pairs.png" title="Image: https://sai977.github.io/images/Word%20Cloud%20for%20non-Duplicate%20Question%20pairs.png"></div></div></div></div></div></ul></h6><h6>Featurizing data<ol><li>freq_qid1 = Frequency of qid1's</li><li>freq_qid2 = Frequency of qid2's</li><li>q1len = Length of q1</li><li>q2len = Length of q2</li><li>q1_n_words = Number of words in Question 1</li><li>q2_n_words = Number of words in Question 2</li><li>word_Common = (Number of common unique words in Question 1 and Question 2)</li><li>word_Total =(Total num of words in Question 1 + Total num of words in Question 2)</li><li>word_share = (word_common)/(word_Total)</li><li>freq_q1+freq_q2 = sum total of the frequency of qid1 and qid2</li><li>freq_q1-freq_q2 = absolute difference of frequency of qid1 and qid2</li><li>Feature: word_share</li><li>The distributions for normalized word_share have some overlap on the far right-hand side, i.e., there are quite a lot of questions with high word similarity</li><li>The average word share and Common no. of words of qid1 and qid2 is more when they are duplicate(Similar)</li><li>Feature: word_Common</li><li>The distributions of the word_Common feature in similar and non-similar questions are highly overlapping</li></ol></h6><h6>Applying Machine Learning models</h6><h6><span><span>S.NO.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; MODEL&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Tokenizer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Train Log loss&nbsp; &nbsp; &nbsp; Test Log loss</span></span></h6><h6><span><span>1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Logistic Regression&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TFIDF&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.44&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.41</span></span></h6><h6><span><span>2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Linear SVM&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TFIDF&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.45&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.45</span></span></h6><h6><span><span>3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;xgboost&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; TFIDF&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; 0.45&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.45</span></span></h6><h6><span><span>4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Linear SVM&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; TFIDF weighted word2Vec&nbsp; &nbsp; &nbsp; 0.51&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.52</span></span></h6><h6><span><span>5&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Linear SVM&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TFIDF weighted word2Vec&nbsp; &nbsp; &nbsp; &nbsp;0.47&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.48</span></span></h6><h6><span><span>6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; xgboost&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TFIDF weighted word2Vec&nbsp; &nbsp; &nbsp; n/a&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<span>0.35</span></span></span></h6>

Python Developers

Algorithm Experts

Quora question pair similarity

Useful links

Company

Information

Content Writers

Freelancers

Freelance Jobs

Online Jobs

Designers

Developers

Translations