<div><img alt="" src="https://sai977.github.io/images/taxi.png"></div><h2>Stack Overflow Tag Prediction</h2><h3>Introduction</h3><span>Stack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers.<br>Stack Overflow is something which every programmer use one way or another. Each month, over 50 million developers come to Stack Overflow to learn, share their knowledge, and build their careers. It features questions and answers on a wide range of topics in computer programming. The website serves as a platform for users to ask and answer questions, and, through membership and active participation, to vote questions and answers up or down and edit questions and answers in a fashion similar to a wiki or Digg. As of April 2014 Stack Overflow has over 4,000,000 registered users, and it exceeded 10,000,000 questions in late August 2015. Based on the type of tags assigned to questions, the top eight most discussed topics on the site are: Java, JavaScript, C#, PHP, Android, jQuery, Python and HTML.</span><h3>Problem Statement</h3>Suggest the tags based on the content that was there in the question posted on Stackoverflow<h5>Objectives</h5><ul><li>Predict as many tags as possible with high precision and recall.</li><li>Incorrect tags could impact customer experience on StackOverflow.</li><li>No strict latency constraints.</li></ul><h5>Type of Machine Learning Problem</h5><span>It is a multi-label classification problem<br>Multi-label Classification: Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A question on Stackoverflow might be about any of C, Pointers, FileIO and/or memory-management at the same time or none of these.</span><h5>Performance metric</h5><ul><li>Micro-Averaged F1-Score (Mean F Score) :<span>The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal.<br>The formula for the F1 score is:</span>F1 = 2 * (precision * recall) / (precision + recall)In the multi-class and multi-label case, this is the weighted average of the F1 score of each class.</li><li>'Micro f1 score':Calculate metrics globally by counting the total true positives, false negatives and false positives. This is a better metric when we have class imbalance.</li><li>'Macro f1 score':Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.</li><li>Hamming loss :Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.</li></ul><h5>Exploratory Data Analysis</h5><ul><li>Distribution of number of times tag appeared questions</li><div><img width="950" alt="" src="https://sai977.github.io/images/Distribution%20of%20number%20of%20times%20tag%20appeared%20questions.png"><li>first 10k tags: Distribution of number of times tag appeared questions</li><div><img width="950" alt="" src="https://sai977.github.io/images/first%2010k%20tags.png"><li>first 1k tags: Distribution of number of times tag appeared questions</li><div><img width="950" alt="" src="https://sai977.github.io/images/first%201k%20tags.png"><li>first 100 tags: Distribution of number of times tag appeared questions</li><div><img width="950" alt="" src="https://sai977.github.io/images/first%20100%20tags.png">&nbsp;Observations:<ol><li>There are total 153 tags which are used more than 10000 times.</li><li>14 tags are used more than 100000 times.</li><li>Most frequent tag (i.e. c#) is used 331505 times.</li><li>Since some tags occur much more frequenctly than others, Micro-averaged F1-score is the appropriate metric for this probelm.</li></ol><li>Number of tags in the questions</li><div><img width="950" alt="" src="https://sai977.github.io/images/Number%20of%20tags%20in%20the%20questions.png"><li>Most Frequent Tags</li><div><img width="950" alt="" src="https://sai977.github.io/images/Most%20Frequent%20Tags.png"><li>Frequency of top 20 tags</li><div><img width="950" alt="" src="https://sai977.github.io/images/Frequency%20of%20top%2020%20tags.png" title="Image: https://sai977.github.io/images/Frequency%20of%20top%2020%20tags.png"></div></div></div></div></div></div></div></ul><h5>Pre-processing And Data Preparation</h5><ol><li>Sample 1M data points</li><li>Separate out code-snippets from Body</li><li>Remove Special Characters from Question title and description (not in code)</li><li>Remove stop words (Except 'C')</li><li>Remove HTML Tags</li><li>Convert all the characters into small letters</li><li>Use SnowballStemmer to stem the words</li></ol><br><ol><li>Applying Machine Learning models</li><li>S.NO.&nbsp; MODEL&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Featurization&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Hamming loss&nbsp; &nbsp; macro F1 micro&nbsp; &nbsp; F1-measure</li><li>1&nbsp; &nbsp; &nbsp; Linear SVM&nbsp; &nbsp; &nbsp; &nbsp;Count vectorizer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0028&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.1575&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.6199</li><li>2Logistic Regression Count vectorizer&nbsp; &nbsp; &nbsp; &nbsp; 0.0026&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.1787&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.6343</li></ol>

Python Developers

Artificial Intelligence

Stack Overflow Tag Prediction

Useful links

Company

Information

Content Writers

Freelancers

Freelance Jobs

Online Jobs

Designers

Developers

Translations