Exploring the Association of Movie Trailer Performance on YouTube and Box Office Success using Neural Net, Python, and R

This was submitted as a project for my Big Data Analytics class in my MS Business Analytics program. The original title is “Exploring the Association of Movie Trailer Performance on YouTube and Box Office Success”. My other teammates are Yi Cai, Michael Friscia, and Zheyu Tian. This project was done for educational purposes only. Click the photos to enlarge. Check out the GitHub page for the files and data set. Due to policies of thenumbers.com regarding their data, that particular data set won’t be uploaded.

UPDATE: If you scroll below, you will see that the final accuracy was 82.55%. Using genetic algorithms and a Sklearn implementation, the accuracy was improved to 98.66% (with a final generation average accuracy of 92.28%). Check out the code in this GitHub repo.

Problem Statement

The purpose of this study is to determine if there is a correlation between the performance of trailers on YouTube and Hollywood movie sales.

Project Significance

  • By evaluating important predictors from YouTube viewers, studios and agencies can create and publish movie trailers on YouTube more efficiently, thus:
    • driving box office ticket sales domestically and globally
    • generating more revenue
  • Trailer performance can be focused on and improved if it shows that there is a correlation to boxoffice/post-show sales

Data Collection

  • Data was collected from YouTube, using its proprietary API, and from thenumbers.com
    • Youtube – trailer performance and comments
    • thenumbers.com – Movie Box Office data
  • 32.4GB (when comments are expanded into 1 line per comment)
  • 1,713 movies
  • 5,244 trailers
  • 2,979,511 comments

Youtube Data

Variable Selection

  • The ROI variable had to be created.

Variables selected

Hypothesis and Rationale

  • There is a positive correlation between Youtube movie trailer performance indicators  and Box office performance/Video Sales.
    • Rationale: “Likes” = Sales
  • There is a positive correlation between Movie trailer comment sentiments and Box office/Video Sales  performance.
    • Rationale: If trailers are viewed in a positive manner, then people will be more likely to watch the movie.

Conceptual Model

  • After data extraction using Python, data was transformed using Python. Output files were CSV and TXT files.
  • Three sentiment models were implemented in the project: polarity-based sentiment models by using Bing Liu’s and Harvard IV-4 dictionaries, and Naive Bayes Classifier: NLTK Sentiment model.
    • To process part of the sentiment analysis, Apache Spark was used.
  • The sentiment scores were also used to help identify the ROI of each movie using a neural network model.

Project Conceptual Model

Results and Discussion

Variable Correlation Test

The graph, which was generated by R, shows the correlations between the independent variables and dependent variables.

There are three main main conclusions based on the graph:  

 1.The graph demonstrated a positive correlation among count Views, Count Comments, and Likes/Dislikes.

2. The graph was also used to test the hypotheses regarding the movie trailer features and movie performance which assumed that the movie trailer comment counts/ Movie Trailer Likes and Movie Box Office are positively correlated.

 3. Unfortunately, three sentiment models have little correlation with the Box Office Data (eg. ROI), which means that the initial hypothesis wasn’t proved. Two feature-based sentiment models have negative correlations with: Count Views, Count Comments, Likes/Dislikes.

Time Series Analysis

  • It was interesting to see that for 2008, even though with the financial crisis, overall ROI turned out to be good.
  • Another interesting finding is that ROI continuously decreased after 2008.

Sentiment Analysis

Two models were implemented for sentiment analysis.

  • a polarity-based model using Bing Liu’s and a Harvard dictionary, which nets the counts of positive and negative words that can be found in each comment, and
  • the NLTK Sentiment Analyzer using the Vader dictionary, which is a rule-based approach
  • Scores were scaled and centered to zero to maintain positive scores > 0 and negative scores < 0. The scale is [-1,1].

  • Comparing the performance of the three models, the Polarity-based models gravitated towards negative sentiment, which could be explained by the internal structure of the dictionaries used; meaning, if there were more negative than positive words, most likely there will be a higher chance of a higher negative-word count.
  • For the NLTK Sentiment Analyzer, results showed more positive sentiment towards the comments.

Sentiment Analysis – Movie Studios

  • Based on the Harvard sentiment dictionary, Paramount Vantage has the lowest average sentiment score whereas Weinstein Company has the highest.
  • The Vader sentiment sentiment dictionary determined that Apparition has the highest average sentiment score while Focus/Gramercy has the lowest sentiment average score.
  • Bing Liu sentiment dictionary predicted that Freestyle Releasing and Apparition have the lowest and highest average sentiment score, respectively.

Sentiment Analysis – Genre

  • When evaluating the Bing Liu and Harvard dictionaries, Romantic Comedies and Documentaries have the highest and lowest average sentiment score respectively.
  • Interestingly, for the NLTK Analyzer, the Concerts and Performances genre has the lowest average sentiment score, while Romantic Comedy has the highest score.

Clustering (to follow)

Predicting Box Office ROI Performance using Neural Net

  • ROI performance was classified using four bins:
    • Poor (less than the 25% quantile)
    • Passing (between 25% and 50% quantile)
    • Ok (between the 50% and 75% quantile)
    • Great (above the  75% quantile)
  • Neural Net implemented using R
  • ROI Performance ~ countsComments + countsViews + Ratio_of_Likes_and_Dislikes + ProdBudget + genre + MPAArating + MovieStudio + BingLiuSentiment + HarvardSentiment + VadeSentiment
  • Model Accuracy = 82.55%

    Neural Net Model Results

Conclusion

  • Due to the success of the neural network model, companies now have the ability to accurately predict the ROI of their movies, specifically with the use of the number of YouTube comments, ratio of likes and dislikes, and their sentiment scores from the three models.
  • With the hypotheses predicted for the research, there is a higher probability of Box Office success which would then in return generate a higher ROI for movie studios and production companies. 
  • Although the sentiment results are different among the three dictionaries, this implicates that some dictionaries used in the models view more neutral words as negative or positive.
    • The best alternative methods  to predict the sentiment of YouTube comments in movies are to use domain-specific dictionaries and the application of  machine learning classifiers paired with a sample comment-sentiment data set. 

Scope and Limitations

  • There are many popular websites and applications that can be used to comment on trailers or movies, such as Rotten Tomatoes, Facebook, Twitter and so on. However, in this case, Youtube is the only trailer source used.

  • Trailers are not the only factors that impact box office and video sales. Other factors such as advertisements,the actors, and  the competition of other movies being released at the same time can have an effect on the movie’s box office sales. However, these factors are not included in this study. Further studies could be conducted with those variables included.

Reference

Predicting success An Indiegogo prediction study

Predicting Success using SPSS: An Indiegogo Prediction Study

This was submitted as a project for my data mining class in my MS Business Analytics program. My team included classmates: Anh Duong and Luoqi Deng. This project was done for educational purposes only. Click the photos to enlarge.

Abstract & Key Learnings

  • The hypothesis that crowdfunding campaigns from Colorado have higher success rates than campaigns that are located in other places was proven.
  • The category, the number of comments, the number of funders, and the fund goal of the campaign contribute significantly to its success.
  • If a campaign creator from Colorado promotes these factors, they will be able to increase their chances of success significantly.

Project Rationale

  • Indiegogo, the largest global crowdfunding and fundraising site online, has funded over one hundred seventy-five thousand (175K) campaigns, with an estimated valuation of eight hundred million (800M) dollars, in the past seven years. Over two million five hundred thousand (2.5M) funders from two hundred sixty-six (266) countries have contributed to the success of these campaigns.
  • Based on a recent tally by Krowdster.co, a crowdfunding marketing and PR solution provider, stated that nine of ten (90%) Indiegogo campaigns fail to reach their goal; which is significantly higher than the 66.6% failure rate of Kickstarter, one of Indiegogo’s biggest competitors.

Problem Statement

This study aimed to determine whether campaign success can be predicted by certain attributes of its profile, campaign activity, and Indiegogo funder engagement.

Data Description/Preprocessing

The data used came from a public data set from BigML.com.

  • 15K rows
  • Global data
  • No time factor
  • Contains finished and unfinished campaigns
Data set variables

Original data set variables

Cleaning

In order to make the data usable, the data set had to undergo cleaning. All the blank cells were turned into zeroes. Variables that had inconsistent units (hours, days, minutes) had to be converted into consistent values. To make analysis easier and more reflective of performance, two calculated fields were created:

  • Attainment Rate = Raised/Goal
  • Attained = if(Attainment Rate >= 100), then 1, else, 0

Variable Selection

State and country were disregarded because the scope of the study was focused on Colorado-only campaigns.

Model Building

For this study, the Decision Tree, Neural Network, Bayesian, and Clustering models were used to determine the most important predictors to a campaign’s success. Multiple models were chosen so that the outcomes could be compared and a more logical conclusion could be confidently drawn. The association model wasn’t used because, although it would be interesting to determine the confidence and correlation of the occurrence of the different variables, it would contribute little to the goal of this study.

Assumptions and Limitations

  • The study is inadequate due to the researchers’ inability to collected all the variables, such as the important dates of the campaigns. The result of our predictive model is solely based on the pre-given attributes
  • The observations are independent of each other
  • Campaigns happened within the same year
  • A successful campaign is when the amount of capital raised by the crowdfunders exceeds the goal set by campaign owners. Therefore, in our dataset, we create two new calculated fields called Attainment Rate and Attained. Attainment Rate is the Raised amount over the Goal amount. If the Attainment Rate is above 100, the campaign is attained, which is indicated as 1 in the Attained field, and 0 vice versa. The two fields will be used as target for our model.

Hypothesis

According to Krowdster.co, only 10% of the Indiegogo campaign meet their goals. In our sample data set of Colorado, the actual result we found is 30%.

  • The p-value (Sig.) of the one sample T-test is less than 0.001; thus, we reject the null hypothesis that the sample mean is equal to the hypothesized population.
  • This means that crowdfunding campaign in Colorado does have higher chance of success than the average campaign globally.

screen-shot-2017-01-19-at-6-27-20-pm-copy screen-shot-2017-01-19-at-6-27-29-pm-copy

Descriptive/Regression Analysis

Correlation Table

Correlation Table

  • Consideration: potential correlation between continuous variables.
    • A strong correlation between two variables will make explaining what’s going on harder. The Correlation table shows the correlation table between the variables. The two pairs Updates & Gallery (0.589) and Comments & Funders (0.681) show strong positive correlation.
    • However, since we already have very few variables, we decide not to conduct the Principle Component Analysis nor Factor Analysis to drop variables.
  • It is helpful to understand whether a high number of comments, updates, gallery or funders is correlated to the attainment rate. If the correlation is indeed strong, campaign owner could promote the variable to increase the attainment rate.
    • Attainment Rate has a correlation of 0.397 and 0.44 with Comments and Funders, respectively, which is a good indicator that a high number of comments and funders could lead to a higher attainment rate
Statistics Table

Statistics Table

  • The Statistics table (Figure 2) shows the descriptive statistics about the continuous variables and the outcome Attainment Rate.
    • All the continuous variables are heavily skewed to the left (mean >> median). The skewness of the dataset makes it difficult to apply hypothesis testing since most of tests are based on normal distribution assumption.
    • Regarding the Attainment Rate, the fact that it has the mean and median below 100 indicates the high failure rate of the sample.
    • The standard deviation of 260 and the maximum value of 14281 suggest that there are many outliers in the dataset.
Distribution of Campaigns based on Category

Distribution of Campaigns based on Category

  • The Category table (Figure 3) shows the distribution table of Category, the only relevant discreet variable of the sample data set since the geographical dimension has been eliminated.

Predictive Analysis

If the results from the classification models are similar to each other, a conclusion about the predictor of success of a campaign can be confidently drawn.

Decision Tree

Preparations:

  • Partition: 50-50 Training and Testing data
  • Irrelevant attributes, such as text and urls, were casted as typeless to prevent noise and misrepresentation in the model itself.
Model Summary

Model Summary

Accuracy test between training and testing data

Accuracy test between training and testing data

Neural Network

Preparations:

  • Partition: 70-30 Training and Testing data
Model Summary

Model Summary

Predictor Importance

Predictor Importance

Accuracy test between training and testing data

Accuracy test between training and testing data

Histograms for Goal and Comments

Histograms for Goal and Comments

Bayesian Network

Preparations:

  • Partition: 50-50 Training and Testing data
Model Summary

Model Summary

Coincidence Matrix

Coincidence Matrix

screen-shot-2017-01-19-at-6-29-09-pm-copy screen-shot-2017-01-19-at-6-29-16-pm-copy screen-shot-2017-01-19-at-6-29-22-pm-copy

Clustering

Model Summary

Model Summary

Distribution of Data

Distribution of Data

Summary

Overall, the performance of our analysis stays at the fair/ moderate acceptance level. The results from our classification models share some consistency with each other, meaning that we can trust their outcomes. The following table shows the summary of our analysis:

Summary of Predictive Analysis Results

Summary of Predictive Analysis Results

Domain Knowledge

Domain knowledge was gathered to be able to ground the study to what happens in the business context.

  • With the advent of social media, it has been more evident that crowdfunding is not bounded by geographical constraints.
  • It was also observed that funders exhibit herding mentality. This means that as a campaign accumulates capital, more and more individual funders are motivated to make the campaign a success.
  • Immediate markets, such as friends and family, play an important role to the early success of a crowdfunding campaign. This market has been known to spike the fund during the first few days of the campaigns.
  • Crowdfunding is a platform of incentives to both the creators and funders.
  • Funders lose motivation in supporting campaigns where creator incompetence and inexperience is evident.

Moving Forward

  • A bigger dataset in terms of dimensions and quantity will definitely impact the overall quality of the classification results.