Answers to the first 40 questions in this list are included in both the full course in product data science as well as the interview material package.
Answers to all the other product questions here are included ONLY in the full course in product DS.
1) How would you improve engagement on FB?
2) We are launching a new driver app with a better UI. The goal is increasing driver earnings by increasing their number of trips. Outline a testing strategy to see if the new app is better than the old one
3) Give an example of a case in which you run an A/B test, test wins with significant p-value, but you still choose to not make the change
4) Do you expect that Uber trips without rider review have been better, worse, or same as trips with reviews?
5) A given category of an e-commerce marketplace, for instance jeans
, is not doing well. How would you estimate if it is a demand or supply problem?
6) What are the drawbacks of using supervised machine learning to predict frauds?
7) If 70% of Facebook users on iOS use Instagram, but only 35% of Facebook users on Android use Instagram, how would you investigate the discrepancy?
8) We made a change to our subscription offering adding new features. We expect this to increase subscription retention. How can we test if the change is successful?
9) Do you think it is better to target ads based on user demographic or behavioral characteristics (past browsing experience)? Why?
10) We ran an A/B test. Test won, so we make the change on the site for all users. But after waiting for some time, we realize that the new version of the site is not performing better than the old one. What could be the reason?
11) You are launching a messaging app. Define 2 metrics that you'd choose to monitor app performance during the first months. Why you chose them?
12) Which feature would you add to WhatsApp? Hypothetically, imagine you have access to all WhatsApp-related data, such as messages and where people click on the app
13) You have to predict conversion rate on Airbnb using user country as one of the input variables. How would you deal with the missing values in "country"?
14) How to estimate the value of a user coming to your e-commerce store when they land on your home-page for the first time?
15) How can we tell if two users on Facebook are best friends?
16) Which variables are important to predict a fake listing on eBay?
17) Explain the drawbacks of running an A/B test by market (i.e. all people in one market get version A of the site and another market version B)
18) How would you measure the performance of the customer service department?
19) FB - Should we add a love button?
20) How would you use data to evaluate if it makes sense to implement two-step authentication when users log in?
21) At Facebook we use as a metric number of likes per user per week. And, each week, we check it year over year to control for seasonality. This week the metric is dramatically down. How would you find out the reason? Logging is fine as well as the query we used to get the data
22) We are running 30 tests at the same time, trying different versions of our home page. In only one case test wins against old home page. P-value is 0.04. Would you make the change?
23) Each user on our site can be described by 100 continuous variables. What's the probability that a user is an outlier on at least one variable? What are the implications of this from a product standpoint?
24) LinkedIn has tested a new UI with the goal to increase the number of likes per user. They test it by giving the new UI to a random subset of users. Test wins by 5% on the target metric. What do you expect to happen after the new UI is applied to all users? Will that metric actually go up by ~5%, more, or less? Assume there is no novelty effect here
25) Describe one example of a classification problem where the cost of a false positive is way higher than false negative as well as the other way round
26) How to calculate for how long I should run an A/B test?
27) Suddenly, our dashboard shows that the number of picture uploads per day by Internet Explorer users went to zero. What could be the reason?
28) LinkedIn has launched its first version of the People You May Know Feature. How would you isolate the impact of the algorithm behind it w/o considering the UI change effect?
29) How would you find out if someone put a fake school on LinkedIn? I.e. they actually didn't attend it
30) You are supposed to run an A/B test for 3 weeks based on sample size calculation. But after 1 week, p-value is already significant with test winning. So your product manager pressures you to stop the test and declare it a winner. What would you tell her? Explain in layman's terms
31) What are the issues with splitting a small dataset (<1K events) in training/test set? What would you do then?
32) Using LinkedIn data, how would you predict when someone is going to change job? Assume you can use all LinkedIn user activity data
33) Between the following two metrics, which one would you choose to measure response time of an inquiry at Airbnb: percentage of responses within 16 hrs or average response time considering only responses within 16 hrs?
34) At FB, we found out that users with filled out profile infos (age, hometown, etc.) are more engaged than those without. Therefore, we figure out a way to fill out those infos automatically for all users hoping it would improve engagement. However, engagement barely changes. Why?
35) You ran an A/B test last year and it lost. When would it make sense to re-run the same test today?
36) How would you identify if an advertiser is using clickbait techniques without having a dataset with labeled events?
37) What are the most important parameters in a Random Forest?
38) In on-line gaming companies, do you expect the average revenue per user to be larger or smaller than the median revenue per user?
39) How would you increase revenue from advertising clicks if you were working for an ads company (i.e. Google, FB, etc.)?
40) Give an example of a site change that we can't test on a subset of users via a controlled experiment. How would you estimate the impact of that change?
Answers to the questions below are included ONLY in the full course in product DS
. The questions below are generally harder and less introductory. They are more applied and focus more on trade-offs between metrics, the statistical side of A/B testing, as well as data-driven product development.
INSIGHTS: Netflix offers a 30-day free trial. Currently, people need to put their CC info to join the free trial. We are thinking of letting people sign-up for the free trial w/o asking for CC info. How would you figure out if running this test makes sense?
INSIGHTS: How much would you charge for youtube premium? I.e. being able to pay to get youtube w/o ads. Would this be the same, higher or lower than the current avg value of a user per month?
INSIGHTS: In your opinion, what was the data-driven hypothesis behind testing FB stories or similar products having the content disappear within one day?
INSIGHTS: At MS Bing, we increased the number of ads shown after a given search. Revenue is up, but the total number of user searches is down. Is this good or bad?
INSIGHTS: Tell me about a situation in which your analysis results were different than what you would have expected. Why was that? What did you do?
INSIGHTS: Is it better to place google ads in a deterministic way (always in the same place on the page) or probabilistically (each search result spot has a fixed probability of being used for ads)? Assume both ways have the same expected number of ads shown.
INSIGHTS: Would you test a new feature that makes it easier for Facebook users to switch between accounts?
INSIGHTS: How would you figure out if it makes sense for FB to run Whatsapp and FB Messanger as two separate apps or merge them into one?
INSIGHTS: We want to build a logistic regression to predict conversion rate. One variable is country. It is categorical with many levels. What’s the difference between building a different regression for each country level vs applying one-hot-encoding to country and building just one regression? Which approach would you choose?
INSIGHTS: How would you minimize the avg number of booking requests per booked trip at Airbnb?
INSIGHTS: FB mobile web stopped working because of a bug. Surprisingly, this led to a spike in engagement per day, defined as total actions/total active users. How would you explain it?
INSIGHTS: If you had to improve FB marketplace, what would you do?
A/B TESTING: FB developed a new feature and performed an A/B test. Results: actions per user is up, likes is up, comments is down, timespent is down. All else neutral. Would you make the change for all users based on these results?
A/B TESTING: Conversion is a dummy variable, i.e 0/1. Why can we do a t-test on conversion rate if the main t-test requirement is that the metric we are testing follows a normal distribution?
A/B TESTING: How would you test the success of a new ad campaign?
A/B TESTING: Define test statistical significance in layman’s terms. Why do people often choose 0.05 as threshold? Wouldn’t say 0.4 or 0.45 lead to higher gains in the long run?
A/B TESTING: We ran an A/B test. Results were non-significant, but slightly so. P-value was 0.06. What would you do?
A/B TESTING: How to test different prices?
A/B TESTING: Some companies run tests with the following strategy: firstly a test is run on a small percentage of users (say 5%). Then if the test group wins, an additional 5% of users enters the test. If this group also wins, the change gets implemented for everyone. What’s the difference between this strategy and running a normal single test?
A/B TESTING: After running an A/B test on conversion rate for 1 week, the width of the 95% confidence interval is about 1%. For how long you should have run the test to have a width of 0.5%?
A/B TESTING: A/B test won. We made the change for all users. After a few weeks, we want to double check if the metric actually went up after the change. How can we do that?
A/B TESTING: A/B testing can lead to over optimize for the current user base missing out on growth opportunities on new/different users. How would you avoid this problem?
A/B TESTING: Can you describe a situation in which a higher pvalue threshold for significance (>0.05) could potentially make sense?
METRICS: At a given ecommerce site, conversion rate goes down, but the absolute number of conversions is up. Is it a good thing or not? Can you think about a scenario that could have this outcome?
METRICS: What are the pros and cons of these two possible Youtube metrics: avg view time per user per day vs percentage of users who watch at least X minutes per day? What are the practical differences in optimizing for one vs the other one? Which one would you choose and why?
METRICS: At Amazon, we are running an A/B test to check if a given UI change increases conversion rate. Would you also test on other metrics, such as, for instance, revenue or number of visitors? What are the pros and cons of testing on multiple metrics?
METRICS: Our fraud algorithm has 98% accuracy. Do you think this is good or bad? Follow up: Assume the cost of not catching a fraud (false negative) is super low. Would you be OK in that case having 98% accuracy?
METRICS: Our dashboard at Google shows a sudden drop for a given metric. It was because of a logging bug and was fixed quickly. Would you still do some analysis related to that event? What would you look at?
METRICS: Same as the previous question, but this time it was because of a product bug. For instance, the video recommendation model stopped working and people started getting random recommendations. The bug was quickly fixed and things went back to normal. Would you still do some analysis related to that event? What would you look at?
METRICS: Avg number of likes per FB user per day is larger, smaller, or same as the median? Can you give an example of a metric where the relationship avg-median is flipped?
Answers for all these questions are included in the Product Data Science course