Estimating linear regression models with a VERY LARGE number of fixed effects in R - Part 1/2

-
It is very common for researchers (or data scientists) not to have an access to some the (independent) variables, which can potentially influence your dependent variable (e.g. market share, sales, choice of brand, etc). These omitted variables can potentially cause biased estimates of marketing mix effects.  

Fixed effect regressions (i.e. dummy variable regression) is widely used to control potential omitted variable biases. In a typical panel data or multi-level data settings in Marketing and Economics, you can easily end up with a VERY LARGE number of fixed effects to obtain consistent estimates. As an example, if you are trying to estimate price elasticities for each of 1,000 stores in 30 markets, which belong to the same/different retail chain, you might need to put in store-fixed effects (1,000) to control potential endogeneity. In addition, you might be worried about the potential correlations between unobserved market-level advertising and prices, thus you may need to put in market-time fixed effects. Assuming weekly data for 2 years (i.e. 104 weeks), this will result in 30 x 104 = 3,120 fixed effects. If we consider both store fixed effects and market-time fixed effects, a linear regression of unit sales of brands may need to control 4,120 fixed effects with 100,000 observations (or sometimes with millions of observations.) Most of common statistical software are not very suitable for this type of estimation. As a example, STATA can only handle up to 11,000 variables (or dummies). Even with 1000 dummies, the estimation takes really long to finish.

An extension of "within" transformation with more than one factor, which is called as "Methods of Alternating Projection" allows us to estimate linear regression models with a VERY LARGE number of dummies. lfe-package in R (written by Simon Gaure; based on alternating projection methods) is a handy tool for the estimation of linear regression models with a VERY LARGE number of dummies. The package has been tested on datasets with 15 covariates and approximately 2,300,000 and 270,000 group levels. The estimation of effects (felm) took ~ 50 minutes on a 8 CPU machine. Recovery of fixed effects (getfe) took 5 minutes.

Shown below are useful links for the lfe package (in R). I will show the test results of this package with fake data in the next posting.

1. CRAN site to download R package (lfe)

2. lfe manual (R package)

3. R documentation (felm)

4. R documentation (getfe)


 


 

Comments

Post a Comment

Popular posts from this blog

Cracking Business Case Interviews for Data Scientists: Part 1

-
Image
Image by Minha Hwang Business case study interview questions are often the most difficult part of the interview process, especially for early data scientists without job experience. Even after landing on a job, it is beneficial to have a structured way of formulating and solving business problems from other teams and stakeholders. Based on my learning as an Ex-McKinsey consultant and data scientist over 10 years, I am sharing a “7 steps of problem-solving” approach. This structured business case/problem-solving process and related framework and templates will help you to become more comfortable in approaching and solving new business problems. In this article, I will discuss: (1) Overview of a “ 7 steps of problem-solving ” approach (2) Step 1: Define a problem  — How to avoid Type III error by solving a proper problem (3) Step 2: Structure a problem  — How to break down a problem to manageable sub-problems (4) Step 3: Prioritize issues —  How to prioritize sub-problems to solve Overv
Read more

How The Influence of Multi-Tiered Private Label Brand Architecture Varies Across Retailers

-
Image
Starting from Europe, many retailers now provide more than one quality tier (e.g. premium, standard, and economy) private labels in a specific category. To distinguish different quality tier products, retailers usually use different brand architecture. The examples below show that the actual branding architecture choice in the toilet paper private labels by three supermarket chains in Los Angeles.   Retailers have many choices in the private label branding architecture. As an example, retailer banners (e.g. Vons) can be combined with modifiers (e.g. premium, standard, and economy) in the "endorsed banner brand architecture." Retailers can choose not to use the retailer banner, but still use a common group brand (e.g. Home Life) with modifiers in the "group brand architecture". Finally, different names can be assigned for each quality tier in the "stand-alone brand architecture." Some combinations of these approaches are also possible. The natural
Read more

Cracking Business Case Interviews for Data Scientists: Part 2

-
Image
  Cracking Business Case Interviews for Data Scientists — Part 2 Image by  Minha Hwang Welcome back to “Part 2" of “Cracking Business Case Interviews for Data Scientists.” In this article, I will discuss the remaining parts of the 7 steps of a problem-solving approach. This article will cover: (1)  Step 4: Develop issue analysis and work plan  — Use an issue analysis worksheet and a work plan to “plan” for actions (2)  Step 5: Conduct analyses  — Sample analyses for simple cases (a toy example) (3)  Step 6: Synthesize findings  — Build storyline pyramids (“Grouping” structure vs. “Argument” structure) (4)  Step 7: Develop recommendations  — Convert a storyline pyramid to outline and final presentation (5 )   Wrap-up: Key Takeaways Step 4: Develop an issue analysis and work plan Once we prioritize issues to analyze, we need to (1) design detailed analyses and (2) conduct proper work planning. Step 4 of the 7 steps of a problem-solving approach helps to plan for action. Figure 28 sh
Read more