> These values come from running a logistic regression on the factors that were most statistically significant. The y-axis represents the standardized value of the coefficients in our regression.
"Most statistically significant" does not imply that the variables themselves are statistically significant, and it does not imply that the logistic regression itself is accurate (especially since the regression uses only 5 variables). What is the accuracy of the model?
The value of the logistic regression coefficient is the log-likelihood of the estimate, not the "Significance of Regression Coefficient", which is a completely different value altogether (the p-value).
Sorry, you're right, that's unclear. We chose to run a logistic regression on a subset of factors that were statistically significant (i.e. every factor in the graph is significant). In other words, we chose the factors that had the largest effect size and then plugged them into the regression.
so wait -- you ran the regression with lots of factors, and then dropped those you found not to be statistically significant?
I would suggest it's not good practice to drop variables, even if they aren't statistically significant (and what an argument that can become if you test things simultaneously). Particularly if there's any chance they are correlated with other variables. Read Pearl; causality (which is what you're really discussing) is a rat's nest.
Also, when you plot the value of the coefficients, I would suggest ordering them by abs(coef) * stddev(var). This may not matter much for you since it looks like most of your variables are indicator variables, but it's still good practice.
edit: oh, hi, you're the Aline that wrote this? Thanks for the interesting analysis.
I ran a number of different regressions in parallel with significance testing. Ultimately, I chose to publish the figures from the one that included the factors with the largest effect sizes (determined in parallel) for simplicity, but the takeaways didn't really differ much when more factors (even ones that weren't significant) were included.
Regardless, thanks for pointing me to Pearl. Linking here for others in case they're interested, too: http://bayes.cs.ucla.edu/BOOK-2K/
You can see the model's fit (when using only the 1st variable, relative salary) in this graph https://plot.ly/~elliotk/17 - there were more variables in the regression but only those 5 had significant coefficients. You can see the error bars on the chart (eg, "made by engineer / founder" is not really significant).
I might have mis-spoke, I think there were a couple other factors we used, but Aline is right we did a dimensionality reduction before running the regression.
"Most statistically significant" does not imply that the variables themselves are statistically significant, and it does not imply that the logistic regression itself is accurate (especially since the regression uses only 5 variables). What is the accuracy of the model?
The value of the logistic regression coefficient is the log-likelihood of the estimate, not the "Significance of Regression Coefficient", which is a completely different value altogether (the p-value).