How do you find the correlation between independent and dependent variables in Python?
Let's say we have 10 independent variable x1,x2,x3,...xn which all are categorical with same levels 0,1,2 (eg., 0 = no color , 1 = Red, 2 = Green) and you have two dependent(response) variables (eg., y1 = pant length in m and y2 = waist size in m). How do we determine which independent variables (x1,x2,x3,...xn) drives the dependent variables (y1 and y2)? Show
Example of the data is as follows:
I tried PLS regression in python and here is my code
Actual result from this approach is as follows: I am getting a numpy array for all the rows in the dataset and is as follows
How to interpret this? And, is there is a way to calculate direct correlations and feature importance in this kind of problems? IntroductionBuilding high-performing machine learning algorithms depends on identifying relationships between variables. This helps in feature engineering as well as deciding on the machine learning algorithm. In this guide, you will learn techniques for finding relationships in data with Python. DataIn this guide, we will use a fictitious dataset of loan applicants containing 200 observations and ten variables, as described below:
Let’s start by loading the required libraries and the data.
python
Relationship between Numerical VariablesMany machine learning algorithms require that the continuous variables not be correlated with each other, a phenomenon called multicollinearity. Establishing relationships between the numerical variables is a common step to detect and treat multicollinearity. Correlation MatrixCreating a correlation matrix is a technique to identify multicollinearity among numerical variables. In Python, this can be created using the
The output above shows presence of strong linear correlation between the variables
Correlation PlotThe correlation can also be visualized using a correlation plot, which is implemented using the The first line of code below creates a new dataset,
python Correlation TestA correlation test is another method to determine the presence and extent of a linear relationship between two quantitative variables. In our case, we would like to statistically test whether there is a correlation between the applicant’s investment and their work experience. The first step is to visualize the relationship with a scatter plot, which is done using the line of code below.
python The above plot suggests the absence of a linear relationship between the two variables. We can quantify this inference by calculating the correlation coefficient using the line of code below.
python
The value of 0.07 shows a positive but weak linear relationship between the two variables. Let’s confirm this with the linear regression correlation test, which is
done in Python with the
python
Since the p-value of 0.2814 is greater than 0.05, we fail to reject the null hypothesis that the relationship between the applicant’s investment and their work experience is not significant. Let us consider
another example of correlation between
python
In this case, the p-value is smaller than 0.05, so we reject the null hypothesis that the relationship between the applicants’ income and their work experience is not significant. Relationship Between Categorical VariablesIn the previous sections, we covered techniques of finding relationships between numerical variables. It is equally important to understand and estimate the relationship between categorical variables. Frequency TableA frequency table is a simple but
effective way of finding distribution between two categorical variables. The
python
The output above shows that divorced applicants have a higher probability of getting loan approvals (at 56.8 percent) compared to married applicants (at 19.6 percent). To test whether this insight is statistically significant or not, we use the chi-square test of independence. Chi-square Test of IndependenceThe chi-square test of independence is used to determine whether there is an association between two or more categorical variables. In our case, we would like to test whether the marital status of the applicants has any association with their approval status. This can be easily done in Python using the
python
The second value of the above output —
5.859053936061414e-06 -— represents the p-value of the test. As evident, the p-value is less than 0.05, hence we reject the null hypothesis that the marital status of the applicants is not associated with the approval status. In order to understand the parameters and the result of the ConclusionIn this guide, you have learned techniques of finding relationships in data for both numerical and categorical variables. You also learned about how to interpret the results of the tests by statistically validating the relationship between the variables. To learn more about data science using Python, please refer to the following guides: How do you find the correlation between variables in Python?To calculate the correlation between two variables in Python, we can use the Numpy corrcoef() function.
Can dependent and independent variables be correlated?Correlation can be used to quantify the linear dependency of two variables. It cannot capture non-linear relationship between variables. Independent variables has NIL correlation, r=0. If r=0, indicates NIL correlation but not a non dependency (Independency), they can be dependent.
How do you find the correlation between two variables in Pandas?By using corr() function we can get the correlation between two columns in the dataframe.
|