# CATEGORY / MIT Data Science: Data To Insights

### P-Value

p-value: probability of observing an outcome which is at least as hostile (or adversarial) to the null hypothesis as the one observed

Example
Null hypothesis: mean lifetime of a manufacturing device = 9.4 years
Accepted: within 0.396 units

50 elements with sample mean of 8.96
What is the probability that when we generate a different and independent sample average of 50 observations, we get the value <8.96 if the null hypothesis is true?

Worse than 8.96
1. Getting a number smaller than 8.96
2. Getting a number larger than 9.84 Conclusion: the larger the p-value, the stronger the evidence supporting the hypothesis.

### Validity Of Binomial Distribution

Binomial distribution: discrete probability distribution of the number of successes in a sequence of independent yes/no experiments, each of which yields success with probability Null hypothesis: there is no there is no significant difference between specified populations, any observed difference being due to sampling or experimental error  ### Hypothesis Testing

Hypothesis testing: using a data observed from a distribution with unknown parameters, we hypothesise that the parameters of this distribution take particular values and test the validity of this hypothesis using statistical methods

Confidence intervals: provide probabilistic level of certainty regarding parameters of a distribution

Example:
1. 2. unknown mean value 3. known normal distribution: estimate of : distribution of : Suppose:   ### Types Of Errors Precision: how often a classifier is right when it says something is fraud Recall: how much of the actual fraud that we correctly detect  Harmonic mean of and      ### Regression Analysis

Definition: discovering correlations between the outcome and the set of regressors (features) : real random variable : vector or random variables Example
For wages, suppose : hourly wage : regressors (experience, gender, education)  : target regressor : controls of components

Prediction: how can we use to predict well?
Inference: how does the predicted value of change if we change a component of holding the rest of the components of fixed?

### Modularity Clustering

Method: define modularity score that we aim to maximise

Characteristic idea: compare communities to a random baseline graph that shares some properties with the actual graph, such as the number of edges and the degree of the nodes

1. Compute the number of edges within a community, then subtract the expected number of edges as per the baseline model ( )
2. : if edge is between  : if there is no edge Modularity score: If score is large, then community is dense

Process:
1. Start with some partitioning, then move nodes between the groups to see if it improves the score
2. Use eigenvectors

### Spectral Clustering

1. Create neighbourhood graph
2. Compute laplacian
3. Compute eigenvectors of laplacian, the ones with the smallest eigenvalues give new features
4. Run k-means clustering on new features

### Eigenvectors

To capture global connectivity structure, eigenvectors are really useful. Results will be spectral clustering.

Spectrum of matrix: set of eigenvalues
Matrix: Laplacian of graph

Adjacency matrix: Laplacian matrix: ### Clusters

Cluster: points that are well-connected with each other (lots of edges)
Number of edges = volume
Volume per node = density
Cut value: separation of clusters
Cut(C) = 1 (number of edges between clusters)

1st criteria: density must be large
2nd criteria: there should not be too many edges between different clusters

Normalised Cut Conductance Good clusters are not too small, internally well-connected and separated from the rest of the nodes

### Principal Component Analysis

Patterns = principal component = vector
Finds major axis of variation in data
Each data point expressed as a linear combination of patterns  Eigenvectors capture major direction that are inherent in the matrix
The larger the eigenvalue, the more important is the vector
Covariance matrix contains terms for all positive pairs of features

- PAGE 1 OF 2 -

Next Page ×