0.3 Data Basics: Features, Variable Types, Data Splits & Pre-processing★

# Garbage In, Garbage Out

• A model is only as good as the data it learns from.
• Feed it messy, missing or lopsided data, and it learns the mess.
• Most of a data scientist's time goes into fixing data, not models.
• So first: what data looks like, and how to clean it.

# Instances, Features and Labels

A dataset is a table:

• Instance (example, record): one row, e.g. one student.
• Feature (attribute, input variable): one measurable property used as input, i.e. a column such as age.
• Label (target): the output to predict.
• Feature vector: all features of one instance, e.g. (20, 8.1, 92%).

# Variable Types PYQ 1M

1. Categorical (qualitative): values are categories.
   • Nominal: no natural order. Examples: blood group, city, colour.
   • Ordinal: a natural order, but gaps aren't equal. Examples: grades A–F, ratings (poor, good, excellent), T-shirt size.
2. Numerical (quantitative): values are numbers.
   • Discrete: countable whole values. Examples: number of children, number of clicks.
   • Continuous: any value in a range. Examples: height, temperature, salary.

• Grades A, B, C, D, E, F are a categorical (ordinal) variable: categories with an order.

# Population, Sample and Sampling Error PYQ 1M

• Population: the whole group of interest, e.g. all engineering students in India.
• Sample: a smaller group actually studied, e.g. 1,000 students surveyed.
• Parameter: a value describing the population (true average height μ), usually unknown.
• Statistic: a value calculated from the sample (sample mean x̄), used to estimate the parameter.
• Sampling error is the difference between a sample statistic and the actual population parameter it estimates.

$$\text{Sampling error}=\bar{x}-\mu$$

• Example: the true average CGPA of all students is 7.2; a sample of 100 gives 7.5, so the sampling error is 0.3.
• Sampling bias: a sample that doesn't represent the population (surveying only toppers). Bigger, random samples reduce sampling error; bias needs better sampling.
• The central limit theorem and the binomial distribution → 1.5.

# Training, Validation and Test Sets

• Training set (≈ 60–80%): the model learns from it.
• Validation set (≈ 10–20%): used to tune settings and choose models.
• Test set (≈ 10–20%): used once at the end to measure real performance.
• The test set must stay unseen during training; otherwise the score is falsely high, like seeing the exam paper in advance (→ 3.4).

# Data Pre-processing

Techniques

1. Data cleaning: remove duplicates, fix typos and wrong values.
2. Handling missing values (data imputation): replacing missing values with estimated ones.
   • Mean or median (numbers), mode (categories).
   • Model-based: predict the missing value from other features (e.g. KNN imputation).
   • Or drop rows/columns with too many gaps.
3. Outliers: detect extreme values (box plot, z-score) and remove, cap or keep them as appropriate.
4. Encoding categorical data:
   • Label encoding: each category gets a number (low = 0, medium = 1, high = 2), suited to ordinal data.
   • One-hot encoding: one 0/1 column per category (Kolkata, Delhi, Mumbai), suited to nominal data.
5. Discretisation: convert numbers into bins (age → child / adult / senior).
6. Feature scaling: below.
7. Dimensionality reduction and feature selection:
   • Dimensionality reduction: compress many features into fewer new ones, e.g. PCA (→ 2.4).
   • Feature selection: keep only the most useful original features (→ 2.5).

# Feature Scaling

Feature scaling is bringing all numerical features to a similar range.

Why it is needed

• Features on large scales dominate those on small scales.
• Distance-based algorithms (KNN, K-means, SVM) and gradient-descent algorithms (linear/logistic regression, neural networks) are affected.
• Tree-based models (decision trees, random forests) don't need it.

Normalisation (min-max scaling): scales to [0, 1].
$$x^{\prime }=\frac{x-x_{\min }}{x_{\max }-x_{\min }}$$

Standardisation (z-score): mean 0, standard deviation 1.
$$z=\frac{x-\mu }{\sigma }$$

• Example: marks with mean 60 and σ = 10: a score of 80 gives z = 2.

Worked example: why KNN needs scaling

• Person A: age 25, salary ₹30,000 · Person B: age 45, salary ₹90,000.
• Unscaled distance: $\sqrt{20^2+60000^2}\approx 60000$. Age has no effect at all.
• Min-max scaled (age range 20–60, salary range ₹20,000–₹1,20,000):
  • A = (0.125, 0.10) · B = (0.625, 0.70)
  • Distance $=\sqrt{0.5^2+0.6^2}=\sqrt{0.61}\approx 0.78$
• Now both age and salary count.