0.4 Parametric vs Non-parametric Models★★★

# Fixed Recipe or Open Notebook?

• One cook follows a fixed recipe: the same five ingredients, every time.
• Another keeps a notebook of every dish and adapts as it grows.
• The first is quick but rigid. The second is flexible but heavy.
• Machine learning models come in the same two kinds.

# Parametric and Non-parametric Models PYQ 5M

Parametric model

• A parametric model is a model with a fixed number of parameters that assumes a particular form for the function being learnt.
• However much data you give it, the number of parameters stays the same.
• Example: linear regression assumes a straight line:
  $$y=w_0+w_{1}x$$
  • Only two parameters ($w_0$, $w_1$), whether trained on 100 rows or 10 lakh rows. After training, the data can be thrown away.
• Examples: linear regression (→ 1.8), logistic regression (→ 1.9), Naive Bayes (→ 1.7), perceptron, linear SVM.

Non-parametric model

• A non-parametric model is a model that makes few assumptions about the form of the function, and whose complexity (number of parameters) grows with the amount of data.
• "Non-parametric" doesn't mean no parameters; it means not a fixed number.
• Example: KNN keeps all training points; a decision tree grows more branches as data grows.
• Examples: KNN (→ 1.3), decision trees (→ 1.4), kernel SVM, random forests.

flowchart TD
    A["<b>Training data</b>"]:::blue --> B{"Assume a fixed<br>form of function?"}:::amber
    B -->|"Yes"| C["<b>Parametric</b><br>learn fixed parameters,<br>then discard data"]:::green
    B -->|"No"| D["<b>Non-parametric</b><br>model grows with data,<br>often keeps the data"]:::green
    classDef blue fill:#5b84d424,stroke:#5b84d480
    classDef amber fill:#d6a24424,stroke:#d6a24480
    classDef red fill:#d0707024,stroke:#d0707080
    classDef green fill:#52a86e24,stroke:#52a86e80

# Parametric vs Non-parametric

# Why KNN Is Non-parametric PYQ 3M

1. No fixed-size model: KNN doesn't learn weights or a formula; it stores the whole training set.
2. Complexity grows with data: more training points means more to store and compare.
3. No assumption about the function's shape: it predicts from the K nearest neighbours, whatever the data's pattern.
4. It is called a lazy learner: all work happens at prediction time.

• Full algorithm → 1.3.

# Pros, Cons and When to Choose

Parametric

• Pros: simple, fast, needs less data, easy to interpret.
• Cons: wrong assumptions give poor results; limited flexibility.
• Choose when: data is limited, the relationship is roughly known (e.g. linear), speed or explanation matters.

Non-parametric

• Pros: flexible, fits complex patterns, few assumptions.
• Cons: needs more data, slower, more memory, can overfit.
• Choose when: lots of data, unknown or complex relationship, accuracy matters more than speed.
• Example: predicting salary from years of experience (nearly linear) → linear regression; recognising handwritten digits (complex shapes) → KNN or kernel SVM.