Information Gain for Each Feature



Information Gain for Each Feature

IG calculation for Fever

Fever feature there are 8 rows having value YES and 6 rows having value NO. As shown below, in the 8 rows with YES for Fever, there are 6 rows having target value YES and 2 rows having target value NO.

+-------+-------+------------------+----------+

| Fever | Cough | Breathing issues | Infected |

+-------+-------+------------------+----------+

| YES | YES | YES | YES |

+-------+-------+------------------+----------+

| YES | YES | NO | NO |

+-------+-------+------------------+----------+

| YES | NO | YES | YES |

+-------+-------+------------------+----------+

| YES | YES | YES | YES |

+-------+-------+------------------+----------+

| YES | NO | YES | YES |

+-------+-------+------------------+----------+

| YES | NO | YES | YES |

+-------+-------+------------------+----------+

| YES | YES | NO | YES |

+-------+-------+------------------+----------+

| YES | YES | NO | NO |

+-------+-------+------------------+---

The 6 rows with NO, there are 2 rows having target value YES and 4 rows having target value NO.

+-------+-------+------------------+----------+

| Fever | Cough | Breathing issues | Infected |

+-------+-------+------------------+----------+

| NO | NO | NO | NO |

+-------+-------+------------------+----------+

| NO | YES | NO | NO |

+-------+-------+------------------+----------+

| NO | YES | YES | YES |

+-------+-------+------------------+----------+

| NO | YES | NO | NO |

+-------+-------+------------------+----------+

| NO | YES | YES | YES |

+-------+-------+------------------+----------+

| NO | YES | YES | NO |

+-------+-------+------------------+----------+

# total rows

|S| = 14For v = YES, |Sᵥ| = 8

Entropy(Sᵥ) = - (6/8) * log₂(6/8) - (2/8) * log₂(2/8) = 0.81For v = NO, |Sᵥ| = 6

Entropy(Sᵥ) = - (2/6) * log₂(2/6) - (4/6) * log₂(4/6) = 0.91

# Expanding the summation in the IG formula:

IG(S, Fever) = Entropy(S) - (|Sʏᴇꜱ| / |S|) * Entropy(Sʏᴇꜱ) -

(|Sɴᴏ| / |S|) * Entropy(Sɴᴏ)∴ IG(S, Fever) = 0.99 - (8/14) * 0.81 - (6/14) * 0.91 = 0.13