Appropriate Problems in Decision Tree Learning

Appropriate Problems in Decision Tree Learning

  1. Instances are represented by attribute-value pairs
    “Instances are described by a fixed set of attributes (e.g., Temperature) and their values (e.g., Hot). The easiest situation for decision tree learning is when each attribute takes on a small number of disjoint possible values (e.g., Hot, Mild, Cold). However, extensions to the basic algorithm allow handling real-valued attributes as well (e.g., representing Temperature numerically).”

  2. The target function has discrete output values

    1. “The decision tree is usually used for Boolean classification (e.g., yes or no) kind of example.

    2. Decision tree methods easily extend to learning functions with more than two possible output values. A more substantial extension allows learning target functions with real-valued outputs, though the

    3. application of decision trees in this setting is less common.”

  3. Disjunctive descriptions may be required.
    Decision trees naturally represent disjunctive expression

  4. The training data may contain errors.
    “Decision tree learning methods are robust to errors, both errors in classifications of the training examples and errors in the attribute values that describe these examples.”

  5. The training data may contain missing attribute values.
    “Decision tree methods can be used even when some training examples have unknown values (e.g., if the Humidity of the day is known for only some of the training examples).”

Information Gain

  • Information gain is the measurement of changes in entropy after the segmentation of a dataset based on an attribute. 

  • It calculates how much information a feature provides us about a class.

  • According to the value of information gain, we split the node and build the decision tree.

  • A decision tree algorithm always tries to maximize the value of information gain, and a node/attribute having the highest information gain is split first. It can be calculated using the below formula:
    Information Gain= Entropy(S)- [(Weighted Avg) *Entropy(each feature)

Entropy is a metric to measure the impurity in a given attribute. It specifies randomness in data. Entropy can be calculated as:

Entropy(s)= -P(yes)log2 P(yes)- P(no) log2 P(no)


S= Total number of samples

P(yes)= probability of yes

P(no)= probability of no