Bayes Decision Theory - Continuous Features
Continuing from the last essay, we will now improve on the model in the following ways:
- Allowing the use of more than one feature - Like adding the shape of the cards as another feature.
- Allowing more than two states of nature - Having a deck also containing clubs and hearts.
- Allowing actions more than just deciding more than the state of nature of the cards.
- Introducing a loss function.
Allowing the use of more than one feature just means that we would replace the scaler y with the feature vector Y, where Y is in a d-dimensional Euclidean space $ R^d $, called the feature space. Allowing more than two states of nature provides a useful generalization with small notational expense. Allowing more actions also opens up the possibility of rejection i.e refusing to make a decision in too close cases. This is can be very useful if being indecisive is not too costly. The Loss function states exactly how costly each action chosen is, and is used to convert a probability determination into a decision. Cost functions enables us to look at situations where certain errors are more costly than others, although we will often only be looking at cases where all errors are equally costly.
Putting this together, let {x1,...,xc} be the finite set of c states of nature and let {k1,...,ka} be the finite set of a possible actions. The loss function λ(ki|xj) describes the loss incurred for taking action ki when the state of nature is xj. Let Y be a d-component-vector-valued-RV, and let p(Y|xj) be the conditional probability density function for Y with xj being the true state of nature. As discussed before, P(xj) is the prior probability that nature is in state xj, therefore by using Bayes formula we can find the posterior probability P(xj|Y):
$ P(x_j|\mathbf{Y})= \frac{p(\mathbf{Y}|x_j)P(x_j)}{P(\mathbf{Y})} \qquad\qquad\qquad\qquad (1) $
where
$ P(\mathbf{Y})= \sum_{j=1}^c p(\mathbf{Y}|x_j)P(x_j) \qquad\qquad\qquad\qquad (2) $
