The Nonconvexities Problem in Adaptive Control Models: A Simple Computational Solution

Marco P. Tucci
University of Siena
Tucci@unisi.it

Abstract

In recent years new attention has been paid to the problem of nonconvexities in the cost-to-go function in adaptive control models. The problem was first discovered in the late seventies (Kendrick (1978) and Norman, Norman and Palash (1979)) but only in the last few years Mizrach (1991) and Amman  and Kendrick  (1995) have returned to it showing that nonconvexities are indeed fundamental in adaptive control models and can occur even in a very simple control problem, the so-called MacRae (1972) problem, with one state variable, one control variable, one unknown parameter and a time horizon of only two periods. Using analytical methods, it has been found that nonconvexities are caused by the probing component of the total cost-to-go, Mizrach (1991), or by the cautionary component of the cost-to-go, Amman and Kendrick (1995). In both cases they are determined by the magnitude of the initial variance of the unknown parameters, the level of the parameter itself and the magnitude of the additive noise in the system equation (Amman and Kendrick (1995)).

One of the reasons of this attention is probably the fact that when nonconvexities are absent the stochastic control problem can be solved with gradient methods. On the other hand when nonconvexities are present the ``gradient solution" may be a local optimum rather than a global optimum. Therefore a full grid search method, considerably slower than a gradient procedure even though ``computational speeds are increasing to the point that even when full grid search methods are required by the nonconvexities one can use them on models of some size" (Amman and Kendrick (1995)), has to be used.

In this paper a simple algorithm, which is based on the usual gradient methods and is able to detect the presence of nonconvexities in an adaptive control problem and to find the global optimum, is first presented. Then its effectiveness in ``realistic" situations is studied performing a set of Monte Carlo experiments. Even though the code is general enough to handle more than one control and up to 33 local optima, the experiments are carried out using the one-dimensional MacRae problem. The reason is that the analytical results available for this case provide a natural framework for the conclusion of this numerical exercise.

At first the unknown parameter is assumed constant, as in Mizrach (1991) and Amman and Kendrick (1995), with a variance ranging from the original 0.5 to 0.05. These values correspond to standard deviations ranging from 0.707 to 0.224, approximately, and the results of these simulations may provide useful guidelines to the applied researcher taking the data for the control problem from an estimated model. To see if nonconvexities occur more frequently at the beginning or at the end of the time horizon used for the control experiment, time horizons of 2, 4 and 8 periods are considered. For each number of periods and for each magnitude of the variance 100 runs are performed. Another set of Monte Carlo runs is carried out, with variances 0.5 and 0.05 and 2, 4 and 8 periods assuming that the unknown parameter varies over time following a return to normality. The goal is to determine whether the presence of time varying parameters increases, or decreases, the number of cases in which nonconvexities occur.