In the last few years, we have seen many remarkable developments in AI technology. This has been the first time many of us have experienced for ourselves the types of AI capabilities which previously were only discussed in science fiction.
However, before ChatGPT can write your party invitation in the style of a medieval troubadour or suggest a recipe based on the contents of your fridge in mere seconds, the Large Language Model that powers it must be trained on an enormous amount of data. During this training, the responses the AI model produces are improved step-by-step in an optimisation process called Gradient Descent. As you read this abstract, huge numbers of software engineers and researchers around the world are continuously tweaking various parameters of Gradient Descent. Their aim is to train the models faster and to make the resulting AI systems generate better responses. The method employed to make those adjustments is, fairly often, simply trial and error. Consequently, the best, known settings for Gradient Descent do not always have a good theoretical justification.
My research attempts to bridge some of this gap between theory and practice by examining how the empirical performance of Gradient Descent is affected by various ways of choosing its step sizes, when the error is modelled by a certain class of functions. The results will be useful for future research aiming to develop new theoretical models of Gradient Descent’s performance. It could therefore contribute towards improving the optimisation method used in a wide variety of applications from Machine Learning and AI to analysis of seismographic data.