LLMs are grounded in data science, but our approach to prompt engineering might strike us as unscientific:
- Manual prompt engineering which does not generalize well: LLMs are highly sensitive to how they are prompted for each task, so we need to handcraft long strings of instructions and demonstrations. This requires not only time-consuming prompt writing process, but the given string prompt might not generalize to different pipelines or across different LMs, data domains, or even inputs. To deal with a new problem we often need to handcraft a new prompt.
- Lack framework to conduct testing: Instead of the usual train-test regime in typical data science applications to pick the model which maximizes a certain metric like AUC, with LLMs we arrive at the best prompt via trial and error, often without an objective metric to say how well our model is performing. Thus no matter how we try to improve the prompt, we can’t confidently say how reliable our application is.
To address these issues, Stanford NLP has published a paper introducing a new approach with prompt writing: instead of manipulating free-form strings, we generate prompts via modularized programming. The associated library, called DSPy, can be found here.
This article aims to show how this “prompt programming” is done, to go deeper in explaining what’s happening behind the optimization process. The code can also be found here.
We will spend some time to go over the environment preparation. Afterwards, this article is divided into 3 sections:
- Basic concept of DSPy: Signature and Module
Basic building blocks in DSPy for describing your task, and the prompt technique used - Optimizer: Train our prompt as with machine learning
How DSPy optimizes your prompt with bootstrapping - Full fledged example: Prompt comparison with LLM
Applying the rigour of traditional machine learning for prompt testing and selection
