At our Google Cloud Next event this past August, we announced the preview of Duet AI in Google Cloud, which embeds the power of generative AI to provide assistance to Google Cloud users of all types to help them get more done, faster. Since the announcement, we received a huge amount of interest from users all around the world. While we are on-boarding more users to Duet AI in Google Cloud preview, we would like to share some behind-the-scenes information around the journey of how we built Duet AI, especially about how we customized the foundation models powering Duet AI to make it better serve our Google Cloud users.
Foundation Models powering Duet AI in Google Cloud
Duet AI in Google Cloud leverages multiple AI foundation models to support a variety of use cases ranging from application development; operations; data analysis and visualization; database management and migration; as well as cybersecurity.
Among many foundation models powering Duet AI in Google Cloud is a family of coding-related foundation models from Google called Codey. Codey was built on Google’s next-generation language model, and was trained on a massive dataset of high-quality source code and documentation, which allows it to understand the nuances of programming languages and generate code more accurately and efficiently. Codey supports 20+ coding languages, including Go, Google Standard SQL, Java, Javascript, Python, and Typescript. It enables a wide variety of coding tasks, helping developers to work faster and close skill gaps through code completion, generation, and chat.
Optimizing Codey for application development with Google Cloud
In order to better support Google Cloud developers and more efficiently and effectively assist them with coding related tasks when they develop applications with Google Cloud technologies and tooling, we further optimized Codey to build Duet AI in Google Cloud. And this was all done without sacrificing performance and quality on other software development tasks. Let’s take a detailed look at how we did it.
Normally, customizing a foundation model — like Codey — to address the use cases of a specific domain, would involve multiple stages. Although the details of each stage may vary depending on an organization’s resources and application needs, the lifecycle of an foundation model application can be broadly outlined as follows:
- Data ingestion and preparation: Collecting and sanitizing domain training data
- Domain pre-training: Incorporating domain knowledge by fine-tuning the foundation model
- Instruction tuning: Tailoring the response style and format to the domain via instruction tuning
- Offline evaluation: Selecting the best model based on auto and human evaluations
Integration: Integrating the model with other building blocks of the LLM application like serving configuration, monitoring, and guard rails
