This is the third piece of the Hello World series, where the broad scope of generative AI is discussed with AI and ML experts at Amazon. If you haven’t already, I urge you to watch my discussions with Swami Sivasubramanian, and with Sudipta Sengupta and Dan Roth.
(The above picture is of me doing my homework in 1988 when I returned to school to study computer science…. :-))
I believe that as developers, we have one of the most innovative jobs in the world. Every day, we strive to create something new. The greatest satisfaction for a developer often comes from knowing that you’ve tackled a complex problem or developed a delightful product for your users. However, coding is just a portion of the role (albeit a crucial one); there’s also collaboration with product teams, designing user experiences, making implementation decisions, and planning system architectures. I contend, and I trust you do as well, that a developer’s time is best utilized in these creative endeavors rather than writing routine code to upload a file to Amazon S3.
Developer tools are an area where generative AI is already making a concrete impact on efficiency and speed, which is why I am enthusiastic about Amazon CodeWhisperer. It is a programming assistant that utilizes a large language model (LLM) trained on open-source projects, technical documentation, and AWS services to handle many of the repetitive tasks associated with building new applications and services.
I recently sat down with Doug Seven, GM of Amazon CodeWhisperer, and Sandeep Pokkunuri, a senior principal engineer at AWS, to explore the influence of generative AI on software development — and to determine if AI programming partners detract from the job’s enjoyment.
Programming partners and code completion software are not groundbreaking concepts. Developers have been using IDEs to iterate through properties and methods for well over a decade. What sets this apart is that LLMs have the potential not only to predict the next code line but to comprehend your intention and deduce context from your existing code (including comments) to generate syntactically correct, idiomatic code. Furthermore, it simplifies mundane and time-consuming tasks like crafting unit tests or translating code across different languages.
As Doug mentioned in our conversation, this is not a substitute for expertise. It is a tool that empowers developers to devote more time to the enjoyable aspect of their work — solving challenging problems.
The full transcript of my discussion with Doug and Sandeep can be found below. For those interested in trying out CodeWhisperer, installation guidelines are available here.
Now, go build!
Recommended posts
Transcription
This transcript has been lightly edited for improved flow and readability.
***
Werner Vogels: Doug, Sandeep, thank you for meeting with me here today. We’re going to delve into the technology behind how we are assisting developers with Generative AI. But first, could you each share your roles within Amazon and in this context?
Doug Seven: Certainly. I lead Code Whisper, our large language model product for developers. With a background spanning around two decades in developer tools, my focus has been on enhancing developer productivity and enabling them to work faster, better, and with more enjoyment.
WV: Have you both worked as developers in the past?
DS: I have been a developer for a long time, which is what initially drew me to this field. I have spent a considerable amount of time writing code and problem-solving.
WV: What about you, Sandeep?
Sandeep Pokkunuri: I have been a developer at Amazon for twelve years. In fact, today marks my twelfth anniversary. My experience includes working on distributed systems, products like DynamoDB and SQS, and in recent years, I have been involved in the machine learning domain, contributing to services such as Lex and Voice ID. Currently, I am engaged in large language model projects.
WV: We often hear about Generative AI, large language models, and similar technologies. The inclusion of "language" in these terms suggests a focus on textual content – such as writing poetry or articles. How are we leveraging this tech to assist developers?
DS: Language encompasses more than just text; it is a form of expression. When developers write code, they are essentially producing textual content. By combining a developer’s process — writing code, contemplating solutions, and so on — with a large language model, we aim to understand the developer’s objectives and suggest the next steps based on that understanding. This could range from completing the current code line to offering parameter suggestions while writing a method signature.
WV: But haven’t we already had code completion features in IDEs for specific signatures?
DS: Indeed, code completion has been present for a long time. It has evolved from simple suggestions like listing available methods and properties when entering a period after a class name. The evolution now includes understanding the developer’s context and intentions to offer more comprehensive code suggestions.
WV: It’s akin to continuous pair programming.
DS: Precisely.
WV: Except your coding partner isn’t human, but rather…
DS: We refer to it as your AI coding companion. It’s like having a collaborative coding session where the AI assists you in problem-solving and code writing.
WV: And it doesn’t need to consult documentation.
DS: It has already absorbed all the relevant information.
WV: Where does the inference process occur? Is it on the local machine or does it rely on the Code Whisperer backend?
SP: Inference is just a component of the complete process. The IDE plugin plays a crucial role. It analyzes factors like the programming language in use, the developer’s current context, and the type of code being written (e.g., function definition, comment completion, loop creation). It determines the optimal time to provide code recommendations. This logic is.< p>embedded in the plugin wherever it is, and then it makes an API request. And even when it shows you one recommendation, it is still working. So all of that logic lives on the service side. And of course, we also have some cutting edge response features such as reference tracker. All of those also reside on the service side, trying to help the developer make the best decision for their customers and their applications.
WV: So tell me a bit about sort of how these models are created? I mean, it’s not all the text in the World Wide Web, I mean, because that won’t help you as a developer. So what sits inside the model?
SP: Generally when we train large language models, we collect a lot of data from the public Internet. We clean it up and make sure that we train these models such that they understand the vocabulary and the structure of the language. How do you make meaningful sentences and paragraphs in the language?
WV: If you look at sort of the imperative programming languages, let’s say you have example code that you’ve found in Java. Would the model be able to translate that into C++? So you don’t need to have the C++ code originally into the model?
SP: Yeah, the models that we build, the transformer architecture absolutely allows for that. So very soon we will be seeing automatic translation from one language to another. Especially some of the legacy languages of the older times. They want to upgrade to a newer language or even the more recent languages. You want to go from one language to another because your development team is more familiar with it or it’s more efficient. For example, Rust is quite popular these days for high performance applications. So absolutely it’s going to be possible with large language models.
WV: So I always thought that as engineers or as programmers, we have one of the most creative jobs in the world. You can go to work every morning and create something new, and it’s fun. Does this take the fun away?
DS: The way I look at this is the idea behind Code Whisper is if you and I were going to sit down and write an application together, you bring to the problem a knowledge set, I bring to the problem a knowledge set, and together we’re going to solve this problem and figure it out. And you might have some suggestions for how to do things that I wasn’t aware of. I’m like, oh, I didn’t ever think of doing it that way, and vice versa. And so Code Whisper and these generative tools work largely in the same way. We’re just going to suggest things and sometimes you’re like, yes, that’s exactly what I would have done, but now I don’t have to type it. And other times it’s like, oh, well, that’s interesting. I maybe wouldn’t have done it that way. One of the most interesting things for me was the ability to approach something that I’m not familiar with. So in my case, I wanted to just try something and I wanted to go use an API that I didn’t have a lot of experience with, and I wanted to use a programming language I hadn’t really worked in before just to see what the experience would be like.
WV: Okay, so there’s a lot of work that goes in there.
DS: A tremendous amount of work.
WV: And it’s truly augmenting my skills as a developer because quite a few of those things I would maybe by myself not be aware of.
SP: I love coding, okay? The part of the job that I do that is the most fun is actually writing code. But to me, my job is actually a lot of creation. It is a creative profession. So it’s a lot about brainstorming with the product managers about what we want for our customers, what is the desired customer experience, what makes our customers delighted? And then the implementation part is, okay, how do I convert that into designs? How do I make sure that this is highly available, highly scalable, all of that. And then finally, the last part is actually writing code. I don’t measure my self esteem based on the amount of code that I write. I measure my self esteem based on how happy the customer is.
DS: Some of my favorite comments are when we talk to people who are like, “this is bringing the fun back!” Because you think about the day in the life of a developer, and the process a developer goes through, like I said, fundamentally you’re problem solving. A part of your day is sort of mundane. A really trivial example is, oh, I’ve got to write a class to represent a data object. That’s just like, I’m going to spend the next three or four minutes typing gets and sets to represent the things that it needs to do. Or I can just type a comment that says, “a class to represent this data object” and I’m going to start generating that code and I’m going to be done with it in like 30 seconds.
WV: So that’s the way you interact with it. Basically, you give it a regular text prompt and it will go and try and find out whether it can help you with that.
DS: There’s essentially two ways. One is, as I’m writing code, so like I was saying earlier, I’m writing method signature and it’s understanding what I’m doing and it’s inferring from that that I’m going to maybe want some parameters or here’s what the function is going to look like. And so as I’m writing code, it’s kind of completing the code, sort of code completion. The other is, before I’m writing the code, I’m documenting my intent. Here’s what I want. I’m going to write a comment that describes what I want, and the language model can understand, can look at that comment and say, okay, I understand what you’re describing, and then it’ll go through and start producing that code.
WV: Okay.
SP: Let’s say you’re writing a Lambda function and you’re inside the Lambda console, Lambda editor, and you say, hey, I just want to read a message from the Kinesis stream and I want to send an SMS to the customer through Twilio. So that’s your top of the Lambda function comment. So from there you just say def read message or something. And then from the context, Code Whisperer can figure out that, okay, this person is trying to read a Kinesis message. Let me read it and let me parse it and let me pick the interesting thing and it’ll fill for me. And if I need to change something, I can just do the last bit. The last mile, I’ll take care. Don’t get me wrong, ultimately the developer is in control. They are the ones who decide whether this code is good. They’re the ones that will run and verify that it is working as expected. They’re the ones that will ship. What the generative AI based tools like Code Whisperer are helping with is you don’t have to do a lot of reading documentation pages. They’re just saying, hey, this is stuff that is easy to get. You as an application developer should be focusing on creating value for your customer by doing higher level things, not boilerplate undifferentiated heavy lifting.
DS: So you’re saying the fun part of being a developer is not reading the documentation?
SP: Yeah, absolutely. Reading documentation is not the fun part of being a developer. For sure.
WV: You’ve been using Code Whisperer probably much longer than we have. So what is it that you really like about it?
SP: To me, the most compelling part of Code Whisperer is the reference tracker feature. It was launched with it. On the day it launched, it was there. So the idea is that you’re training on a lot of public code and it’s possible that the models, the large language models, they may repeat something that they have seen at training time. And the person who is using the assistant, they may just accept your recommendation and move on. But that may not be the ideal thing
Utilizar el código puede requerir una licencia asociada al repositorio de donde se obtuvieron los datos de entrenamiento. Es importante que la persona que utilice ese código esté al tanto de esta licencia, ya que implica ciertas obligaciones que se deben cumplir. El desarrollador puede revisar la licencia y decidir seguir adelante o editar el código por su cuenta, o elegir otra recomendación de la lista que su empresa haya hecho.
WV: O que haya hecho tu empresa.
SP: Exacto.
WV: Esto cambia radicalmente la vida de los desarrolladores. ¿Significa esto que los conjuntos de habilidades de los desarrolladores van a cambiar? ¿Los requisitos? Es decir, ya no se necesita un título de informática de cuatro años para poder hacer estas cosas en realidad.
DS: Estamos haciendo que los desarrolladores sean más productivos, ayudándoles a hacer las mismas cosas más rápido. Aún así, siguen teniendo que saber lo que están haciendo, examinar las sugerencias que reciben y entender qué están haciendo. Decidir si eso es lo que quieren o tal vez es lo que desean, pero solo quieren cambiar una o dos cosas. En cierto modo, siempre lo comparo con una clase de matemáticas. Al aprender matemáticas, hay que comprender los fundamentos, como la suma, la resta, la multiplicación, la división. Y luego pasas a aprender algunos algoritmos básicos y algunas capacidades de álgebra básica. Eventualmente llegas a un punto en el que tu profesor dice: "De acuerdo, ahora puedes traer una calculadora a clase, y la vas a usar para acelerar el proceso de hacer las cosas que ya aprendiste a hacer a mano". Y eso es lo que hace Code Whisperer. Es la calculadora para un desarrollador.
WV: A veces se ve como un cambio de paradigma, pero creo que se trata más del espacio de las herramientas que de los cambios que vimos con la orientación a objetos o la programación funcional, por ejemplo. ¿Hacia dónde crees que se dirige esto? ¿Cuál es el Santo Grial?
SP: El cambio de paradigma no ocurrirá en el núcleo del proceso de desarrollo de software de programación. Estamos avanzando por el mismo camino. En lugar de ir en bicicleta, vamos en un Ferrari o algo así. Eso es lo que estamos haciendo aquí.
DS: Es un cambio enorme en la forma en que trabajan los desarrolladores. La Inteligencia Artificial Generativa se ha vuelto tan importante en nuestras conversaciones y en todo lo que hacemos, que queremos poner esto en tantas manos como sea posible, permitir que la mayor cantidad de personas puedan usar esta herramienta, aumentar la productividad y hacer más cosas.
SP: Es parte de nuestra historia de democratización de la IA. Por lo general, estas herramientas de productividad las pueden pagar las grandes empresas para sus desarrolladores. Pero al mismo tiempo, hay muchos desarrolladores de aplicaciones y freelancers que están comenzando. No tienen grandes empresas que paguen por estas licencias y demás. Apenas están empezando a construir una aplicación móvil. Quieren hacer una prueba de concepto rápida y obtener retroalimentación de sus clientes. Deberían avanzar al mismo ritmo que una persona que trabaja para una empresa muy grande que puede costear esas licencias.
WV: Están construyendo herramientas increíbles y espero que podamos desarrollar muchas más para hacer que nuestros desarrolladores sean mucho más exitosos.