Reviewing AI-Written Code with CodeRabbit

Transcript

Hi everybody, Julien here. Today we're going to review a pull request that I didn't write. I asked a coding agent for one thing in my Rack chatbot, per user document collections, and the pull request now stands at 998 new lines across 24 files. Backend, frontend, tests, all of it. This video is sponsored by CodeRabbit. They paid for it and gave me access to the product. But as usual, The review, the findings, and the opinions are strictly mine. Writing the code got cheap, reviewing it did not. So the future isn't about writing code, it's about reviewing it. Let's get started. This is my local ragchatbot. It's a real project. 12 Python modules, about 1600 lines, a React front-end test files, and about 80% test coverage that runs in CI and fails the build. If I mess things up. So this is small enough to hold in your head, but it's big enough that a large change is pretty hard to read. So when an agent hands me a pull request this size, I really have four jobs. Review it. Is it correct? Prioritize it. Is this the thing I should look at first? Understand it. What did I actually change and in what order? And secure it. Did it open anything? Up so that's the structure of this video one job at a time code rabbit is going to help me through this first you have to set it up and it is the easiest thing all you have to do is connect it to your github account and allow repositories so here I've got my chatbot already authorized and if I wanted to authorize another one like maybe why not my commander project all have to do is just add it to the list and save okay and that's all you need to do and that only takes a few seconds in the interest of time obviously I built the feature before the video and that's the prompt I used add per user to command collections each user gets their own pdf corpus and chroma collection chroma is the vector db that I use here selected per request okay so I built it with cloud code My session has already run. We have the code. And this is what we want to review. And although my prompt is fairly simple, this pull request actually touches a little bit of everything in the application, the configuration, the vector store, the API, the front end, and the tests. And of course, it adds a notion of a user to an app which never had one. Authorization and security problems will live. It took the agent about six or seven minutes to build everything. I didn't touch the output, not one line. So now we're ready to review it. As we can see here, we've added 998 lines, removed 89, and touched 24 files. That includes the review fixes that the agent made later, which we'll see in a few minutes. I've read none of it so far. Let's see what CodeRabbit can do with this. CodeRabbit posted its review automatically about eight minutes after I opened the pull request. So we get a summary and we get a very detailed walkthrough layer by layer and we get change so ai found things is a low bar what I really care about is whether it found something I would have missed on a tired friday afternoon so here's a major problem that code rabbit found the agent guards first time indexing with a global lock so if one user's pdfs take minute to index every other user has to wait behind them but the tests pass and it works on my laptop but obviously it fails the minute a few people try to use it at the same time and then there's the security review two things first the user id is not giving me access control anyone who can reach the api can read anyone's documents by typing their name not great uh to be fair you know I'm glad the agent found it the second problem is the one I really like the per user pdf folder must never sit inside the shared pdf folder because the shared loader reads that folder recursively and every user's files would end up in everyone's index not great the agent found it it wrote a comment and a test about it code rabbit noticed that nothing in the code actually enforces it it's one environment variable away and that can really be overlooked fairly easily. When the fix is small, it's right there. User IDs can be 64 characters, the prefix adds five, and Chroma's collection name stops at 63. So let's just click this and commit it to the branch. Of course, I can also ask. Before recording, I asked CodeRabbit to walk every code path from a request user ID to a Chroma collection and to tell me where the color gets checked the answer took about 14 minutes and it starts like this the caller is not checked against the requested collection anywhere in the api path and then it walks all three endpoints the registry and the vector store with line numbers ci stays green so no need to fix the build today there's still a one click finishing touch on offer doc strings because the agent documented about a third of the functions it touched Here's my real problem though. This is how GitHub wants me to read the change, alphabetical file order. So the API next to config file, next to test with nothing telling me what depends on what. So that order has nothing to do with how the change was built or how it should be read. This is the change stack in CodeRabbit, the same pull request, but reorganized. 24 files, one stack, five layers, in dependency order. The foundation first, then what builds on it. Here's layer one, user-specific path and vector stores. And here's layer two, the collection registry and startup, and it says so, depends on the user-specific paths and vector stores, then the user-scoped API request, then the front end, and last, my review and CI settings, off to the side where they belong. Order I would explain this change to a colleague in it's now the order files are sorted in and that's the difference between reading a diff and reading a change the overview tell me what changed why it matters and what needs attention before I open a single file five layers 24 files several findings high risk and the security blast radius draws what this reaches five change components two downstream and two potentially impacted. Here's another feature called CodePeak. I can click on any function and see where it's used without leaving the review. Jumping to a definition did not resolve on this repository, so let's just see the references. And we can see the findings sit in the layer they belong to with a severity on each. Critical, major, minor, trivial. When this has to ship today, that's where I go. Start I didn't fix the rest by hand either I just gave a review back to the agent minus the one click fix you just saw and it took about a minute it fixed the lock it made the server refuse to start if the folders overlap and it improved my ci file and it also declined the access control one in writing that needs real authentication and that's out of scope fair enough then CodeRabbit reviewed the new commit. The locked thread resolved. The overlap and CI concern, gone. And one thread was still open when I ran that review, the one-click fix, which I've since deployed. And a new major problem popped up. The agent's fix keeps a lock for every user ID anyone ever asks for, including ones that don't exist. So if you ask for a million made-up names, the server will keep a million locks. So we need to fix that too. And because the agent explicitly left access control alone, the security rating went up, not down, moderate to high. So that's the loop. The agent writes, CodeRabbit reviews, the agent fixes, CodeRabbbit checks the fix, and I guess I'm the one who decides when it's done. My second job is what do I look at first? So this is my open pull request list on GitHub. Across four repositories sorted by date which is to say not sorted by anything that matters this is code rabbit triage same pull request for this repo ranked by risk review effort and what blocks what so we see we have one p1 and one p3 so the top one is the one I would have open first and the card says why it matters we have a security risk The other one is markp3 and starts with defer this change. So we can read this ranking as input and still make our own decisions. Fair warning, triage will only see the repositories where CodeRabbit is installed, so that's why you only see two PRs for this repo. This is obviously built for teams with hundreds of PRs, so this is definitely easy mode you're seeing here. My last job is what security risk Did this open up? Well, you've already seen half of it. CodeRabbit security reviews every pull request, and on this one, it rated the change high risk once the agent was done. CodeRabbbit has another security feature called AI Deep Scan. This will look at the whole code base, not just the diff. So before the agent touched anything, I run a deep scan on my main branch, on the backend request, and storage, and prompt code. Took about 20 minutes and it mapped 53 security relevant points and verified 39. And it found one thing in the code I wrote, and that's this orange box here. Let's click on it. My chat window renders the model's answer as markdown, images included, with no restriction on where an image can load from. So if you put a prompt injection inside one of the PDFs, that carries your private question to someone else's server and your browser fetches it the moment the answer renders. It starts at PDF ingestion and ends in my React component. So no diff would ever show you that. The PR scanner will see the diff, but this is looking at the whole code base. So what is this like on a real project? I guess three things I would tell anybody before they start. Number one, At the time of recording, the security features are open beta, as mentioned in CodeRabbit's docs. Also, AI DeepScan is built on usage outside the plan, so make sure you understand the pricing model. Number two, time. The first review that I ran took about eight minutes, and a chat answer took 14. That's fine for a pull request, but it's not an interactive loop. On a personal account, the queue here is tiny. I just had a bunch of PRs. So if you have a team project with more PRs, of course, you want to run some more tests to make sure this does work for you. But this is exactly what it was built for. So none of these changed my main point. I think the hard part of working with coding agents isn't getting code out of them anymore. That's fairly fast and simple. It's really knowing, what you're about to merge and ship to production. So that's what I wanted to tell you today. Thank you to CodeRabbit for sponsoring this video. All the links are in the video description and the pinned comment. You can start your free two-week trial with no credit card and just a couple of clicks. The pull requests and the prompts and all the CodeRabit reviews are public in the GitHub repo. The link will also be in the description. That you can read everything yourself. Thank you so much for watching. I hope this was fun and interesting. And until next time, my friends, you know what to do. Keep rocking.