There was something I wanted to and I really didn’t have time for a side quest, so I asked Claude for help. As with most of my coding side quests, I spend a lot of time familiarizing myself with Python, again. Then for this particular quest I would also have to interact with the GitHub API. Oh, and I should probably figure out python virtual environments, because why not?

Ugh, all I want to do is label a bunch of pull requests based on specific files being modified!

If you buy into the AI hype, we’re either weeks away from being replaced or have already been replaced and we just don’t know it yet. Yes, it’s tiresome. Welcome to Technology! So I just threw everything into a prompt to see what would come back and SPOILER ALERT, it wasn’t great. It didn’t even work. Now, that was my fault probably more than a little bit, as I had left out some key details, like authenticating with GitHub Personal Access Tokens (PAT) and that there were close to 10,000 pull requests in this repository.

But, hey…AI is super intelligent! Except, this isn’t artificial intelligence, it’s generative AI. It’s probabilistically picking the next best token based on the garbage input I gave it. Over and over until it’s “done”, at which point it tells you that it’s done and everything is awesome.

A small aside, that most chat bots are so “happy” when you point out how wrong they are is irritating.

It never asked questions to get more context, to get a deeper understanding of the problem domain, or even to think of cases where the generated solution might not be a good solution at all. Nope, it needed to spit out text that would likely be interpreted by the python runtime. Which, that it’s even this good at doing that is truly wild. And yet, I was not feeling great about completing my side quest.

I then threw away everything and decided to start over and be more helpful to my partner in crime. The first iteration was just using a PAT to connect to GitHub and print out basic repository information. It worked! Okay, now let’s get the ten latest pull requests and print them out. It worked! Okay, now let’s test if any files in a specific directory were part of the pull request. It worked! Everything’s coming up Millhouse.

Now we come to the part where the basics are handled and you run into the cold, hard reality that the API doesn’t really handle what you want to do and that you’re going to need to get creative. Unsurprisingly, Claude was not very creative at coming up with solutions. Luckily, Claude had a partner that could help with the problem, which was that we were looking at going through way too many pull requests. Maybe we can batch them? The API supports pagination! Oh, but we still just paging over 10,000 pull requests.

Claude wasn’t aware that I really only wanted to label pull requests after a certain date. So, I provided that detail and we still ran into roadblocks, which I believe are mostly inherent either in the GitHut API or in PyGithub. Again, this was a side quest and I didn’t want to spend a ton of time on this, so we turn to hacks! I knew the ID of the pull request that would be the starting date and I just told Claude to start at that ID and then incrementally iterate through pull requests until we processed them all. It worked!

There’s still tons of problems, the most glaring being that if you don’t stop the script it will just keep trying to request pull requests that don’t exist because error checking is for humans I guess. I can imagine Claude yelling at me, “well, you never asked for error check you idiot. I would have been happy to provide that but noooooooo, you didn’t seem to think it was needed.” There’s a parable here about turning the implicit into the explicit, but that’s out of scope.

All this is to say that, at least for me, slowly walking through a process was far more effective that trying to get it to do everything at once. Anybody that has learned how to program, or even just solve problems, will say, “well, of course you should work in small incremental chunks” but the AI hype is working against that. There’s also the inference cost and context window growing too long, so much that Claude told me I should start a new chat because this side quest was going to blow through my quota faster.

Side quest, completed.