How good is AI at software coding?
Carl Brown, founder of the YouTube channel Internet of Bugs, has made a name for himself by holding AI claims up to scrutiny. So we asked the physicist and software developer to help us assess how good AI is at coding tasks.
Using our AI testing software, which simultaneously queries five leading AI models, Brown asked the models coding questions. He published the results on his YouTube channel and spoke with us about his work on our Ingredients video interview series.
He found that they were not adept at simple tasks such as debugging code that cropped and resized an image. And he found that they also were not good at an important part of software development – estimating the size and scope of a task. Four of the five models estimated that building a simple photo browser would be the same size task as building a replica of the TikTok app.
“Much of the job of a software developer, or at least a senior software developer, is actually talking to people,” Brown said.
Brown started his channel because his child is studying computer science and was worried that AI would make their field obsolete before their career even began. For now, Brown says he doesn’t believe the hype of AI models performing at a human programmer level is real.
