Coding AI benchmark

[ndrean]

Not sure what this "ranking" means anyway.

At least, this is supposed to be up-to-dated, so it gives you an idea of what's up!

https://apxml.com/leaderboards/coding-llms

More informations: https://apxml.com/posts/best-llms-for-coding

[nelsonic]

Thanks for opening/sharing. 🔗 ❤️
Always good to visualise the "state of the art" as a table like this. 👌
Which of these are you using?

---

Rant:

For now, I see Ai Coding Assistants as Technical Debt Accelerators. 📉
People can churn out lower quality Ai-assisted code at a much higher rate than they can understand how it all works and thus they, by definition, push Tech Debt!

I wouldn't hire someone to my team who claimed to use an Ai coding assistant,
not because it wouldn't help them write code faster, ⏩
but because they would be dependent on it ⛓️
and would gradually lose their ability to think and problem-solve ... 🤔

I can't help but think that AI Code Gen is driven by low-cost devs in an off-shore code factory with assistance from LLMs, but that there's still a lot of Smoke and Mirrors going on ... 🙊

Don't mean this with any disrespect to anyone.
Just that Builder.ai has recently unraveled: https://youtu.be/TuAsFCcvWPg
Which showed that "AI" code was actually being written by a few thousand engineers in India ... 🧑🏽‍💻🇮🇳

Often see the following comment on YT videos about Ai:

AI - An Indian
API - A Person in India
LLM - Low-cost Labor in Mumbai
AGI - A Genius Indian
GPT - Gujarati Professional Typist

It's funny, but could it be true...? 🤷‍♂️

With the amount of cash that OpenAI has raised and is burning
https://www.wheresyoured.at/wheres-the-money

I wouldn't be at all surprised if they had an army of people in India
reading your ChatGPT prompts, and typing furiously to reply to questions.
And those human written replies form the basis for future replies to similar questions. 💬
i.e. training the LLM what to reply to similar questions that other people have.
You -> ChatGPT -> Human (Actual Indian) -> LLM Answer

Note: not saying that Ai / LLMs / Stable Diffusion aren't awesome tech! 😍
Just that they are the "Microwave Oven" that gives people Ready Meals 🥘
So they people forget (or never learn) how to cook from first principals.

At the same time I'm endlessly impressed by what DeepMind are doing ... 🧠
Highly recommend: https://deepmind.google/discover/the-podcast/ 🎧

[ndrean]

I love your TLAs about India !!!

My tiny local RAG experience convinced me that there is no such Army of India behind.

I claim that it is useful because a semantic search is - most of time - superior to a full text search.

I shamelessly use the Copilot agent in VSCode, where I can use Gemini 2.5 or Claude Sonnet 4.
It is free.
I would not pay - as an Elixir hobbyist - for Cursor or Phoenix.new

I don't use it extensively, and maybe I should use it more, but my usage is:

• let it review my error messages, and contemplate its response,
• use it mainly for CSS styling
• check SQL queries and db related things
• ask for test proposals, whether in the console or as code,
• ask for a code review and possible refactors.

But... you have to review and challenge every single line it proposes to add.

The first reflex should be RTFM.

For example, I experimented the Elixir SSR rendering ContEx library. I understood the usge (a bit) only by reading the documentation and reading the code.

For JavaScript, it is rather good.
Just like researching into Elixirform or asking on the stackoverflow or slack for help, it is just another always available tool.

I would not discourage from using this and would not blacklist a person using it.

[nelsonic]

Definitely not "blacklist" or "greylist", don't worry.

This is just my opinion as someone who has worked with and mentored many "junior" devs over the years. (Which is not your case; you learned your core skills long before LLM-coding was mainstream)
I see them becoming dependent on Cursor or Windsurf and then when the service is unavailable they are unable to work! 🙃
I see the same people unable to make their own food and become dependent on Uber Eats (or similar).
Again, not saying that everyone should cook their own food; that's why we have chefs who specialise and we can pay some of our money in exchange for gourmet meals.
But it makes me sad when I see people who don't want to know how to do things from first principals
and just want the instant gratification of a result even if it's way lower quality than what they could achieve with even modest effort.

Again, definitely not saying this is the case with you specifically.
I've seen your pre-LLM work and it's A+. 🤩
So I assume you're now able to learn and get more done in less time. ⏩

The tools are just tools. 🛠️
If people use them appropriately they can be awesome. 😍
Totally agree that running a local RAG or Image Gen model shows how they are not "An Indian". 😉
We saw it clearly on our image-classifier project. 🖼️ -> 🤖 -> 📝
Like I said, I'm super bullish on this wave, especially DeepMind. 📈 🚀

I'm choosing the "Ai Skeptic" view point because I have to think about it critically.
I know that "Teenage Nelson" would absolutely have let ChatGPT write his English essays ... 🙃
So I have to fight the urge to take shortcuts. 🙅

The "Cognitive Debt" described by MIT Researchers is very real:
https://www.media.mit.edu/publications/your-brain-on-chatgpt/ 🔗
Full Paper: https://arxiv.org/pdf/2506.08872 📜

The takeaway isn’t “Don’t use ChatGPT.” It’s: don’t let it think for you.

"shamelessly use the Copilot agent in VSCode, where I can use Gemini 2.5 or Claude Sonnet 4.
It is free."

Top tip. 👌

[ndrean]

Very nice !! -> https://www.media.mit.edu/publications/your-brain-on-chatgpt/

I remember this girl -> Nataliya Kosmyna. She worked on interfaces machine/brain to pilot things "by thinking" so I tend to believe the findings of this paper.

I believe that we all know that the feeling of "the vertigo of the blank page" is an essential part of the learning process. For essays, for code, for mathematics, before doing sports, for everything. If you can't face it but pretend you can, you are fooling only yourself, or - briefly - others, and in this case, you don't bring any value long term wise.
Saying that, we all need to make €s so I don't blame them.
My concern is more about how fast technology is advancing, and that work is becoming scarce. Perhaps these "super" users might win at the end?

[nelsonic]

Indeed Nataliya Kosmyna is a legit BCI researcher https://www.media.mit.edu/people/nkosmyna/overview
Zero reason to think she has faked any of this paper; I'm sure plenty of people (and bots) have scrutinised it.

"Super" Engineers have always existed; 10x or even 100x. 🦸‍♀️
The problem has been the engineers who appear to be 10x
but are actually just creating massive tech debt for others to suffer with. 😢
Now it's just easier for a novice to think they have gained super-powers
when in fact they just have a super-tool they outsource thinking to.

Will Ai-assisted engineers "win" in the short-term; definitely.
Because they can churn out more "features" in less time.
Will they continue winning in the long-term, maybe.
They will be seen as "high value" by the company because they churned out so much (Ai-gen) code.
The real test of these Engineers will be when there is a Bug or Security Vulnerability
that "their" (Ai-generated) code introduced and they have no recollection how any of it works. 💔
And they are having to debug it in Prod at 3am on a weekend while hackers are stealing the company's data.

The Ai arms race will continue for the foreseeable future. 🚀
Speaking of BCIs ... would you get a Nerualink (or similar)
when they become generally available to 10000x your bit rate? 💭

[ndrean]

I watched https://www.youtube.com/@DanielBergholz. In particular:

You - and I - are stone age hunters 😬.

Or maybe only me, but we did not mention Claude code.

He says that Copilot is nowhere near Cursor or Windsurf.

I don't know because I have no paid subscription.

What he describes is exactly how I thought of using it. One really important point - I believe - is helping on testing.

Maybe I will try this and get rid of 20€?

[nelsonic]

Love Daniel's channel and don't mean him or anyone else any offence,
but his Ai-workflow example (adding a counter) produced very naive (beginner) code. 🔰
it's not well architected at all and would need to be refactored to be useful ... 👎
perhaps it could be used as the basis for building a feature,
but if an engineer submitted that code as the "solution" it wouldn't be good.

I'm still super bullish on Ai and it will be awesome when the code it produces "thinks" beyond the superficial use-case.
But I'm not holding my breath. And certainly not for creating anything more creative.

[ndrean]

I probably missed the AI generated counter workflow in one of his videos where is this?

[nelsonic]

https://youtu.be/bUg_jxtnT3k T03:50
Basic counter. Too basic to be useful.
But it’s hailed as “amazing” by people who have never analysed data.

[nelsonic]

I totally get it that it’s still “early days” for Ai Code Gen and it will improve. But the models have already trained on all of GitHub/GitLab, Software Engineering Books, Academic Papers, Tutorials/Examples etc.
how much more training will they need to do actual software architecture? Or will that still be the preserve of humans with decades of experience? 💭⏳

[ndrean]

Ah, with this Tidewave MCP thing. He is using GitHub Copilot with agent Claude Sonnet 4 and Tidewave.

Now he talks about Claude Code. So no more Tidewave?

🤔

So I will just try Tidewave! What is missing is having an idea of an app 😀

[nelsonic]

6 Weeks of Claude Code: https://blog.puzzmo.com/posts/2025/07/30/six-weeks-of-claude-code/
via: https://news.ycombinator.com/item?id=44767003

[ndrean]

I was curious to see the evolution.
Gemini 2.5 | Chatgpt o3 | Claude 4 opus

In VSCode, I have the following models proposed:

I use unpaid version. I use it for lots of small tasks: generate fake data for a test, propose a good name for a function, propose a documentation for a function, and github commands, review code and propose a test.
However, it will quickly lack of context and you consume the free credits quickly.

I believe you need a paid version to get the most out of this.
I am thinking of getting one. Looks like ChatGPT is the best choice.

=> Any feedback?
=> Which MCP?