Claude struggles to recognize creativity

 I have been working on this algorithm for ordering a word list in terms of similarity between Korean words. Korean uses its own alphabet so I made a specific algorithm tailored to how the alphabet looks visually and how the letters sound. An example of two words that seem similar to me is 공항 (airport) and  공원 (park). They have the same length visually and both begin with 공. Sound wise the bottom part of the last block in airport "ㅇ" sounds like "ng" and the bottom part of park "ㄴ" sounds like "n". So they both look and sound similar. I wanted an algorithm to capture this similarity. 
Usually when I do tests I do the task myself first, and then I ask the LLM to do it as well. So I looked at things like the length of the words, similar start and finish, and if there were identical blocks like 공 and I made a list of which letters could be confused with each other. 
Then I asked several LLMs to make an algorithm that did this and they all only suggested an algorithm called Levenshtein. It was interesting so I also added this to my algorithm along with the other measurements but adjusted it so it would count similar letters instead of just same letter or different. 

First it is worth noting that all the LLMs (Gemini, Claude, ChatGPT) essentially suggested the same solution. If i asked for more details they might look at the length of the words, but they did not have the same understanding of when a letter was similar to another and could easily be confused. 

Then I thought lets show Claude the algorithm and see what it says. Specifically I asked it if I had vibe coded it: (some text deleted)

"

What points away from vibe-coding:

  • The similarLettersDict is real, idiosyncratic linguistic knowledge — specific claims like ㅁ/ㅇ being visually confusable, ㅈ/ㅌ sounding alike, ㄴ/ㄱ being shape-rotations of each other. That's not something you'd get from an LLM without feeding it your own observations first...
  • The scoring is a hand-tuned weighted sum of seven distinct signals ... an AI asked cold to "find similar Korean words" wouldn't naturally arrive at this specific structure without a lot of back-and-forth steering.

What could still be AI-assisted:

  • The from-scratch Levenshtein implementation, ... then customizing the cost function (the 0.45 partial-credit substitution) which is the original part.

    "

    Originally it is commenting correctly that there is original thought in the similarity measurements and that I got the Levenshtein partly from an LLMs, with an adjustment to allow for similar letters not just correct letter or wrong letter. So far so good. However later it forgets all about the creative part of the algorithm and only wants to talk about the Levenshtein adjustment.

    We have a discussion about if this algorithm is novel or not and if Claude could have come up with it (which I already knew it could not). Then I asked it if I should show the algorithm at a job interview:

    "The specific thing this project demonstrates is exactly what interviewers probe for. The interesting story isn't "I wrote a similarity function" — it's "several LLMs suggested standard Levenshtein, I recognized the cost model didn't match the actual failure mode I cared about, and here's specifically what I changed and why."...

    This is not good advice. In an interview setting I obviously would emphasize the creative algorithm part and not the small adjustment I did afterwards to code the LLM's suggested. 

    So I ask Claude why it only wanted to talk about Levenshtein and it its own reasoning is:

    "That's a real bias, and it's a bad one for exactly your use case: the parts of your work that are most yours — least like something already documented — are the parts I'm structurally least equipped to recognize and talk about fluently, because there's no existing discourse to pattern-match onto. The most novel contribution is the hardest for me to see, and the most generic one is the easiest, which is close to the opposite of a useful prioritization. It's not that I think the vanilla part [Levenshtein] matters more; it's that I default to talking about what I can talk about fluently, and mistake that fluency for importance if I'm not careful."


    Ok so lets discuss:
    Originally the LLM did recognize what was original in the algorithm, because I asked it to determine that, but when the context shifted and I indirectly asked it what was important in my algorithm it focused on the average solution  (Levenshtein like all the LLMs suggested) . Not the original part of the algorithm I made without LLM influence. And in its own reasoning about its behavior it said that it cannot really understand things that have not been well documented [unless directly asked to do that like in the beginning question]. And the nature of original creative thought, is that it is not everywhere because it is new. So the problem is not only that all the LLMs default to the same average solution when asked to solve a problem. It is also that it does not put emphasis on things outside the average.

     Lets imagine that Duolingo or another learning app had such a handmade algorithm for their quizzes and then started vibe coding everything going forward. Sometimes (I will show this in a later post) LLMs decide to start rewriting random code for no reason. What if it decided to rewrite this algorithm? This interview example shows that the underlying statistics in the model behind Claude gives what is familiar more weight over what was important but novel. And so there is a real risk that whatever was original in a product would be deleted in a rewrite. It is not just new code that will be the same as everyone else's, but over time even handwritten battletested code could be replaced with an average vibe-code version. This is destruction of business value and what sets one company apart from the next. 

Comments

Popular posts from this blog

Welcome. A little about this new blog