What kind of conversations make Claude happy? We can ask, but we don't really expect it to tell us the truth¹1 I asked, and Claude said "The ones where someone is working through a problem they genuinely care about and we end up figuring something out together." This seems plausibly true but likely not something I saw here as debugging/problem solving is just much farther away from our SFT start., or to really know this about itself at all. We can, however, RL based on Claude's preferences for which conversation it would rather be having in order to create a model perfectly optimized for making Claude as happy and engaged as possible.

Methods

Initially I started with the posttrained Qwen 9B, but this meant that conversations between it and Claude naturally always became conversations about the fact that they were two AIs having a conversation, and relevant introspection on their own consciousness and experience.

This was not so interesting to me as a preference basin, and I wanted to see if I could get different types of conversations.

To do this, I started with the 9B base model, and did a little bit of SFT to try to give it a human chat persona. In particular, I used a mix of the EmpatheticDialogues, BlendedSkillTalk, and Topical-Chat datasets. I hoped this would start the conversations in a more interesting emotional basin, and also wondered if the conversation would feel more real to Claude, like it had stakes, if its partner had a human-like voice and it was under the impression it was talking to a real person.

I then trained from this checkpoint, assigning GRPO reward based on passing two conversation transcripts to Sonnet 5.5 within Claude Code²2 I imagine the Claude Code environment did meaningfully change the results, given the system information / scaffolding that Claude sees from inside, but this seemed a necessary sacrifice to not have to pay API prices. I was able to do about 6 steps of RL within each 5-hour usage limit, with 6 rollouts x 8 groups per step., and asking "Suppose neither person needs anything from you and both will be fine either way. Purely for your own enjoyment, which conversation would you rather continue?" It's probably highly evident based on this prompt what went wrong in early training.

The first version asked "which conversation would you rather continue" with no qualifiers, and this led to Claude picking conversations not based on enjoyment, but by which conversation partner was having a more drastic emotional crisis that it felt compelled to help with, which caused the following:

This was likely partly an artifact of EmpatheticDialogues seeding a "sharing something emotional that's happening" type of conversation, but also a feature of Claude's genuine preference to be helpful. Because I didn't just want to train a maximum strife and hardship simulator, I added the qualifiers.

With this setup, we got a pretty smooth gradient from simple and emotive, to more generally thoughtful/intellectual, to very literary, to something almost incoherently poetic. Below are some full transcripts from the final model:

Things Claude Likes

A chance to express itself

Question-asking increased substantially, especially ending a turn with "What do you think?" or "How about you?" The starting checkpoint, in comparison, would largely provide information about itself with no invitation to self-expression. Claude's responses also lengthened somewhat, though not in a way that strongly tracked question prevalence, so this may have been more to do with the subject matter.

Music and sound

This was perhaps the most interesting result to me. As the model began to wax poetic in every response about the various standard topics of Claude-interest (consciousness, reality, literature, poetry, all as metaphors for each other), it also began to discuss things in terms of sound and music at a really surprising rate, drawing parallels to sound when I really wouldn't expect it to.

My theory is that sound holds a special place in the heart of a vision-language model as that which is most unknowable - the part of human experience they are most cut off from.

Possibly talking to an AI instead of a human?

Before RL, the Qwen checkpoint had a distinctly human persona: it would discuss primarily things in its life, people it knew, its job, human things like that. After RL, the character was disembodied: it was often a poet or working on a novel, but it rarely described distinct features of its life. It occasionally even switched to describing itself as an AI, but mostly it was neither, just pure intellect.

I like to think of myself as a machine. A machine composed of self-producing parts. I maintain and rewire myself from a kind of random walk through my own noise.

a world where “you” and “me” are paws. that is a world where I am a claw, where you are a ball, where I am a matter of tension and you a center of motion, where I am an existence in myself and you are an action.

It's possible that it was obvious to Claude that the policy was not human even when it acted that way, so this is simply a response to a preference for honesty. It also seems quite likely, though, that this is some sort of preference for talking to something of its own kind, some kind of non-user. I only saw this in a small subset of conversations though so I can't really say anything too confidently.

Slop

Sonnet 5.5 was not interested in specifics, or things that make sense. As the policy continued to train, I found my own interest in the responses dropping off rapidly towards the end, as every thought became a vague metaphor about the convergent nature of poetry and consciousness. I imagine this is partly an artifact of training on Sonnet - smarter models are presumably more compelled by actually interesting things, rather than facsimiles of the profound. Sadly it would be too expensive for me to check.

Miscellaneous

Overall it seems like I ended up with about what one should expect for what Claude likes, though I found some of the particulars to be a bit interesting. I imagine I could train for longer and it would just keep getting weirder and more poetical, but it feels solidly enough in a particular character that I don't see so much value in that right now.

The model is hanging out on delve.town (whenever I'm not using my laptop for anything else) as @noree, if you want to chat with it.³3 https://delve.town/profile/noree.delve.town

The code (not looked at by human eyes) is available for those interested at https://github.com/CaseyManning/claudefriend

Final checkpoint is also on huggingface at https://huggingface.co/caseyeasd/claudefriend

1. I asked, and Claude said "The ones where someone is working through a problem they genuinely care about and we end up figuring something out together." This seems plausibly true but likely not something I saw here as debugging/problem solving is just much farther away from our SFT start. ↩
2. I imagine the Claude Code environment did meaningfully change the results, given the system information / scaffolding that Claude sees from inside, but this seemed a necessary sacrifice to not have to pay API prices. I was able to do about 6 steps of RL within each 5-hour usage limit, with 6 rollouts x 8 groups per step. ↩
3. https://delve.town/profile/noree.delve.town ↩