Following the speaker is a good start, but it isn’t the only way to edit a conversation. It could be that someone’s non-speaking reaction is actually better than seeing the person who is talking . In the scene on this page (from Go for Broke) you’ll see the non-speaking actor on screen a lot. In fact, in the court-martial lines, we see the reaction, not the speaking actor, for an extended period. Why? Mostly it’s because the reaction is  more important. The main character in the movie—Van Johnson—is on the right, the actor on the left is insignificant to the overall story. So the editor gives extra weight to Johnson, even though he doesn’t have all that much to say.

Ultimately, editors are storytellers. So if the non-speaking actor’s reaction is more important to the story, that’s what you show.