James notices exactly which table he picked, and based on how Togusa's behaving so far, he can guess why. He leads them to the table and then takes the seat facing the rest of the cafeteria, so that Togusa can take the seat facing the corner where the only person in sight is him.

As soon as they're sat down, he opens his mouth.

"Seriously, Togusa. What's going on?" Where he sounded inquisitive before, now he's more serious and demanding.