Of course I wasn't kidding is nothing that he says, because he never thought that Togusa thought he was kidding.

"I do..." He puts down the rifle on the table. Even if he remembers being able to use it effortlessly, he is far from comfortable with it right now. If Togusa wants to take a closer look on it though, he's free to. "It was only a very brief memory, and I couldn't see who or what I was fighting." The thought of it though... He doesn't want to fight anyone.