"It is hardly the same thing", he retorts. "My lack of what you call a proper shirt will do me no physical harm. But I suppose, if you insist..."
Other options:
no subject
"It is hardly the same thing", he retorts. "My lack of what you call a proper shirt will do me no physical harm. But I suppose, if you insist..."