I came upon an essay on this topic on the blog of Jerome Weeks
Here's an excerpt:
"Basically, these are often novels that use the hard-boiled conventions but don't fit simply into the genre. Or they are the essential definers/redefiners of the genre itself. Those conventions include, but are not limited to, a sense of moral complexity if not outright confusion, a society that is compromised or corrupt and violent, crime treated not as a puzzle to be solved but as an act of violence that typifies something about this noir world, a protagonist who doesn't so much solve a puzzle as make a dangerous moral choice or act of redemption (the protagonist himself is most likely implicated or compromised, too) -- and a menacing mood, a laconic or succinctly vivid style and various characters to suit all of this."
I think this is a goddamned brilliant encapsulation of noir and literary thrillers. That's what so many people fail to grasp: It is ALL about the dangerous - or compromised - moral choice, and the choices made in the fallout. And sometimes, that moral choice or position can seem forced upon the protagonist simply because of who the protagonist is, because of unexamined assumptions underlying the protag's personality and life. There's the ambiguity. And the act of violence MUST be emblematic about the world portrayed, not merely horrific, not merely puzzling.