Maybe Astral Chain? Part of me wants to suggest all the fire emblem and xenoblade games as well because they're so diametrically opposed to what Nintendo is in my head.

@TimAlien I think FE and XB are too affliated with the Nintendo brand for them to be uncharacteristic, and as far as RPGs go they're very routine. You could say FE4 because of its dark tone and XBC2 due to being raunchy but even then that's a stretch. I'll think about it. Added Astral Chain

Metroid Prime I think might qualify, with the grim sci-fi tone and the fact that it's a first person shooter.
I was thinking Star Fox Adventures, though it might be a little too Nintendo-y. Star Fox is a big Nintendo franchise, but the game itself is a weird Zelda-clone developed by Rare of all people. The gameplay itself is not foreign to what Nintendo has been doing for years, but the tone of the game feels very influenced by Western sci-fi and pop culture (feels tonally like a Pixar movie, basically, and I don't mean that as an insult).
Lastly I would suggest Codename STEAM for the 3DS. The whole visual style looks like Borderlands rather than anything out of Nintendo, it has Abraham Lincoln as a playable character, and it's steampunk America subject matter isn't something I would expect out of Nintendo.
Anyhoo, feel free to pick and choose which of these suggestions meet your criteria (if any).

@DizzySkullKid19 Funnily enough, I think Prime is a very Nintendo take on the FPS genre. T-rated, focused on exploration and puzzles rather than combat, associated with motion controls, and notoriously marketed as a "first person adventure" in order to further distance itself from FPS. I think Star Fox Adventures would count if it retained its Dinosaur Planet story and aesthetics. I completely forgot about Codename STEAM, I've added it.

@bwoe Fair enough, I'm thinking in mainly aesthetic terms because thats my natural inclination, but Prime's gameplay is pretty typical of Nintendo's philosophy.