(With reference to this post here.)
See, this is why you start with asking for the thing you’re actually looking for, rather than circling the question with a series of tangentially related hypotheticals. We could have gotten here a lot faster! 😉
So, tabletop RPGs apart from Dread (see previous post) that use their mechanics to foster horror by messing with the players’ minds rather than merely by threatening their characters:
Don’t Rest Your Head is a game about characters who gain super powers from insomnia exploring exploring a twisted reflection of their home town. It has a couple of notable features that are relevant to the subject at hand.
First, the way the dice pools work decouples the dominant influence on a roll’s outcome from who actually succeeds or fails, so it’s possible for the players to succeed at a roll, yet have everything get worse in the process.
Secondly, there’s a reverse death spiral: the more exhausted the player characters are, the more powerful they become, but the more exhaustion dice you roll, the more likely it is that exhaustion will be the dominant influence on the outcome, causing your exhaustion level to involuntarily increase. Having exhaustion go too high and crashing out is the only way for a player character to permanently die, and there are very few opportunities to reduce it.
The result is an inversion of the usual horror RPG curve where players can afford to be cavalier early on, but are forced to play increasingly conservatively as opposition ratchets up and their resources deplete; here, you start out playing it safe, and by the end you’re taking totally deranged risks because you’re fucked anyway and there’s nowhere to go but down – just like your increasingly fatigue-addled player character would.
For a totally opposite approach, Ten Candles goes straight for the tragic mindscrew. It’s basically impossible to play online because of how heavily prop-dependent it is, but it’s pretty fantastic in person.
In a nutshell, it’s meant to be played in a room lit by ten candles (hence the name). At the start of each scene, the players start out with a communal pool of six-sided dice equal to the number of candles that are still lit. Whenever a danger or conflict comes up, the affected player rolls the dice, and if at least one six comes up, they succeed. However, any dice that show ones are discarded. Play proceeds until a conflict arises whose roll shows no sixes; at this point, a crisis occurs, a candle is snuffed out, and the scene ends. The player dice pool then refills to a number of dice equal to the number of remaining candles, and the next scene begins.
Individual traits come into play in the form of a stack of index cards, which form your character sheet. After any roll, you can choose to invoke one of your traits to reroll all of the dice that showed 1s. You can do this in order to try and turn a failure into a success, or to reduce the number of discarded dice on a successful roll. (Note, however, that as the dice pool shrinks, it becomes increasingly likely that a roll with no sixes will also show no ones, so opportunities to invoke traits to salvage faillure become rarer.) The index card representing that trait is then touched to one of the remaining candles and burned up – in a fireproof bucket, of course. Safety first!
When the final candle is snuffed, the final confrontation begins. At this point, the room is totally dark, but that’s okay because your character sheet has been completely burned up and there are no more dice to roll. Any conflict at arises during this scene automatically kills the affected player character. When the last player character is dead, the game ends.