Bangladesh have never before tasted success in New Zealand, losing all 33 matches played across three formats, including nine Tests.
But the odds of a maiden victory rose as they reduced New Zealand to 147 for five at stumps at Mount Maunganui, with Ebadot Hossain at one stage taking three wickets for no runs in eight balls.
Ebadot has four for 39 from 17 overs.
New Zealand have a small 17-run lead, but with their top three scorers from the first innings, Devon Conway, Henry Nicholls and Will Young all dismissed.
Captain Tom Latham and Tom Blundell have also been removed, leaving Ross Taylor unbeaten on 37 with Rachin Ravindra on six.
There did not appear to be any demons in the pitch, which is unlike a typical New Zealand Test surface, lacking an emerald green cover and it was showing little sign of wear after four days.
When Bangladesh were all out for 458, a 130-run lead on the fourth morning, Latham and Young confidently took New Zealand to lunch at 10 without loss.
But when play resumed, Latham (14) chopped Taskin Ahmed on to his stumps and Conway (13) edged Ebadot on to his pads and the ball ballooned to Shadman Islam, who took the catch.
Young and Taylor mounted a rescue mission but when their 73-run stand ended, with Young out for 69, Nicholls and Blundell quickly followed without bothering the scoreboard.
If Bangladesh had a problem, it was with their fielding.
Liton Das dropped a regulation chance from Young early in his innings and Taylor also had lives.
Shadman spilled a straightforward offering from Taylor off Mehidy Hasan, and there was a run-out opportunity where Bangladesh originally threw the ball to the wrong end, and Taylor was still out of his crease when the throw went to the right end — but missed the stumps.
Bangladesh, sent into bat before lunch on day two, occupied the crease for 176.2 overs before Shoriful Islam was the last man out before lunch on Tuesday.
It was an impressive team effort with half-centuries for skipper Mominul Haque, Liton Das, Mahmudul Hasan Joy and Najmul Hossain Shanto, as well as a 47 by Mehidy.
When New Zealand thought they had seen off all recognised batsmen, Mehidy and Yasir Ali put on 75 for the seventh wicket.
Unfortunately for Bangladesh, Joy required stitches to a split hand, injured while fielding, and it was unsure if he would open the batting when Bangladesh begin the final chase.