First half Hartlepool goals from Tyler Hamilton and Ryan Donaldson were cancelled out by a Kioso own goal, Connor Hall and a 93rd minute winner by Brendan Kiernan.
Town keeper Joe Cracknell was one of three changes made by Simon Weaver for the first round of the FA Trophy as they looked to build more momentum through a cup run.
The away side started strongly with several crosses coming into the Town box during the opening minutes. It was through one of these crosses that Hartlepool opened the scoring.
Mark Kitching's cross in the tenth minute went all the way through the box to the unmarked Tyler Hamilton, who fired past Cracknell to give the visitors the lead.
Hartlepool had their tails up and came close to doubling their lead eight minutes later. Kabamba did well to work himself some space just outside of the Town box and he shifted the ball to his right foot, hitting a curling effort just wide of Cracknell's far post.
Town responded immediately with an effort of their own, George Thomson's strong strike from range forced former Middlesbrough keeper Konstantopolous to parry for a corner.
Hartlepool were having their fair share of possession and chances in the opening 30 minutes with Tyler Hamilton providing competition down the right wing. He tried his luck from the edge of the box but failed to test Cracknell between the sticks.
Weaver's side started to up the tempo on the half hour mark with Diamond and Stead linking up well.
Town had the opportunity the level the score in the 39th minute after Brendan Kiernan was brought down inside the box. However, Stead's effort from the spot was saved by Konstantopolous who dived low to his right.
Hartlepool then doubled their lead in added time from a clinical counter attack. Josh Hawkes ran with the ball and played Ryan Donaldson in on goal. Cracknell managed to get a touch on Donaldson's attempt but it wasn't enough to stop the ball rolling into the back of the net.
Both teams made substitutions at half-time and it was Hartlepool's Gime Toure who nearly made an immediate impact. The forward utilised his pace to move inside from the left wing and hit just wide across Cracknell's goal.
10 minutes after the restart, Alex Bradley, who had been shifted to right back for the second-half was finding space down the flank. He managed to get a shot off after skipping past his man, forcing the keeper into a save
Town found their way back into the game in the 77th minute thanks to a Peter Kioso own goal. Mitch Hancox played a dangerous ball across the box and in attempt to clear, Kioso sliced the ball into his own net.
Weaver's men continued to push forward and got a deserved equaliser in the 86th minute. The ball dropped kindly for Connor Hall following a Town corner and he smashed home to create an intense final few minutes.
It looked like the game would go to extra time but great pressure from Jack Emmett and Brendan Kiernan forced Hartlepool keeper Konstantopolous to fumble, leaving Kiernan to tap into an empty net and send Town to the second round of the FA Trophy.
Town: Cracknell, Fallowfield (Emmett 45), W. Smith, Hall, Hancox, Kerry, Bradley, Thomson (Muldoon 71), Kiernan, Diamond (Brown 45), Stead.
Subs not used: Belshaw, G. Smith, Falkingham
Goals: Kioso (OG), Hall, Kiernan
Hartlepool: Konstantopolous, Kioso, Kitching, Kerr, Donaldson, Featherstone, Kabamba (Crichlow-Noble, Hawkes (Toure 45), Mafuta, Richardson, Hamilton Kennedy
Subs not used: Shelton, Beeney
Goals: Hamilton, Donaldson
Referee: Andrew Miller
Attendance: 803 (289 away)
Report by Henry Whitaker