The Bulldogs (16-4, 3-2 Southeastern Conference) won in Memorial Gym for the first time since 2004 and snapped their own three-game skid in Nashville. They appeared headed toward another loss after trailing 39-28 at halftime.
Vanderbilt (14-5, 4-1) had plenty of chances to extend its winning streak but missed three shots after Bost's layup, the last a 3-point attempt by John Jenkins that hit the front of the rim just before the buzzer.
Bost scored 24 points, reserve Jalen Steele added 15, and Arnett Moultrie had his 11th double-double with 21 points and 14 rebounds.
Jenkins led Vanderbilt with 21 points, Jeffery Taylor had 19, and Festus Ezeli had 12 points and 14 rebounds. Lance Goulbourne added 12 points with a career-high four 3s.
In a game featuring four of the SEC's top scorers - and with 16 NBA scouts watching - each team took turns blowing leads. Vanderbilt was up 39-26 late in the first half, while Mississippi State was ahead 63-53 with 7:50 left and had a 68-62 lead with 1:56 remaining in regulation after Moultrie scored.
Vanderbilt rallied with Jenkins, the league's best scorer for two seasons running, hitting his fifth 3-pointer with 32.8 seconds left to tie it 68-all.
Rodney Hood missed a jumper for the Bulldogs then Renardo Sidney tried to put it back up and clanked a short jumper off the left side of the rim before Dai-Jon Parker got the rebound for Vanderbilt with 1.3 seconds left. Vandy had the last shot with Brad Tinsley tossing the ball into Ezeli, who missed a jumper in the lane to send it into overtime.
Vanderbilt led twice in the extra period, the last on a 3-pointer by Goulbourne with 1:09 remaining at 77-76. Bost then scored, and Vanderbilt couldn't take advantage of both Bost and Moultrie missing the front end of one-and-one down the stretch. Tinsley missed from long range then drove to the basket with his layup hitting off the rim.
After Moultrie's miss, Vanderbilt had 3.4 seconds left and the length of the court to go. Goulbourne threw the ball to Ezeli, who caught it in midair and threw it to Jenkins, but the nation's 3-point leader came up short.
The loss leaves No. 2 Kentucky, a winner over Alabama earlier Saturday, as the only undefeated team in SEC play. Vanderbilt now has lost six straight overtime games, including three this season.
Mississippi State bounced back from a loss to Ole Miss by outrebounding Vanderbilt 37-3, but the Bulldogs had a big edge at the free throw line, going 14 of 21 compared to 5 of 9 for Vandy.
Renardo Sidney, limited to 2 minutes with two fouls in the first half, helped revive the Bulldogs in the second. He teamed with Moultrie to give the Bulldogs plenty of presence in the paint, and Mississippi State came out of halftime with a 14-0 run that erased a 39-28 deficit at the break.
Vanderbilt didn't score until Taylor's layup with 16:43 left to pull within 42-41. Kedren Johnson hit a 3 from the top of the key to pull Vanderbilt to 54-53 midway through the half, but Mississippi State was in a stretch where the Bulldogs hit five straight 3s - four by Steele, the last for their biggest lead at 63-53.
Official Game Stats