The female voice may have startled Phillip into taking out his knife, ready to fight, but it's when he hears the blatant lie that he relaxes, "Joke's on you! I've raided those buildings!"

And that's how the Phone Guy calls his bluffs. Clearly, he's so used to his boss' bullshit back home, that threats against his person are seemingly brushed off like water off a duck.

"There's plenty of dead Tributes, y-you can raid those instead, easy pickings!" Except one and he'd rather not think about her at the moment. Venus' display of power spooked Gray into being so cautious of anyone that had a beacon over their heads.