You don't interact with the POI but with some arbitrary thing beneath it, which doesn't seem to show up in any way. Good luck, all future players.