This leads to a meta joke: Mr. Poe is so inept at taking care of the Baudelaires that he loses them between episodes since the episodes are supposed to be following the books. In the beginning of The Miserable Mill: Part One, he looks at the camera and breaks the fourth wall, saying, "It's a catastrophe! It's unprecedented! It's off-book! It's unfortunate! The Baudelaires are lost and unsupervised! Do you know what that means?" Mr. Poe says this to Daniel Handler as the fish head salesperson.
When he says off-book he is referring to the part being different in the books than the TV show.