Note that the Passageway trait will trigger them too, and that they're hunting through passageways if at act 3. As such, it's recommended to have extra clues before going to the Hidden Library if you didn't get rid of them.

Finally, note they could have been much nastier though if the Halls shared a trait.