You can add the groups to each question. Only the users in those groups have to answer that question.