From Unaided Choice to Multi-Criteria Disclosure: A Human Study of ML Model Selection in Higher-Education Quality Assurance

Selecting a machine learning model for higher-education quality assurance is a multi-criteria decision problem that cannot be reduced to a single leaderboard metric. This paper presents a two-step disclosure protocol in which users first make an unaided model choice and then revise it after structured multi-criteria disclosure including criterion weights, normalized values, and a ranked recommendation. The overall study used a three-step experimental protocol, with the disclosure manipulation itself implemented as a two-stage intervention within Step 2. The protocol was evaluated with 38 participants under a stable Dean-oriented advisory framing across three institutional prediction tasks, yielding 228 confirmatory scenarios after predefined quality filters. Decision quality was operationalized as regret reduction relative to a frozen governance-oriented multi-criteria scoring policy. Results show that structured disclosure significantly improved policy-aligned decision quality (Wilcoxon p = 2.30 × 10⁻11, rank-biserial r = 0.864) and increased self-reported decision confidence (p = 7.62 × 10⁻12, r = 0.646). Importantly, significant improvement was already observed in the information-only stage before any explicit recommendation was shown (p = 5.18 × 10⁻5, r = 0.629). Post-decision trust changes were small and did not reach significance in the confirmatory analysis, and are therefore treated as exploratory. The findings provide protocol-level evidence that structured multi-criteria disclosure can improve alignment with a predefined governance-oriented model selection policy in educational QA settings.