A Qwen 1.7B decision model scores 59.38% before calibration

Nish Tahir published a hands-on article on October 10 that turns Qwen3-1.7B into a constrained decision model with five possible outputs. On a 1,221-item CommonsenseQA sample, the base model scored 0.5938 and a quick fine-tune scored 0.6241. The example also produced 0.9978 confidence for an ambiguous question, while calibration bins showed that the 0.9 to 1.0 range was only about 70% correct.

The article applies temperature scaling with a value of 3.797280788 and shows why a valid label is not the same as a trustworthy confidence score. A small classifier may be cheaper than a general LLM for a fixed routing job, but it still needs held-out examples, an abstain path, and error review. Raw scores should not be treated as probabilities until they are checked on the intended population.

This is a reproducible engineering lesson rather than a claim that Qwen is a production decision service. The sample is one dataset and the fine-tune is deliberately quick. A builder can copy the code, replace the questions with local labeled decisions, and compare accuracy, calibration, and fallback behavior against a current model.