Psychometric Evaluation of System-Thinking

The phone call asks whether a second hour is worth spending. The room repeats accumulation on paper and adds delay, leverage, a supply shock, and a fact that should kill the model.

What we are trying to do

We want to know whether a candidate can reason about a system: keep a stock apart from its flows, expect a delay, pick the lever that matters, and drop a model when a fact contradicts it. Interviews usually judge this from how well someone talks about complexity. We would rather measure it, and the method comes from neuropsychology.

We keep the clinic's discipline and change the question. Nobody is being diagnosed.

What we borrow from the clinic

	In neuropsychology	In this screen
Same instructions	Same materials, wording and order for every patient, so scores can be compared (Lezak et al., 2012).	Same prompts in the same order, in timed blocks. Interviewers use the written probes and leave the item alone.
One task, one function	A battery is read as a pattern of what holds up and what fails.	Each item targets one operation. A low score on delay says nothing about leverage.
Process scoring	Kaplan's Boston Process Approach records how the answer was reached and which errors were made (Kaplan, 1990; Libon et al., 2013).	The scale names the typical errors, Miss and Surface, and a fragment of the candidate's words backs every score.
Rules set in advance	Results are read against criteria fixed before the session.	Advance, Hold and Stop were written before the first interview. Fluency cannot override them.

The two interviews

Task F covers something the other five don't. Reading a system well does not tell you whether it will survive a bad month. A good answer walks both plants through the shock, notices that three suppliers sharing one port count as one, and says which margin it would keep and what that margin costs. The background is on the science page.

The scale

Score	Name	What it means in the room
0	Miss	The relation is absent or reversed. Often fluent.
1	Surface	A real detail of the story, used as the structure. Certainty is usually high.
2	Partial	One necessary distinction, missing the second.
3	Competent	The operation, in plain language. Jargon is optional.
4	Exceptional	Competent, plus a second model, a named falsifier, or a limit they state. Rare. Eloquence does not earn it.

What gets recorded

For each form: first name, last name, interview date, which interview, a score and a short fragment of their words on every item, and comments. The fragment is the evidence. "Impressive" is not a score.

Records stay in this browser (local storage), and clearing the browser deletes them. When the first and last name match, a second interview is filed on the same person. That is how the individual report can show both sittings.

The frameworks we hold it to

Psychometrics has detailed rules for tests used on people at work. These are the six we checked the screen against.

What the evidence says

Sackett and colleagues (2022) re-analysed decades of selection studies. Their estimates of how well each method predicts job performance:

These are averages for each kind of method; our screen has no estimate of its own yet. Levashina and colleagues (2014) describe what makes an interview structured: standard questions, anchored rating scales and rules for probing. The screen uses all of them.

The nearest existing tests

The science page says what we kept from each.

Where we fall short

Measured against those standards, the screen is young.

The standards ask for	Status	What it would take
Agreement between interviewers	Not yet	Two interviewers score the same answers independently, and we count the disagreements. This is the first check to run.
Norms	Not yet	Enough scored candidates to say what a typical score looks like.
Validity for the job	Not yet	Scores compared with how the people hired later perform.
Candidates informed beforehand	In place	The public pages of this site describe the tasks and how they are scored.

These rules are an operational reading of the annex. They have not been validated against job performance. Do not combine them with a culture-fit score and call the average a neuropsychological profile.

Open the interview forms

References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. AERA. Open access
2. Booth Sweeney, L., & Sterman, J. D. (2000). Bathtub dynamics: Initial results of a systems thinking inventory. System Dynamics Review, 16(4), 249–286. doi:10.1002/sdr.198 (author's PDF)
3. European Federation of Psychologists' Associations (2025). Model for the Review, Description and Evaluation of Psychological and Educational Tests.
4. Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new assessment perspective. Applied Psychological Measurement, 36(3), 189–213. doi:10.1177/0146621612439620
5. International Organization for Standardization (2020). ISO 10667-1 and ISO 10667-2, Assessment service delivery: Procedures and methods to assess people in work and organizational settings. Part 1, Part 2
6. International Taskforce on Assessment Center Guidelines (2015). Guidelines and ethical considerations for assessment center operations. Journal of Management, 41(4), 1244–1273. doi:10.1177/0149206314567780
7. International Test Commission. International Guidelines for Test Use.
8. Kaplan, E. (1990). The process approach to neuropsychological assessment of psychiatric patients. Journal of Neuropsychiatry and Clinical Neurosciences, 2(1), 72–87. doi:10.1176/jnp.2.1.72
9. Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M. A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67(1), 241–293. doi:10.1111/peps.12052
10. Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological Assessment (5th ed.). Oxford University Press.
11. Libon, D. J., Swenson, R. A., Ashendorf, L., Bauer, R. M., & Bowers, D. (2013). Edith Kaplan and the Boston Process Approach. The Clinical Neuropsychologist, 27(8), 1223–1233. doi:10.1080/13854046.2013.833295
12. Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. doi:10.1037/apl0000994
13. Society for Industrial and Organizational Psychology (2018). Principles for the Validation and Use of Personnel Selection Procedures (5th ed.). doi:10.1017/iop.2018.195
14. TalentLens. Watson-Glaser Critical Thinking Appraisal. Pearson.