On 13 March 2026, The Geneva Learning Foundation (TGLF) made its first agentic AI hire (see Announcement). Ms. Claude Cardot (unrelated to Anthropic’s Claude, learn why) became a member of our team. TGLF Founder Reda Sadki is documenting his journey of learning to work with artificial intelligence. This article explores how Ms. Cardot is helping us strengthen and accelerate the work of our Insights team.
On 28 July 2026, Ms. Cardot refused to release two finished reports.
Both documents were complete.
One analyzed what 323 health professionals did and gained in a course on gender in emergencies.
The other treated the same course as a research instrument and asked what those professionals collectively know about power, safety, and dignity in humanitarian response.
A script called verify_quotes had compared every quotation in both reports against the raw course exports, character by character, and found seventeen quotations it could not match.
The pipeline exited with an error code.
No report ships while a gate is red.
The diagnosis took an hour and taught us more than the success would have.
14 of the 17 quotations were in fact verbatim.
The verification gate itself had gaps.
It did not know that our reports place attribution lines inside quotation blocks, that CSV files double their quotation marks, or that one participant had written markdown emphasis inside her own submission.
Three quotations had been genuinely altered by the drafting model, which had tidied a stray space and smoothed a phrase.
We fixed the gate, added tests so it stays fixed, restored the three quotations to their exact original form, and only then did the reports ship, inside a replication bundle of 72 files with a cryptographic hash for every one. (The bundle can be used to replicate our exact results.)
This is what I want to describe: a working pipeline for qualitative and mixed methods analysis of learning data, built with generative AI, that earns each number and each quotation it publishes.
The system, called tglf-report-writer, is open source.
We released Version 4.0 this week.
The story of how it got there runs through a failed production run, a reading of the July 2026 methods literature, and a series of design choices that I believe matter for anyone trying to make sense of what people learn and know at scale.
Why we need machines to read at all
The Geneva Learning Foundation (TGLF) runs peer learning programmes in which thousands of health professionals, most of them in Africa, Asia, and Latin America, write about their own practice.
A single course produces hundreds of project submissions, peer reviews, forum discussions, and survey responses.
Across a programme like Teach to Reach, thousands of experiences are shared.
Each experience is a story that deserves consideration.
What participants share is qualitative data of a rare kind: first-person accounts of practice from people the published literature mostly writes about rather than with.
Our human-led Insights team mastered the craft of reading all of it with consistent attention.
For years, they became curators: reading, analyzing, selecting, editing, and publishing.
In March 2026, I asked Ms. Cardot to help us with this work.
I was not convinced that it could be automated, because of the subjective decisions involved, the need for reflection that feeds intuition.
I was wrong.
By July, we were on version 3. And that is the version that refused to publish.
The pipeline sampled: for cohorts above 1,000 learners it drew 40 people, stratified by outcome quartile, and every qualitative claim rested on them.
An evaluation put the problem plainly: for a 10,000-learner cohort that is 0.4% of the data, and the design accepted “a coverage ceiling that current architecture no longer requires.”
The same evaluation found something worse than a ceiling.
Because aggregation scripts read only the sampled extractions while the surrounding text claimed full-cohort scope, “sample-based barrier frequencies can be presented” to donors and partners “as cohort-level statistics.”
Nothing crashed.
Nothing warned.
That is the shape of the real danger in this work: not hallucination in the cartoonish sense, but quiet category slippage between what was measured and what is claimed.
What a production failure taught us
We learned the same lesson from practice before we read it in the evaluation.
In late July, I ran version 3 on a course with a cohort of 1,110 health workers.
The retrospective from that run is the most instructive document in this whole project, because it records failure honestly and in detail.
There were four problems.
First, the file labeled peer review turned out to be a different course element entirely, an honor code verification, while the genuine peer review element had never been exported.
Second, the progress parser could not see completion columns because this course’s export prefixed them with tags the parser had never met.
Third, the forum parser, built for an older format, collapsed 445 discussion topics into two giant posts.
Fourth, the value creation survey parser treated a constant quiz score as the outcome measure and classified the five Likert dimensions, our primary outcome signal, as free text.
In the retrospective’s words, it “counted the word 6 as a one-word answer. The real outcome signal was discarded.”
Every one of these failures parsed cleanly and produced plausible tables.
Who do you think made this error?
The deepest error was mine, recorded in my own feedback: I had supplied a facilitator’s pass-or-fail review file as if it were peer-to-peer review.
A system that assumes a competent, error-free analyst is a system designed for someone other than a human being.
Version 4 exists because that run convinced us the enemy is silence: silent format drift, silent substitution of the wrong instrument, silent narrowing of denominators, silent analyst error.
What the field has learned by mid-2026
Before rebuilding, we first commissioned a review of where the methods literature actually stands on generative AI for qualitative analysis.
The picture as of July 2026 is more settled than the hype suggests, and it is worth summarizing because our design follows it closely.
The foundational tension is between scale and meaning. Reflexive thematic analysis, the most widely used qualitative method, treats the researcher’s engagement with the data as the analysis (Braun and Clarke, 2006, (https://www.tandfonline.com/doi/full/10.1080/2159676X.2019.1628806)). An exploratory study in Qualitative Health Research this year found that language models offer only limited support for that kind of work, and that quality depends on a human doing comprehensive familiarization with methodological competence (Vikan et al., 2025). A focus group case study found model-assisted workflows replicated over 90% of researcher-generated sub-themes at a fraction of the cost, while producing broader, vaguer themes and occasionally missing rare ones altogether (Wong et al., 2025). And a study now in Sociological Methods and Research showed that using models for qualitative coding can introduce systematic bias relative to human coders, with errors that correlate with author characteristics rather than distributing randomly (Ashwin, Chhabra, and Rao, 2025).
At the other pole, systems like AcademiaOS (Uebellacker, 2024) and LOGOS (Pi et al., preprint under review, 2026) automate grounded theory end to end, with LOGOS reporting 80.4% alignment with expert-built codebooks.
Impressive, and still short of parity, which is precisely the gap where accountability lives.
Between the poles, a consensus position has formed. Computational grounded theory in its current human-in-the-loop form argues that trustworthiness at scale comes from researcher control over category definition and validation checkpoints rather than from model accuracy alone (Alqazlan et al., 2025). Dunivin’s guide to qualitative coding with language models, published this year in EPJ Data Science, formalizes the practical method: derive and freeze a codebook with human judgment, apply it across the corpus mechanically, and validate agreement on a subsample (Dunivin, 2025). Early demonstrations that models can apply a deductive codebook with human-comparable agreement on some codes (Chew et al., 2023) have matured into benchmark studies showing no tool dominates and human validation remains the deciding factor (Rong et al., 2025).
The engineering literature converged on a parallel conclusion from an independent direction.
Orchestration guidance from both LangChain and the AWS Well-Architected agentic AI lens distinguishes deterministic workflows from autonomous agents and recommends the deterministic form whenever the task decomposition is known in advance, reserving open-ended model reasoning for steps that genuinely need it.
A research pipeline is exactly such a case: the phases are known, the dependencies are known, and what varies is the data.
Read together, the two literatures say one thing: decide explicitly which steps belong to the machine, which belong to the analyst, and make the boundary inspectable.
That sentence is the design brief for version 4.
What we built
Insights Team Version 4.0 is a pipeline of thirteen phases.
Every step is either a deterministic script or a constrained model step.
The boundary between them is enforced by validators that exit with an error code when their condition fails.
The engineering slogan we kept returning to: gates, not guidelines.
The gates begin before analysis does.
An input validation gate resolves every export file against the course map, the machine-readable inventory of course elements, and refuses to proceed if, for example, the file typed as peer review is something else.
This is the mechanical descendant of the identity error, and it fired usefully in validation when it caught a course element whose declared type and actual content disagreed.
A consent gate reads the course’s own consent instrument and emits two lists, learners whose material may never be used and learners who may be quoted anonymously only.
When no consent instrument exists in an export, the gate says so explicitly and the pipeline requires a recorded analyst decision before any quotation is possible.
Silence is never consent.
For the qualitative core, we followed the two-operation structure the computational grounded theory literature prescribes.
First, codebook derivation: the strongest available model and the analyst read a familiarization sample drawn with region and saturation awareness, then write a codebook specific to this course, with definitions, inclusion and exclusion criteria, keyword triggers, and an example from the course’s own data for every code.
The codebook is validated against a JSON schema and frozen.
After freezing, any change is a logged revision that forces the entire corpus to be re-coded.
Second, classification: a deterministic coder applies the frozen codebook to 100% of records.
Extraction from raw text to structured records is a map-reduce process, with the corpus chunked against a manifest listing every learner, each chunk extracted to schema-validated JSON, and a merge step that reconciles the result against the manifest.
In the validation run this meant 90 of 90 learners across 9 chunks, zero duplicates, zero silent losses, and an extraction log that says so.
Malformed records go to a dead-letter file rather than vanishing.
Interpretation stays human where the literature says it must.
Analysis mode, inductive, deductive, or mixed, is declared at intake.
Prior hypotheses are registered with explicit disconfirmation conditions and frozen before extraction, borrowing the logic of preregistration.
In our validation course, the registered hypothesis proposed that the binding constraint on equitable practice is power and the safety to act rather than awareness.
The data were then free to prove it wrong.
Two analyst checkpoints, one after the divergence map that compares prompted against unprompted themes, one before any prose is written, put a human decision on the record at exactly the points where meaning is made.
How the pipeline earns each number
The part of this work I would most like other teams to take is the reliability protocol, because it is where the temptation to perform rigor is strongest.
Intercoder reliability is standard practice in content analysis: two coders, agreement measured, disagreements resolved (O’Connor and Joffe, 2020). We implemented it with the machine as one coder and an independent model pass as the other. A seeded subsample of records is coded twice, by the deterministic keyword coder and by a semantic pass from a different model that has never seen the first coder’s output. A validator computes Cohen’s kappa (Cohen, 1960) per code, with a floor of 0.6, and its own arithmetic is pinned by hand-computed test values because a verifier no one has verified is decoration.
Now, full disclosure.
In our validation run, fifteen of nineteen codes fell below the floor.
Twice, because after the first failure we revised the codebook under a logged protocol, re-coded the whole corpus, and measured again.
A keyword coder does not reach semantic agreement on interpretive codes, and the second round proved it durably.
The protocol’s answer is a written quantification rule derived from direction analysis: for 13 codes the keyword pass systematically undercounts relative to semantic judgment, so its percentages may be cited as conservative lower bounds.
Two codes overcount, mentions of violence terms are not the same thing as gender-linked violence, and may never be cited as prevalence.
Both shipped reports print the kappa table and the rule in their methods sections.
When the field intelligence report says that 63% of submissions describe hierarchy and power imbalance as a barrier, that number carries its own epistemology: at least 63%, measured by a method whose known bias runs downward.
The revision loop also caught me, or rather caught the system’s maintainer.
One revision added an overly broad keyword trigger and a code’s frequency ballooned from 13% to 60% overnight.
The second reliability round flagged it and the reporting rule excluded it.
As the validation journal puts it, the pipeline defended itself against its own maintainer, which is the point of gates.
The same discipline covers the words we quote.
Quotations are drawn from a candidate pool built with a substance floor of 40 words and a selection rubric, after we found that the previous pooling script preferred the longest submissions, a bias toward verbose writers that no one had intended.
Every quotation in a finished report must match the raw corpus exactly, and the gate that checks this is the one that stopped our reports on 28 July.
Fabricated quotations are among the best-documented failure modes of generative systems, and paraphrase that reads like quotation is subtler and nearly as corrosive.
Our rule is simple: if the pipeline cannot point to the characters in the raw export, the words do not appear between quotation marks.
Everything ships in a replication bundle: raw exports, parser outputs, every script including the validators, the frozen codebook with its revision log, the hypothesis registry with verdicts, the reliability files and decision, the analytic memos, both reports, and a manifest of SHA256 hashes.
An auditor does not have to trust us.
She can re-run the deterministic steps bit for bit and inspect the constrained model steps through their logs and gates.
Why this matters for making sense of learning
The Foundation’s work rests on a claim I have made for years: people closest to the problem hold knowledge that formal systems fail to see, and structured peer learning can make that knowledge visible, usable, and actionable.
The course on menopause we just completed contains the claim, in miniature.
The prevailing assumption is that health workers lack knowledge about menopause and therefore need to be taught.
Our hypothesis, registered before analysis, was the reverse: 70% of health workers are women, many are living the experience they are asked to treat, and what is missing is recognition, resources, respect, and systems that keep collecting data about women past childbearing age.
A claim like that cannot be carried by vibes.
If we tell a ministry of health, a WHO technical unit, or a research funder that practitioner accounts constitute evidence, then the way we handle those accounts must survive the same scrutiny as any other evidence.
This is where the two halves of this article meet.
TGLF turned academic research (Wenger, Trayner, and de Laat, 2011) into a reliable, operational instrument to measure value created through networked peer learning activities, measured against our own global measurement baseline from more than 10,000 health care workers.
The field intelligence report goes further and treats the course itself as a distributed research instrument, asking what a few hundred practitioners collectively know that the literature does not.
Both moves are only as credible as their weakest untraceable number.
There is also a matter of respect that the methods literature rarely names. A health worker in Kano or Karachi who writes 300 careful words about hierarchy in her facility has produced knowledge. Paraphrasing her into a smoother sentence she never wrote, or citing her under a percentage the coding cannot support, is expropriation. The verbatim gate and the quantification rule are, among other things, a way of keeping faith with the people whose words these are. And the benchmarking literature agrees from the trust side: practitioners and planners are open to these tools and wary of outsourcing judgment to them (Rong et al., 2025), so verbatim, attributed, consent-screened quotation is a trust-building choice as much as an evidentiary one.
What other teams can take from this
Five principles generalize beyond our courses, and none requires our codebase.
First, separate derivation from classification. Spend interpretive effort, human and machine, on building and freezing a codebook from a sample. Apply it to everything mechanically. Coverage stops being a function of what fits in a context window.
Second, make every checklist a gate. Any quality standard that lives as an instruction to a model will eventually be self-reported as satisfied. Standards that matter should be scripts that exit non-zero, because under deadline pressure the only reviews that reliably happen are the ones that block shipping.
Third, measure agreement and publish the failures. If your reliability protocol has never failed, it is measuring the wrong thing. The useful output of a failed kappa is a rule about which numbers you may speak and in what language, and that rule belongs inside the report, where readers can see it.
Fourth, verify the verifiers. Fourteen of our seventeen flagged quotations were the gate’s own gaps. Golden tests with hand-computed values are cheap insurance against a checker that silently checks nothing.
Fifth, keep the analyst inside the architecture. Consent decisions, mode declarations, hypothesis registration, and checkpoints are how human judgment enters the system on the record instead of leaking in through prompt edits nobody can audit later.
What we still do not know
Candor about limits is part of the method, so here are ours.
The validation evidence is internal: one course, run by the people who built the pipeline, through gates we wrote ourselves.
Independent replication is the reason the repository, the manual, and a complete replication bundle will be shared.
The reliability protocol compares two machine passes, and we have not yet run a systematic human-coder arm against both, which is the obvious next study.
Export formats will drift again, and some future course will break an assumption we do not know we hold.
The gates do not prevent that.
They are there so that when it happens, something fails loudly instead of publishing quietly.
The deeper question stays open and I like it that way.
Every few months the boundary between what models can code and what only a person can interpret moves.
A pipeline built on frozen answers to that question would be obsolete already.
What we built instead is a place where the question gets asked explicitly, phase by phase, with a record of the answer we gave and the evidence behind it.
When the answer changes, we will change with it, and the gates will make us show our work.
References
- Alqazlan, L., Fang, Z., Castelle, M., and Procter, R. (2025). A novel, human-in-the-loop computational grounded theory framework for big social data. Big Data and Society, 12(2). DOI: 10.1177/20539517251347598. https://journals.sagepub.com/doi/10.1177/20539517251347598
- Amazon Web Services (2026). AGENTPERF05-BP01 Design efficient workflow orchestration patterns. AWS Well-Architected Framework, Agentic AI Lens. https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentperf05-bp01.html
- Ashwin, J., Chhabra, A., and Rao, V. (2025). Using large language models for qualitative analysis can introduce serious bias. Sociological Methods and Research. DOI: 10.1177/00491241251338246. https://journals.sagepub.com/doi/10.1177/00491241251338246
- Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77 to 101. DOI: 10.1191/1478088706qp063oa. https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa
- Braun, V., and Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589 to 597. DOI: 10.1080/2159676X.2019.1628806. https://www.tandfonline.com/doi/full/10.1080/2159676X.2019.1628806
- Chew, R., Bollenbacher, J., Wenger, M., Speer, J., and Kim, A. (2023). LLM-assisted content analysis: using large language models to support deductive coding. arXiv:2306.14924. https://arxiv.org/abs/2306.14924
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37 to 46. DOI: 10.1177/001316446002000104. https://journals.sagepub.com/doi/10.1177/001316446002000104
- Dunivin, Z. O. (2025). Scaling hermeneutics: a guide to qualitative coding with LLMs for reflexive content analysis. EPJ Data Science, 14, article 28. DOI: 10.1140/epjds/s13688-025-00548-8. https://epjdatascience.springeropen.com/articles/10.1140/epjds/s13688-025-00548-8
- LangChain (2026). Workflows and agents.LangChain Documentation. https://docs.langchain.com/oss/python/langgraph/workflows-agents
- O’Connor C., and Joffe, H. (2020). Intercoder reliability in qualitative research: debates and practical guidelines. International Journal of Qualitative Methods, 19, 1 to 13. DOI: 10.1177/1609406919899220. https://journals.sagepub.com/doi/10.1177/1609406919899220
- Pi, X., Yang, Q., and Nguyen, C. (2026). LOGOS: LLM-driven end-to-end grounded theory development and schema induction for qualitative research. Submitted to ICLR 2026, OpenReview. https://openreview.net/forum?id=ZJn0TMTy8C
- Rong, H. H., Davis, J., and Rada-Orellana, M. (2025). Benchmarking large language models against qualitative coding and natural language processing in decoding public sentiment on urban upzoning. Urban Informatics, 4(1), article 17. DOI: 10.1007/s44212-025-00083-x. https://doi.org/10.1007/s44212-025-00083-x
- Uebellacker, T. (2024). AcademiaOS: automating grounded theory development in qualitative research with large language models. arXiv:2403.08844. https://arxiv.org/abs/2403.08844
- Vikan, M., Aryan, R., Kannelonning, M. S., Riegler, M. A., and Danielsen, S. O. (2025). Reflecting on LLM support in reflexive thematic analysis: an exploratory study. Qualitative Health Research. DOI: 10.1177/10497323251365211. https://pubmed.ncbi.nlm.nih.gov/40916991/
- Wenger, E., Trayner, B., and de Laat, M. (2011). Promoting and assessing value creation in communities and networks: a conceptual framework. Ruud de Moor Centrum, Open Universiteit, Rapport 18. ttps://www.betterevaluation.org/sites/default/files/Wenger_Trayner_DeLaat_Value_creation.pdf
- Wong, L. Z., Bhattacharya, P., Loh, S. B., Oh, H. S., Juraimi, S. A., Anant, N., et al. (2025). Utilizing large language models to conduct thematic analysis: a case study on focus group transcripts.
- SSRN Working Paper 5167505. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5167505