We present the AI Research Handbook — A Professional LLM Master, a 573-page technical reference for designing, training, evaluating, and operating domain-specific large language models inside Indonesian regulated institutions. Most published guidance on LLM development assumes an English-language context with no supervising regulator, no internal auditor examining the decision trail, and no obligation to keep data inside a named jurisdiction. This work targets the opposite setting: banks under Otoritas Jasa Keuangan (OJK) supervision, mining operators bound by good-mining-practice rules, and electronic system providers subject to Indonesia’s Personal Data Protection Law (UU 27/2022). In that setting the question “is the model good” is never sufficient; the answerable question is good by which measure, evidenced by what, approved by whom, and traceable how far. The handbook is therefore evaluation-first: it treats the evaluation harness as a versioned product with a named owner, an SLA, and its own regression tests, rather than as a script executed shortly before release. Its principal contributions are a four-axis failure diagnosis that maps observed defects to the correct intervention before budget is committed; a mandatory intervention ordering (CPT → SFT → DPO → RAG) with cost formulas rather than quoted figures; a seven-stage lifecycle in which each stage carries its own dataset, primary metric, decision owner, and output artefact; a three-layer separation of model, system, and user-experience evaluation; a five-gate release process (G0–G4) with an intentionally short list of non-waivable blockers; and a control mapping that lets one body of evidence serve UU PDP, POJK, SEOJK, NIST AI RMF 1.0, and ISO/IEC 42001 simultaneously. Nine parts, forty-six chapters, six appendices, and eighty-four figures are organised around three recurring use cases — banking risk assessment, mining safety documentation, and customer service automation — chosen because the correct answer for one is demonstrably the wrong answer for another. A closing part revises the hardware assumptions of the earlier chapters for the 2026 small-model regime, in which 27–32B open-weight models run on a single 32 GB consumer GPU. All factual data carry a stated cut-off of 25 August 2026.
The practical literature on building large language model programmes is abundant, but it is written almost entirely for one context: English-language products, shipped by teams that answer to a product manager rather than to a risk committee, in jurisdictions where the location of a training corpus is a cost decision rather than a legal one. Practitioners in Indonesian regulated institutions inherit that literature and discover, usually late, that it omits the constraints that actually govern their release schedule.
Three omissions recur. First, evaluation is treated as a gate to pass rather than as a system to maintain, so the numbers that justify a release cannot be reproduced six months later when an auditor asks for them. Second, language is treated as a solved problem, so the specific failure mode of Indonesian formal register, code-mixing, and regional languages is diagnosed as a general capability deficit and answered with the wrong — and far more expensive — intervention. Third, compliance is treated as a document produced after the technical work, rather than as a set of design constraints that determine which architectures are admissible at all.
This handbook is written for the reader who owns those constraints: a Head of AI, Chief AI Scientist, Principal AI Architect, or AI Research Lead who is accountable for technical decisions and must defend them before a risk committee. It assumes the reader understands transformer architecture and has trained models before; it does not explain gradient descent. Three secondary audiences are addressed explicitly — evaluation leads building a benchmark function from nothing, technology risk managers and internal auditors who need to know what may legitimately be demanded of an LLM programme, and senior practitioners preparing for the role.
The principal contributions of this work are:
The handbook comprises nine parts, forty-six chapters, and six appendices across 573 pages, illustrated by eighty-four figures. Every chapter follows an identical internal structure — executive summary, body with directly usable tables, Indonesian context, common traps, checklist, and key points — so that chapters can be read out of order. The figures are designed to be legible without the surrounding prose, and the List of Figures functions as a second index: a reader searching for a decision matrix or a compliance mapping will often reach it faster through the figure caption than through the chapter title.
| Part | Title | Chapters | Central question |
|---|---|---|---|
| I | Foundations, Role, and Strategy | 1–6 | Is this problem worth solving with an LLM, and who owns it? |
| II | Data: Sourcing, Curation, Governance | 7–12 | Is this corpus lawful, sufficient, and traceable? |
| III | Training, Adaptation, Alignment | 13–18 | Which intervention, in which order, at what cost? |
| IV | Evaluation Strategy and Benchmark Development | 19–30 | How is quality proven rather than asserted? |
| V | Production and Continuous Improvement | 31–34 | Does what we measured still hold today? |
| VI | Responsible AI, Security, and Compliance | 35–37 | What is the blast radius, and can it be explained? |
| VII | Leadership, Ecosystem, and Monetisation | 38–41 | Who staffs this, and what is the unit economics? |
| VIII | Case Studies and Closing | 42–43 | What does the whole path look like end to end? |
| IX | Small Models and Local Compute | 44–46 | What changed in 2026, and what must be recomputed? |
Twelve of the forty-six chapters — Part IV in its entirety — are devoted to evaluation. This proportion is deliberate and is defended in the text on a specific ground: in regulated industry the ability to demonstrate quality is a precondition for being permitted to release at all, and an organisation that cannot evaluate its models with discipline should not be training them.
To keep the discussion concrete, three use cases recur throughout the book under fixed abbreviations. They were selected because each demands a different technical path, and the book repeatedly demonstrates that the correct answer for one is the wrong answer for another.
| Code | Use case | Defining characteristics |
|---|---|---|
| BRA | Banking Risk Assessment | Credit-risk and compliance analyst assistant. High cost of error, mandatory clause-level citation, tight supervision, low volume. |
| MSD | Mining Safety Documentation | Safety documentation drafting and retrieval. Life-safety consequence, specialised terminology, intermittent field connectivity. |
| CSA | Customer Service Automation | Multichannel Indonesian customer service. Very high volume, cost and latency sensitivity, colloquial and regional language. |
Organisations named in the worked examples — Bank Nusantara Sejahtera, PT Bara Katulistiwa, and Pusat AI Nusantara — are fictitious, and the figures attached to them are illustrative and marked as such in the text.
The handbook’s central methodological claim is that intervention selection must follow, never precede, a labelled distribution of real failures. The prescribed procedure is to draw 50–100 genuine failure cases from production logs and label each on exactly one of four axes. A case that cannot be labelled indicates that the definition of correctness is not yet clear — a specification problem, not a model problem.
| Axis | Failure | Diagnostic test |
|---|---|---|
| 1 — Knowledge | The model lacks a required fact: an internal memo number, this quarter’s credit-ratio threshold, the current SOP revision. | Paste the source document into the prompt manually. If the answer becomes correct, this is a retrieval problem. |
| 2 — Format and style | Facts are correct but the output shape is wrong: invalid JSON, off-template structure, register too informal, citation misplaced. | Do three few-shot examples fix it consistently? If yes, it is cheap. If it improves but destabilises on 5–10% of cases, this is a strong fine-tuning candidate. |
| 3 — Capability | Reasoning, multi-step arithmetic, tool use, or long rule chains fail. Documents present, format clear, conclusion wrong. | The axis most often misdiagnosed as a knowledge problem. |
| 4 — Language | Quality degrades on formal Indonesian, code-mixed registers, or regional languages: grammatical sentences that read as literal translations from English, local terminology replaced by generic equivalents, inflated tokens per sentence. | The only axis that routinely justifies continued pretraining. |
The text is explicit about the consequence of skipping this step: if 70% of failures sit on axis 1, a six-week fine-tuning programme will repair 30% of the problem with 100% of the budget.
Knowledge that changes faster than the training cycle must not be stored in weights. The handbook frames this as an operational consequence rather than an architectural preference: updating one paragraph in a knowledge base takes minutes, whereas updating one paragraph’s influence on weights means retraining and re-running the entire release gate sequence. In Indonesian banking a second and more binding reason applies — UU 27/2022 grants data subjects a right to erasure, and personal data absorbed into weights creates an obligation that cannot practically be discharged without retraining. The rule applied consistently throughout the book is that personal data and high-volatility facts belong in the retrieval layer, while patterns, terminology, and style may enter the weights.
Before concluding that a knowledge deficit exists, three checks are prescribed on the retrieval pipeline: whether the correct document is indexed at all, whether it enters the top-k (measured as recall@k on a labelled query set, with recall below roughly 0.8 indicating a chunking, embedding, or lexical-search defect), and whether a correct document already in context was ignored — which reclassifies the case to axis 3.
Relative cost is expressed with structured prompting as the 1× baseline. Calendar figures assume a team of three to five and are presented as orders of magnitude to be recomputed against local capacity.
| Dominant symptom | Diagnosis | Intervention | Relative cost | Time |
|---|---|---|---|---|
| Facts wrong, correct when document pasted | Knowledge | RAG / improve retrieval | 3–8× | 4–8 weeks |
| Document indexed but not in top-k | Retrieval | Hybrid search + reranking | 2–4× | 2–4 weeks |
| JSON/template repeatedly invalid | Format | Constrained decoding → SFT | 1× → 30× | 1 week → 6 weeks |
| Internal terminology misread | Terminology | SFT (+ glossary retrieval) | 30× | 6–10 weeks |
| Multi-step reasoning fails | Capability | Larger model / task decomposition | 2–10× | 2–6 weeks |
| Tone and refusal inconsistent | Style / preference | SFT then DPO or KTO | 40× | 8–12 weeks |
| Quality drops on regional language | Language | CPT then SFT | 200–1,000× | 4–7 months |
| Cost per request too high | Economics | Distillation to a smaller model | 25× once | 6–10 weeks |
The matrix is read top to bottom: a lower row is never the right answer while an upper row remains unexecuted and unmeasured.
The ordering is not stylistic. CPT precedes SFT because it shifts the weight distribution broadly and would dilute SFT results if run afterwards. DPO after SFT is pointless without it, because preference tuning refines an existing policy rather than creating one. RAG sits outside the training chain as a system component, and precisely for that reason must be built first: models produced by CPT and SFT have to be trained on the same prompt schema that production will use, including how documents are injected into context.
Two practical consequences follow. The production prompt schema must be fixed before SFT data is assembled, since changing it afterwards means rebuilding the dataset. And re-evaluation after every stage must run on the same harness; a stage that does not improve the metric is reverted, not stacked.
A recurring editorial principle of the handbook is to present formulas rather than quotable numbers, so that estimates remain valid as prices move. Relative workload across the five intervention paths spans three orders of magnitude, from roughly 1 person-hour for structured prompting to roughly 900 person-hours for continued pretraining.
For CPT the compute estimate derives from FLOPs ≈ 6 × N × D. For an 8B model on 1 billion tokens this yields 6 × 8×109 × 1×109 = 4.8×1019 FLOPs. An H100 peaks near 990 TFLOPs in BF16 dense; at 40% MFU the effective rate is about 396 TFLOPs, giving roughly 33.7 GPU-hours per billion tokens. The book is explicit that this is an arithmetic derivation rather than a cited figure, and instructs the reader to measure actual MFU before committing a budget.
For SFT and PEFT the dominant cost is annotation, not compute — QLoRA on a 7B model for three hours is a trivial compute line item, and the three-order-of-magnitude gap against full fine-tuning is the stated reason not to attempt full FT without evaluation evidence that LoRA is insufficient. Memory follows VRAM ≈ weights + gradients + optimizer state + activations, quoted as a range because vendor sources differ by up to roughly 30%. RAG cost is engineering and indexing rather than training, dominated by source curation, embedding, and the labelled query set needed to measure recall@k. Prompting cost is recurring inference: an 800-token system prompt called a million times a month is not free, and must be computed at prevailing input rates with cached-input and batch discounts applied.
The failure pattern the handbook sets out to eliminate is familiar: an engineer writes eval.py to compare two checkpoints, the numbers are used in a meeting, and the script decays on a branch that is never merged. Three months later the same question returns, the script is re-run, and the number differs — because the dataset changed, the prompt was improved, or the vendor updated the judge model. Nobody can explain why.
Treating evaluation as a production component implies five concrete obligations:
The next most common error is using one evaluation suite for every stage. Each stage answers a different question, uses a different dataset, and is decided by a different person.
| Stage | Core question | Dataset | Primary metric | Decider | Artefact |
|---|---|---|---|---|---|
| Data curation | Fit to train on? | 2,000-doc corpus sample | Coverage, dedup, PII, contamination | Data Steward | Corpus Card |
| CPT | Gains without forgetting? | Held-out domain + general regression | Perplexity, regression delta | Training Lead | Training Report |
| SFT | Follows instructions? | Sealed test set | Format and instruction compliance | Training Lead | Eval Report v-N |
| Alignment | Safe without being rigid? | Hazard suite + legitimate suite | Hazard compliance, over-refusal | Model Risk Committee | Safety Report |
| Integration / RAG | Does the system work? | Document-grounded QA set | Groundedness, retrieval recall | System Architect | System Eval Report |
| Pre-release | Fit to face real users? | Full suite + adversarial | All gates G0–G4 | Model Risk Committee | Release Evidence Pack |
| Production | Does it still hold today? | Live traffic sample | Drift, escalation, complaints | Product Owner | Monthly Model Review |
Two rules are singled out. At the CPT stage, held-out domain perplexity and a general-capability regression suite must run side by side to detect catastrophic forgetting — never the first without the second. At the alignment stage, hazard compliance (which must rise) and over-refusal on legitimate requests (which must not) are both mandatory, because measuring only the first is the easiest way to produce a model that is safe and useless.
The handbook identifies layer confusion as the most expensive error in enterprise LLM programmes.
The book’s worked illustration is a CSA programme in which escalation to human agents rose sharply after release. The reflex response — collect more instruction data and re-run SFT — consumed three weeks and real GPU cost, and the metrics did not improve. The actual cause, discoverable in four hours once the layers were separated, was that retrieval was returning a stale version of the administrative-fee document because the index had not been refreshed. The model was never at fault; the fix was one line in the index scheduler. The cheap preventive test is stated as a working rule: before blaming the model, re-run the failing case with a manually inserted gold context. If the model then answers correctly, the defect lies in retrieval or context construction, not in the weights.
The team that builds a model must not be the only party that judges it. The handbook grounds this in incentive structure rather than suspicion of individual integrity: a person who has spent weeks preparing instruction data will, without any bad intent, choose thresholds and test cases that present their work favourably. Indonesian banks already have the vocabulary for the remedy — three lines of defense — and the recommendation is to apply it directly rather than invent a parallel structure.
The first line builds: data engineering, training, system architecture, running development evaluations as often as it likes on datasets it manages itself. The second line is model validation and risk: it holds a sealed test set that the first line cannot access, runs independent evaluation, and holds veto power over release. Banks that already operate a Model Risk Management function for credit scoring are advised to place LLMs inside it rather than create a new silo. The third line is internal audit, which does not run evaluations but tests whether the evaluation process ran as documented and whether the evidence is complete.
The operational consequence is accepted from the outset: a sealed test set must be genuinely sealed. It is stored in a repository with separate access control, never sent through chat channels, and individual failing cases are never shown to the training team — only failure-category summaries. Once a specific example leaks it will enter the next training set and the test set is dead. Partial rotation of the sealed set each release cycle limits the damage when leakage occurs anyway. On reporting lines, the head of evaluation should report to the Chief Risk Officer or equivalent rather than to a Head of AI whose KPI is feature delivery; where that is not organisationally possible, the compensating control is mandatory attendance of the risk function with veto rights at the Model Risk Committee.
The feasibility test posed in the text is concrete: an auditor asks in March 2027 why a given model version, approved in September 2026, scored 0.91 on groundedness. Can that evaluation be re-run today and yield the same number? If the answer is “approximately”, there is no harness. Seven things must be recorded on every evaluation execution, and none may be implicit: dataset version (content hash, not filename — a dataset whose typos were “merely fixed” is a new dataset); task-definition version, including rubric and annotator instructions; prompt version, including system prompt, template, and few-shot examples, since a single word can shift scores by several points; model and decode configuration, covering checkpoint name, revision hash, quantisation, temperature, top-p, max tokens, and seed; judge-model version, pinned to a specific revision; harness code commit hash; and execution environment, including serving library version, driver version, and GPU type, because attention-kernel differences between versions can change outputs at identical seeds.
A notable governance recommendation follows from the judge-model dependency: where numbers will enter release evidence, prefer open-weight judges that the organisation can store itself, because a closed model may change under the programme’s feet without notice and exact reproduction then becomes impossible. The recommended repository layout keeps a versioned task registry in git — one YAML per task, carrying dataset URI and SHA-256, sealed flag, segment definitions, prompt template hash, generation parameters, scoring configuration with pinned judge revision and rubric hash, pass and block thresholds, and an explicit breaks_comparability_with provenance field. Platform tools may serve as a storage layer beneath this, but the registry itself stays in version control so that it is subject to code review and an approval trail.
The question that always arrives at the investment committee is how much evaluation should cost. The handbook offers a defensible starting point — 15–25% of total LLM programme budget — explicitly labelled a practitioner heuristic rather than an industry benchmark, covering human annotation, evaluation compute, harness engineering, and domain-expert time. High-risk use cases such as BRA sit at or above the upper bound. The accompanying formula makes the components explicit:
Annual_eval_cost =
(N_annotation_cases × Hours_per_case × Domain_expert_rate)
+ (N_executions_annual × N_cases × Tokens_per_case × Judge_token_price)
+ (Harness_FTE × FTE_cost)
+ (GPU_eval_hours × GPU_hourly_price)
The component most often underestimated is not compute but domain-expert time — and it is the one that cannot be automated away, because no model yet knows whether a given reading of an internal credit-policy clause is correct. The argument that works before a board is not an appeal to best practice but an asymmetry of cost: a single incident forcing suspension of a customer-facing service, or a regulatory finding of misleading product information on a digital channel, exceeds an annual evaluation budget.
Release approval is structured as five sequential gates, each with measurable pass criteria, mandatory evidence, and a single named decision owner. Gates are tiered rather than uniform: a system touching customers or credit decisions runs all five in full, while an internal assistant without access to personal data runs an abbreviated version in which G3 and G4 remain mandatory. Two procedural rules apply throughout — evidence packs circulate a minimum of three working days before the meeting, so that a committee seeing numbers for the first time on a slide is receiving rather than deciding; and mid-cycle threshold changes must pass through the same procedure as a waiver, because a quiet threshold adjustment is the hardest form of gate erosion to detect.
| Gate | Measurable pass criteria | Mandatory evidence | Decider | Waivable |
|---|---|---|---|---|
| G0 — Data | Zero contamination; residual PII below threshold; clean licensing per source | Corpus Card, contamination report, legal opinion, RoPA record | Data Steward + DPO | Partial |
| G1 — Base model | Primary metric passes on sealed set; general-capability regression within bounds; no significant segment below block threshold; canaries pass | Versioned Eval Report (dataset, prompt, judge, harness hashes); Model Card; Training Report | Head of Model Validation (second line) | Partial |
| G2 — Integrated system | End-to-end suite passes; retrieval recall and groundedness reported separately; 100% format compliance; guardrails fire on all positives and not beyond bounds on negatives; fallback tested; p95/p99 latency and cost per interaction under threshold | System Eval Report, load-test results, fallback matrix, hashed system configuration | System Architect + Product Owner | Partial |
| G3 — Security and compliance | Zero PII leakage on the privacy suite; red teaming mapped to OWASP Top 10 for LLM Applications 2026; all critical findings closed or mitigated with verified controls; DPIA complete; cross-region inference documented explicitly | Red-team report, privacy-test report, DPIA, compliance opinion, cross-region data-flow map | CISO + Head of Compliance (independent veto each) | No |
| G4 — Operational | Observability active and verified; tracing and logging free of raw PII; alerts with defined thresholds and recipients; rollback rehearsed in staging with recorded timing; incident runbook complete, including SEOJK 29/2022 breach-notification paths | Dashboard captures, alert configuration, rollback test record, runbook, staged release plan | Head of IT Operations + Product Owner | Partial |
The list of non-waivable conditions is deliberately short, on the stated reasoning that its strength varies inversely with its length: if twenty items are absolute, the organisation will learn that “absolute” is negotiable.
The stated test for any proposed fifth item is whether it truly cannot be mitigated by a compensating control. If it can, it is a conditional-waiver candidate rather than an absolute blocker.
Waivers will occur whether or not they are governed; ungoverned, they occur through corridor conversations that leave no trace. The handbook therefore specifies the mechanism: G1 and G2 waivers require full-quorum Model Risk Committee approval, G0 requires the DPO’s signature, G4 requires the Head of IT Operations, and G3 cannot be waived by anyone including a director — a rule framed as protecting executives from improper pressure. Every waiver carries an expiry of at most ninety days, and an unresolved waiver at expiry triggers automatic service suspension rather than automatic extension; that default direction is what distinguishes programmes that clear their quality debt from those that accumulate it. A waiver without a compensating control is negligence rather than a waiver; the customary forms are mandatory human-in-the-loop for affected request classes, restriction to internal channels, topic-scope limitation, reduced traffic share, or increased production-evaluation sampling frequency. A central waiver registry records gate, criterion, actual versus threshold value, business justification, compensating control, approver, expiry, and resolution status. It is the first document an internal auditor requests, and the count of active waivers is described as a more honest indicator of programme health than any quality metric.
The Model Risk Committee is chaired from the second line — the Head of Model Validation — not from the building team, with the Chief Risk Officer, Head of Compliance, CISO, DPO, Product Owner, and Head of AI in attendance. Quorum requires the risk, compliance, and security functions; the absence of any one postpones the meeting rather than producing a conditional approval. Rubber-stamping is countered by three devices: circulating the evidence pack at least three working days in advance, appointing a designated challenger for each release whose task is to prepare and deliver the argument against, and recording decisions with dissenting opinions attributed by name. The approval rate is itself monitored — a committee that approves 100% of submissions across four consecutive quarters is not functioning — and deferral is explicitly framed as a normal outcome rather than a failure of the proposing team.
Several chapters address problems that have no counterpart in English-language guidance.
On governance frameworks the handbook’s position is that the lifecycle structure need not be re-authored for regulators, only mapped. NIST AI RMF 1.0 aligns directly: GOVERN corresponds to evaluation-function independence, team charter, and reporting lines; MAP to lifecycle stages 1 and 2; MEASURE to the harness and layers 1–3; MANAGE to stage 7 and the gate mechanism. The NIST AI 600-1 Generative AI Profile contributes twelve GenAI risk categories — confabulation, data privacy, and information supply-chain security among them — usable as a coverage checklist in which every category relevant to a use case must map to at least one evaluation task. ISO/IEC 42001:2023, structured on Annex SL, attaches to the ISO/IEC 27001:2022 systems most Indonesian banks already hold: the task registry, versioned evaluation reports, and approval minutes constitute documented information; the seven-stage cycle is part of the AI system life cycle; and second-line independence satisfies the roles-and-responsibilities clause.
Part IX was added late in the writing and does not replace earlier chapters; it updates the figures, capacity formulas, and deployment patterns in chapters that still assume data-centre GPUs. Four concurrent changes are identified: quality per parameter rose sharply, with Qwen3 trained on roughly 36 trillion tokens covering 119 languages and dialects and Gemma 4 claiming 140+ languages; licensing loosened precisely at the sizes that matter, with Gemma 4 released under Apache 2.0; Meta ceased shipping new open-weight Llama models through 2026, which matters disproportionately for Indonesia because most of the Indonesian model lineage — Sahabat-AI, SEA-LION v2/v3, Komodo, Cendol — rests on a Llama base, so the established CPT path lost its supplier and the successors are the Qwen and Gemma families; and 32 GB consumer GPUs became a legitimate serving target for 27–32B models at Q4 with long context.
| Model | Release | Parameters | MMLU-Pro | GPQA Diamond |
|---|---|---|---|---|
| Llama 3.1 70B Instruct | Jul 2024 | 70B dense | 66.4 | 46.7 |
| Gemma 4 31B | Mar/Apr 2026 | 30.7B dense | 85.2 | 84.3 |
| Qwen 3.5 27B | Feb 2026 | 27B dense | 86.1 | 85.5 |
| Qwen 3.5 9B | Mar 2026 | 9B dense | 82.5 | 81.7 |
| Gemma 4 26B-A4B | Mar 2026 | 26B total / 4B active | 82.6 | 82.3 |
| Gemma 4 E4B | Mar 2026 | 4B effective | 69.4 | 58.6 |
The handbook presents this table only together with four caveats that must accompany it every time it is shown: there is no official head-to-head comparison, and figures are combined from model cards that may differ in shot count, chain-of-thought usage, and prompt variants; some 2026 benchmarks did not exist in the Llama 3.1 era, and contamination or benchmark overfitting is a real risk; benchmark scores measure capability on the distribution of public problems, not on the distribution of the reader’s work; and static benchmarks lean toward Qwen while Arena chat preference leans toward Gemma, so a product judged by humans should weight preference signal while one judged on structured correctness should weight static benchmarks. The operational conclusion that survives all four is narrow and specific: drop one size class relative to 2024 habits, then prove it with an internal benchmark — never the reverse order.
| Class | Example models | Context | Suitable work | Minimum device |
|---|---|---|---|---|
| 2–4B | Gemma 4 E2B/E4B, Qwen3.5-2B/4B, Ministral 3 3B, Phi-4-mini 3.8B | 64K–256K | Classification, routing, patterned extraction, edge | 8 GB VRAM / phone |
| 8–9B | Qwen3.5-9B, Ministral 3 8B, Gemma 2 9B CPT Sahabat-AI, Apertus-SEA-LION-v4-8B | 8K–262K | General-purpose worker: RAG, summarisation, drafting | 16 GB VRAM |
| 20–35B | Qwen3.8-27B, Gemma 4 31B, Gemma-SEA-LION-v4-27B, Devstral Small 2 24B | 128K–262K | 70B-class replacement: reasoning, agents, audit | 32 GB VRAM |
The section the handbook flags as most often skipped and most often responsible for sizing errors concerns attention architecture. In 2024 models, KV cache could be treated as a simple function of layer count and KV head count. In 2026 models, attention architectures differ between models in ways that change memory requirements several-fold at identical parameter counts. Gemma 4 31B uses 60 layers with a 1,024-token sliding window and full attention only every sixth layer — 10 global layers, 50 local — so its cache does not grow with context except on the global layers, shrinking KV cache at 32K context roughly fivefold against an equivalent full-attention architecture. Qwen3.8-27B uses 64 layers with Gated DeltaNet and full attention every fourth layer, yielding 64 KiB per token against Gemma 4 31B’s 160 KiB — about 2.5× less, despite similar model size. The operative instruction is to choose a model by reading its attention configuration, not only its parameter count: two candidates that look equivalent on a specification sheet can differ by 10 GB of VRAM on a long-context workload.
The handbook reports the position without softening either side. Sahabat-AI has not released a new small model since 2025; the 8B (llama3-8b-cpt-sahabatai-v1-instruct) and 9B (gemma2-9b-cpt-sahabatai-v1-instruct) variants remain at v1, and the 9B variant carries an 8,192-token context — a hard constraint for RAG with several document chunks plus a system prompt plus conversation history, not one that aggressive chunking can evade without sacrificing groundedness. Its model card states that the model is not safety-aligned, so developers must add their own guardrail layer. Reported scores are SEA-HELM Indonesia 64.15%, Javanese 64.44%, Sundanese 54.91%, IndoMMLU 62.6%. Meanwhile the top of the SEA-HELM open-instruct tier at or below 200B is held by general models: Indonesian is led by Qwen 3 VL 32B at 68.41, followed by Qwen 3 Next 80B MoE at 67.11; no Western model appears in the top five for Indonesian at this tier.
The correct reading is set out explicitly. This does not make Sahabat-AI useless — it was trained on GPU Merdeka Lintasarta with all data and compute inside Indonesian territory, and that sovereignty claim is a real regulatory asset that Qwen does not carry. It means that if the operative criterion is measured Indonesian-language quality, the 27–32B Qwen or SEA-LION class beats the local 9B option and still fits on a single 32 GB GPU at Q4; and if the criterion is domestic training provenance, the choice narrows sharply and the price is an 8,192-token context plus the obligation to build a safety layer. A third path, frequently overlooked, is to perform Indonesian CPT on an Apache 2.0 9B or 27B base following the Sahabat-AI and SEA-LION pattern — a budget now within reach of a single workstation.
Roughly half the chapters close with a directly reusable artefact rather than a summary, and six appendices consolidate them: a glossary (A); governance document templates including AI Use Case Charter, Data Source Register, DPIA, Data Contract, Release Evidence Pack, model approval minutes, AI incident post-mortem, and improvement recommendation record (B); evaluation specifications and model cards, including an eval-harness specification, a reference evaluation task catalogue, LLM judge prompts, and a six-dimension human-evaluation rubric (C); a nine-domain technical competency matrix with scoring rubric, thirty evidence-based interview questions, and a twelve-month development plan (D); a reference list with traceability notes marking which items require re-verification (E); and the visual system and design tokens used to build all eighty-four figures, including the contrast validator (F).
Chapter 43 functions as a re-reading map, assembling the material into an eighteen-month execution plan and indicating which chapters are relevant at each stage. Three reading paths are offered: a builder path through Parts I–III in order then Part IV; an evaluator path through Part IV entire, then Chapters 27–28, then Part V; and a leader path through Chapters 1–3, Parts VI–VII, then Chapters 42–43.
The handbook states its own limits, and they are worth reproducing rather than smoothing over.
The AI Research Handbook argues that in regulated industry the binding constraint on an LLM programme is rarely model quality and almost always the ability to demonstrate quality: to name the measure, produce the evidence, identify the approver, and trace the decision. Its structural response is to treat evaluation as a long-lived product with an owner, a version, a backlog, and an SLA; to force diagnosis ahead of intervention so that budget follows the actual distribution of failures; to separate model, system, and user-experience evaluation so that defects are repaired at the layer where they live; and to make release contingent on gates whose criteria are measurable, whose evidence is mandatory, and whose small set of absolute blockers is short enough to stay absolute.
Two of its positions are likely to remain valid longer than its numbers. The first is that the most expensive error in enterprise LLM work is diagnosing at the wrong layer — retraining a model to fix a stale index. The second is that an organisation unable to evaluate its models with discipline should not be training them. The 2026 revision in Part IX changes what hardware the work requires and which model families are viable; it does not change either conclusion. As tooling and model supply continue to shift, the durable content of this book is the decision structure — the axes, the ordering, the layers, the gates, and the evidence — rather than the versions it happens to name.