Beyond the Heat Map: Calibrated Risk for Title and Survey Diligence

01How diligence reports summarise risk

Site control has become a gate. Under PJM's reformed interconnection process, a project now has to show proof of site control before it is studied at all ^[1]. When PJM opened its first reformed cycle, Google-backed Tapestry processed the 811 applications' 4,581 site-control documents into 2,328 agreement bundles and ran 9,312 compliance checks on contract term, chain-of-title continuity, exclusivity and acreage, returning page-level citations to the PJM engineers who made the final determinations ^[2].

A title commitment lists, in Schedule B-I, the requirements to be met before a policy issues, and in Schedule B-II, the exceptions the policy will not cover: easements, rights of way, mineral reservations, leases, restrictions. An ALTA/NSPS Land Title Survey locates what can be located on the ground and annotates the rest. A project layout places arrays, roads, collector lines, a substation and a fence on top. Diligence reads all three together to establish which of these exceptions can hurt this project, and by how much.

The answer is usually delivered as a risk matrix. Each exception is given a likelihood band and a consequence band, typically on five-point scales; the product of the band indices becomes a score, and the score a colour. Both Cox and Thomas et al. define the quantity a matrix is meant to approximate in the same way, as expected loss ^[3,4]:

Equation (2) is a different function from equation (1), and the failures documented below arise from that difference.

02What the evidence says about matrices

Cox set out the conditions a matrix must meet to rank risks coherently. Weak consistency: every risk in the top colour must exceed every risk in the bottom colour. Betweenness: a small change in risk should not jump a risk from the lowest colour to the highest without passing through the middle. Consistent colouring: cells of one colour should hold risks of roughly equal size. Few colourings satisfy all three, and even those that do leave typical matrices able to “correctly and unambiguously compare only a small fraction (e.g., less than 10%)” of randomly selected pairs of hazards ^[3]. Cox names the consequences: identical ratings for quantitatively very different risks (range compression), higher ratings for smaller risks, and, where frequency and severity are negatively correlated, decisions he describes as worse than random.

Thomas, Bratvold and Bickel reviewed 30 published oil-and-gas papers that recommend and demonstrate matrices and added three flaws ^[4]. Rankings are arbitrary: five of the 30 papers score in descending order, so what ranks first in one ranks last in another. Rankings are unstable: varying the category boundaries with a single multiplier changes the priority order. And the matrix distorts distance non-uniformly, which they measure with a lie factor; a top consequence band of “above $20 million” treats $20 million and $50 billion identically. Ball and Watt found that assessors given the same hazard may assign vastly different ratings ^[5].

The largest and most recent study, by Meyer, compared 1,144 paired qualitative and quantitative assessments of the same risks from 91 mining capital projects across 43 organisations ^[6]. Probability ratings agreed at the ordinal level; impact ratings and composite scores did not. Quantitative reassessment changed the escalation band of 73.3% of risks, 59.4% moved to a lower band, and 37.2% moved two or more bands.

Six exceptions, drawn at true scale

The failures are easiest to see when the matrix is drawn on the quantities it stands for. Take a common 5 × 5 design: likelihood bands with edges at 5%, 20%, 50% and 80%; consequence bands at $25k, $100k, $500k and $2M. Six hypothetical exceptions on one site are scored below. The probabilities and consequences are invented for illustration; the scores and expected losses are computed from them.

• –Reversal. B scores 16 and leads the report. A scores 10 and ranks third, yet A's expected loss is 1.6 times B's.
• –Compression. C, D and F share a score of 6, while their expected losses span a factor of fourteen.
• –Boundary splits. C and E differ in expected loss by 1.4×, but sit on either side of two band edges, so E's score is double C's.

03What the 2026 ALTA/NSPS standard makes checkable

The Minimum Standard Detail Requirements for ALTA/NSPS Land Title Surveys took effect on February 23, 2026 ^[8]. Several changes matter for energy sites. Fieldwork must now be performed “using practices generally recognized as acceptable by the surveying profession”, in place of “on the ground”; the new wording admits drones, LiDAR and imagery. Surveyors must now obtain adjoining deeds themselves, and a complete copy of the most recent title commitment must be provided to them ^[9]. A new optional Table A item 20 requires potential encroachments and related conditions to be summarised in a table and located on the plat ^[8].

The most consequential clause for diligence is Section 6.C.ii. For every right of way, easement or other survey-related matter in the title evidence, the survey must give its record information, state whether it lies within or crosses the property, and add a note for each of the following conditions where present ^[8]:

1. 01its location is shown;
2. 02its location cannot be determined from the record document;
3. 03there was no observed evidence at the time of the fieldwork;
4. 04it is a blanket easement;
5. 05it is not on, does not touch, or does not affect the surveyed property;
6. 06it limits access to an otherwise abutting right of way;
7. 07the documents are illegible;
8. 08the surveyor has information that it may have been released or terminated.

Where the property is made of several parcels, the survey must also say which parcels each matter crosses or touches ^[8]. For a solar or wind site assembled from many parcels, this turns the survey's annotations into a structured record: for each Schedule B-II exception, one of eight conditions and a set of parcels. Together these form a label set, so the reconciliation between commitment and survey, which diligence teams do by eye, can be checked by machine, and a model's reading of the survey can be scored against the surveyor's.

04From exception to probability

The method we propose replaces the ordinal score with an estimate of equation (1) for each exception, made from evidence that can be checked. For an exception e with features xₑ, the method estimates how likely it is to draw an objection, a cure requirement or a redesign, and what that would cost:

Each step that produces xₑ leaves a record a reviewer can check:

1. 01Extract. Each requirement and exception is read from the commitment with the page and region it came from, and every recorded instrument it references.
2. 02Reconcile. Each exception is matched to the survey's Section 6.C.ii note and parcel list; a mismatch between the two documents is itself a finding.
3. 03Intersect. Plottable exceptions are intersected with the layout, by component: arrays, roads, collector routes, substation, fence.
4. 04Estimate. A calibrated model estimates the probability and consequence in equation (3), with an interval, from the features above.
5. 05Decide or abstain. Where the estimate cannot meet the tolerance agreed with the user, the item goes to a person, with the evidence attached.

Unlike a matrix, these estimates aggregate. A site's exposure is a sum of exceptions that may share parcels and causes, and can be simulated as a distribution: the chance that diligence findings delay financial close by more than thirty days, for example.

Reading is the weak link

CiteVQA scores document models only when both the answer and the cited region are correct; on 711 PDFs averaging 40.6 pages, the best of 20 models scored 76.0 and the best open-source model 22.5, with models often giving the right answer while citing the wrong place ^[10]. Across 58 parsing models, scores fell by 11.61 points on average from clean renders to real captures ^[11], and recorded instruments are often decades-old scans. On long multi-document legal cases, models are “more prone to omitting key information than hallucinating” ^[12]; in diligence, a missed exception is the expensive error. And on 3,396 legal claims, even models reaching 0.97 accuracy still marked unsupported claims as supported when given the full source ^[13].

Each extracted claim therefore has to be checked against the region it cites, and each field's error rate measured on held-out documents from the counties and decades it will meet in use.

05Knowing when not to answer

A diligence system has to know which of its answers it can stand behind. Conformal risk control gives a guarantee for that: a threshold λ̂ chosen on a calibration set bounds the expected loss on new items, under exchangeability, at a tolerance α the user chooses ^[14].

A 2026 result gives the exact price of that guarantee. If the base error rate of a step is μ and the tolerance is α < μ, any distribution-free method must abstain on at least a fixed share of items ^[15]:

The same work reports that heuristic abstention policies miss their stated risk targets by 7.5–12.5% ^[15]. Two further results shape how calibration should be done. A marginal guarantee does not hold group by group: under a shift in group composition, standard conformal risk control broke its budget in up to 47% of trials ^[16]. And calibration depends on the documents as much as on the model: in clinical extraction, the same models were underconfident on structured documents and overconfident on free text, and reached 90% coverage with 9–13% of items rejected ^[17]. Title evidence varies the same way by county, decade, instrument type and scan quality, so thresholds should be set per group, and the abstention rate reported with every result.

06Scoring the matrix against outcomes

A matrix can be kept for communication if it is rebuilt as a decision rule on calibrated probabilities. Taggart and Wilke do exactly this for weather warnings: each cell is a threshold p on the forecast probability of a severity class, and the matrix is scored against what happened with a consistent scoring function, one whose expected value is minimised only by honest, accurate probabilities ^[18]. For a threshold p, forecast f and outcome y:

With outcomes recorded (objections raised, cures required, endorsements issued, layouts changed), a rubric's matrix and a calibrated model can be scored on the same footing, on the same exceptions. Meyer's conclusion points the same way: separate the qualitative communication from the quantitative decision support ^[6]. We have not found public base rates for lender or title objections on energy sites. That dataset has to be built from diligence reports scored against what happened next.

07The report we are building towards

Prism runs the extraction step of this method today. Its ratings come from a fixed rule over five cited inputs; each row is sampled three times, extended to five and then seven on disagreement, and the median stands. A rating of 4 always goes to a person, and silence yields “Unknown” with an information request. A claim stands only if code finds its quote on the cited page or within two pages of it, and 3,280 dated public-source quotes verify at every cut-off they serve. Every flagged row in every frozen revision enters an outcome ledger. The report we are building towards shows each exception as a line of evidence: the page and region it was read from; the survey's Section 6.C.ii condition and the parcels it touches; the layout components it intersects; the estimated probability of an objection or cure, with its interval and the consequence range; and whether a person reviewed it. Items the system abstained on are listed first. The matrix remains, drawn on true scale and coloured by expected impact computed from the calibrated quantities, so a reader who wants the familiar picture still has it.

Title agencies will ask how such a system is governed. ALTA's Best Practices 5.0, published October 8, 2026, made vendor management a pillar of its own ^[19]; an AI system that reads title evidence is one of those vendors. A report that shows its sources, its abstentions and its error rates is one a title team can vet.

The ledger scores each frozen revision against what happened next. Calibration by rating category is reported once 30 or more outcome events have matured; until then, Prism reports descriptive counts.