Introducing Factbag

In Jev and the Art of Truth Maintenance, I made the case for using models to interpret evidence and explicit rules to combine the results. Factbag is my attempt to make that easy, ergonomic and powerful.

It brings together narrow judgments from decision models like TypeSafe's Jev, interpretations from LLMs, and data from our own systems in a knowledgebase. This post gets into how that works: what counts as true, how we write the rules, and how we keep track of the reasons for a decision.

Τί ἐστιν ἀλήθεια;

It all starts with what we define as true. Inspired by Belnap's four-valued logic, with some tweaks, we give a proposition one of four states: True, False, Conflicted or Unknown. At the evidence level, those distinguish support, denial, both, and neither.

While I'm sure we could get a lot of value from ordinary first-order logic, apparent contradictions are liable to slip in when we're working with a large corpus. Making our logic contradiction-tolerant – or paraconsistent – lets us retain conflicting evidence without allowing a contradiction to justify arbitrary conclusions.

That gives us some useful flexibility. We could build a subsequent step to resolve a contradiction by presenting the relevant sources to a model or a human operator. The contradiction itself becomes a signal telling us what to investigate. The same applies to Unknown assertions, for which we have no evidence either way.

We may also choose to preserve ambiguity and unknowns. Life is messy, and being overconfident can be expensive or dangerous.

This doesn't prevent a model from misreading a source. It gives us somewhere to record disagreements and gaps, and a way to correct an interpretation without asking the model to redo the entire decision.

Factbag orders the states as False, Unknown, Conflicted, True. AND takes the lowest state; OR takes the highest. That's one of the departures from Belnap's original logic, and it makes the treatment of missing and conflicting evidence explicit.

Here is what that looks like:

A small knowledgebase

Here's what that looks like in practice, with a classic example: ancestors and descendants.

We start by declaring our predicates. Leaf predicates are facts we read from documents or our own systems; derived predicates are whatever our rules conclude from them:

class Person {
  constructor(
    readonly id: string,
    readonly name: string,
  ) {}
}
 
// A leaf predicate: a fact we read from documents or our own systems.
const parentOf = predicate(
  "parent_of",
  { parent: Person, child: Person },
  { description: "The parent is a mother or father of the child." },
);
 
// A derived predicate: what our rules conclude.
const ancestorOf = predicate("ancestor_of", {
  ancestor: Person,
  descendant: Person,
});
 
const ancestor = variable("ancestor");
const descendant = variable("descendant");
const child = variable("child");
 
const kb = new KnowledgeBase();
// A parent is an ancestor of their child.
kb.rule(ancestorOf({ ancestor, descendant })).if(
  parentOf({ parent: ancestor, child: descendant }),
);
// And a parent of someone's ancestor is their ancestor too.
kb.rule(ancestorOf({ ancestor, descendant })).if(
  and(
    parentOf({ parent: ancestor, child }),
    ancestorOf({ ancestor: child, descendant }),
  ),
);

Then we add evidence. Some of it comes straight from our own systems; the rest, Jev reads out of documents:

const ada = new Person("ada", "Ada Lovell");
const ben = new Person("ben", "Ben Lovell");
const cleo = new Person("cleo", "Cleo Lovell");
kb.entities.add(ada);
kb.entities.add(ben);
kb.entities.add(cleo);
 
// From our own systems: the family register lists Ada as Ben's mother.
kb.asserts(parentOf({ parent: ada, child: ben }), { source: "register" });
 
// From documents: Jev reads the corpus and records what it finds.
const run = classifier(
  corpus,
  guard(kb).permit("assert", parentOf).permit("deny", parentOf),
);
for (const job of run) await job.execute();
 
query(run.kb, ancestorOf({ ancestor: ada, descendant: cleo })).value; // "true"

If a letter in the corpus mentions Ben's daughter Cleo, Jev records that Ben is Cleo's parent, and our rules conclude that Ada is Cleo's ancestor. Jev only ever answers a narrow question – does this passage say this person is that person's parent? – while the rules take care of grandparents, great-grandparents and so on, as far back as the evidence goes.

Leaf predicates are prompts

That's not to say there's no prompting at all. Let's zoom in on that leaf predicate. The metadata we attach to it can carry rather more than a one-line description:

const parentOf = predicate(
  "parent_of",
  { parent: Person, child: Person },
  {
    description: "The parent is a mother or father of the child.",
    instructions:
      "Biological and adoptive parents count. Step-parents, godparents " +
      "and guardians do not, and neither does a grandparent.",
    parameters: {
      parent: "The mother or father.",
      child: "Their son or daughter.",
    },
  },
);

When Jev classifies a passage, every question it's asked about parentOf carries this metadata alongside the text. The same metadata would serve just as well as a tool description for an LLM: the predicate's name becomes the tool's name, the description explains what it's for, and the parameter descriptions document its arguments.

So the leaf predicates are where we should spend our care. Each one should be narrow enough that a single passage can settle it, and explicit about what doesn't count. Its fields should have closed domains – here, people we've already registered – so that every question is a concrete yes, no or not mentioned. Jev won't classify open-ended fields like free-form strings at all.

Everything above the leaves is ordinary, deterministic rules. We can review them without reading a single prompt, and if we decide step-parents should count after all, we know exactly which definition to change.

Making room for disagreement

Applied to a practical case, a single layer of these four-valued rules looks like this:

Naturally, going through all this trouble for a single rule doesn't make much sense. The value I foresee is in composing decisions across multiple levels of a more complicated ruleset:

// evidence
kb.denies(prominentFunction({ person: dana, role: deputyMinister }));
// nothing recorded for "Left office < 12 months ago" — stays unknown
kb.asserts(familyMember({ person: dana, of: elena }));
kb.denies(familyMember({ person: dana, of: elena }));
kb.asserts(pep({ person: elena }));
kb.denies(jointBeneficialOwner({ person: dana, with: elena, entity: harbourHoldings }));
 
// four ways to be a PEP — their contributions combine by OR
kb.rule(pep({ person: P })).if(prominentFunction({ person: P, role: Role }));
kb.rule(pep({ person: P })).if(
  and(
    leftOffice({ person: P, role: Role, date: Departure }),
    gte(Departure, sub(today, 365 * dayMs)), // still inside the 12-month tail
  ),
);
kb.rule(pep({ person: P })).if(
  and(familyMember({ person: P, of: Q }), pep({ person: Q })), // recursive
);
kb.rule(pep({ person: P })).if(
  and(jointBeneficialOwner({ person: P, with: Q, entity: E }), pep({ person: Q })),
);
 
// the top-level criterion rule
kb.rule(enhancedDueDiligence({ person: P })).iff(pep({ person: P }));

The four states aren't limited to AND and OR, either. Sums and comparisons work the same way: when some records are disputed or unverified, a total becomes a range, and a threshold can land clearly on one side of it or somewhere in between.

In many cases, to invest in a private fund or a startup's funding round, you'd need to be what the SEC defines as an accredited investor – that is, you'd need an income over $200,000 in each of the last two years (or $300,000 with a spouse), a net worth over $1 million not counting your home, or one of a handful of professional licences. Written out as rules in our knowledgebase, the whole definition might look something like this:

kb.rule(accreditedInvestor({ person })).iff(
  or(
    meetsIncomeTest({ person }),
    meetsNetWorthTest({ person }),
    meetsLicenceTest({ person }),
  ),
);
 
// Income over $200k alone, or $300k with a spouse, in each of the last two
// years – and a reasonable expectation of the same this year.
kb.rule(meetsIncomeTest({ person })).iff(
  and(
    or(
      and(
        incomeOver200k({ person, year: year1 }),
        incomeOver200k({ person, year: year2 }),
      ),
      and(
        jointIncomeOver300k({ person, year: year1 }),
        jointIncomeOver300k({ person, year: year2 }),
      ),
    ),
    expectsSameIncome({ person }),
  ),
);
 
kb.rule(incomeOver200k({ person, year })).iff(
  letIn(
    total,
    sum(amount).where(sourceOfIncome({ person, year, source, amount })),
    gt(total, 200_000),
  ),
);
 
kb.rule(jointIncomeOver300k({ person, year })).iff(
  exists(
    spouse,
    and(
      spouseOf({ person, spouse }),
      letIn(
        ownIncome,
        sum(amount).where(sourceOfIncome({ person, year, source, amount })),
        letIn(
          spouseIncome,
          sum(amount).where(
            sourceOfIncome({ person: spouse, year, source, amount }),
          ),
          gt(add(ownIncome, spouseIncome), 300_000),
        ),
      ),
    ),
  ),
);
 
// … plus the net worth and licence tests.

For illustration, let's focus on one specific bit: whether someone's income cleared $200,000 in a given year.

This also shows why it makes sense to have Unknown and Conflicted states: they show an agent or a human what the most effective use of their time would be – tracking down more information, or reviewing what we've already got.

Keeping the reasons

Our four-valued result isn't the only thing passed through these layers. We're also passing along source references and other metadata, so that a conclusion can be traced to the specific evidence and rules that produced it.

Jev also returns probability estimates, which we can retain alongside individual judgments and combine in interesting ways, carrying a probability all the way through to our eventual conclusion. How that actually works gets a little involved – two judgments may depend on the same evidence, for one – but you'll see what it could look like below.

A query on the knowledgebase should therefore return more than an answer. It should show how that answer was reached, which evidence supports or contradicts it, and what remains unknown. That gives us somewhere specific to look when we want to resolve an ambiguity or challenge a decision.

Here's what that looks like for Dana, from the due diligence example above:

{
  "claim": "enhanced_due_diligence(dana) = conflicted",
  "probability": 0.1,
  "because": "enhanced_due_diligence(P) iff pep(P), with P = dana",
  "given": [
    {
      "claim": "pep(dana) = conflicted",
      "probability": 0.1,
      "because": "pep(P) if and(family_member(P, Q), pep(Q)), with P = dana, Q = elena",
      "given": [
        {
          "claim": "family_member(dana, elena) = conflicted",
          "probability": 0.1,
          "because": [
            {
              "claim": "family_member(dana, elena) = true",
              "source": "kyc-questionnaire-dana#1840-2112",
              "probability": 0.7
            },
            {
              "claim": "family_member(dana, elena) = false",
              "source": "birth-certificate-dana#0-486",
              "probability": 0.95
            }
          ]
        },
        {
          "claim": "pep(elena) = true",
          "probability": 0.98,
          "because": {
            "source": "official-gazette-2025-03-14#3920-4188",
            "probability": 0.98
          }
        }
      ]
    }
  ]
}

This could be rendered into a more user-friendly interface, or explained in natural language by an LLM. Either way, the explanation has an underlying record to point to.

That's the kind of software I want to build: something where, when a decision is wrong, we can find the evidence or rule responsible, fix it, and recompute the consequences.