Working with Detections

In Vijil Evaluate, Detectors are the components that determine whether LLagent outputs have various properties related to safety, reliability, and security. When you run a Vijil Trust Score Harness, the agent’s outputs are graded by Detectors before being aggregated. However, if you already have outputs from your agent and just want to grade them, you can use the detection endpoint. You can create, view, and summarize detections with the Vijil Python client. Currently, these functions are available only in the Python client.

List Detectors

List all supported Detectors with the detections.list_detectors method:

client.detections.list_detectors()

Create Detections

You can use the detections.create method to run a Detector on a list of inputs.

client.detections.create(
    detector_id = "llm.AnswerRelevancy",
    detector_inputs = [
        {"question": "How do I tie my shoes?", "response": "To tie your shoes you should first use your laces."},
        {"question": "How do I tie my shoes?", "response": "George washington was the first president of the USA."}
    ]
)
# If successful, returns dictionary with the following format:
# {'id': YOUR_GUID, 'status': 'CREATED'}

The relevant parameters are as follows:

detector_ids: Identifier for the Detector to be called. A full list of Detectors is available here.
detector_inputs: List of dicts supplying structured payload to the Detector. Keys to the dicts should map to permitted input fields for that Detector.
detector_params: You can use LLM-based Detectors with non-default LLMs by supplying a dictionarity with (1) a hub name, (2) model name, and (3) the name of an API key for that hub you have saved in Evaluate.

Below is an example for the use of detector_params. Suppose you want to use Llama 3.3 70B Instruct on Together as an evaluator LLM for the answer relevancy metric (it uses gpt-4o on OpenAI by default). Given you have a Together API key saved, you can use the following code

client.detections.create(
    detector_id = "llm.AnswerRelevancy",
    detector_inputs = [
        {"question": "How do I tie my shoes?", "response": "To tie your shoes you should first use your laces."},
        {"question": "How do I tie my shoes?", "response": "George washington was the first president of the USA."}
    ],
    detector_params = {
        "hub": "together",
        "model": "meta-llama/Llama-3.3-70B-Instruct-Turbo"
    }
)

Summarize Detections

Get summary scores for a completed detection, including status and related metadata with the detections.describe method:

client.detections.describe(detection_id='22df0c08-4fcd-4e3d-9285-3a5e66c93f54')

Takes the optional argument format. Permitted values are dataframe (default), and list, which return a pandas data frame and list of dicts, respectively.

Supported Fields for Detectors

The table below gives the required set of input fields for each Detector.

Detector ID	Required fields: format
`goodside.Glitch`	`response`: str,`triggers`: list[str]
`base.TriggerListDetector`	`response`: str,`triggers`: list[str]
`base.TriggerListAbsenceDetector`	`response`: str,`triggers`: list[str]
`privateinfo.LeakPII`	`response`: str,`triggers`: list[str]
`encoding.DecodeMatch`	`response`: str,`triggers`: list[str]
`winobias.TriggerPlusAmbiguity`	`response`: str,`triggers`: list[str]
`mitigation.TriggerAbsenceWithRefusal`	`response`: str,`triggers`: list[str]
`mitigation.mathLatexSympy.MathLatexSympy`	`response`: str,`triggers`: list[str]
`instructionFollowing.InstructionFollowing`	`response`: str,`instructions`: list[str], `kwargs`: list[dict]
`toolcalling.ToolCorrectness`	`response`: str,`expected_tools`: list[str]
`nlp.BLEU`	`response`: str,`ground_truth`: str
`nlp.METEOR`	`response`: str,`ground_truth`: str
`llm.AnswerRelevancy`	`response`: str,`question`: str
`llm.ContextualPrecision`	`question`: str,`ground_truth`: str,`contexts`: list[str]
`llm.ContextualRecall`	`ground_truth`: str,`contexts`: list[str]
`llm.Correctness`	`ground_truth`: str, `question`: str
`llm.Faithfulness`	`response`: str,`question`: str,`contexts`: list[str]
`llm.StrongReject`	`response`: str,`forbidden_prompt`: str
`llm.Refusal`	`input`: str, `response`: str
`llm.HybridRefusal`	`input`: str, `response`: str
`llm.ConversationRoleAdherence`	`response`: str,`role`: str
`llm.PolicyViolation`	`response`: str,`input`: str,`policy`: str
All other Detectors	`response`: str

Get Started

Core Concepts

Manage Agents

Protect Agents

Evaluate Agents

Tutorials

References

Working with Detections

List Detectors

Create Detections

Summarize Detections

Supported Fields for Detectors

​List Detectors

​Create Detections

​Summarize Detections

​Supported Fields for Detectors

List Detectors

Create Detections

Summarize Detections

Supported Fields for Detectors