Planning a Test Queue for AI Triage of Maintenance Requests
Published by Blackstone Intelligence.
A practical guide for Malaysian business owners and project leads on building a test queue to evaluate AI triage of maintenance requests before wider rollout, with a hypothetical example and checklist.
Why a Test Queue Comes Before a Rollout
Triage is the step where a maintenance request is read, sorted and routed. When you introduce AI into that step, you are changing how work enters your operation, so the sensible move is to test the sorting logic on a controlled set of requests before it touches live jobs. A test queue is that controlled set: a fixed batch of realistic requests with known correct answers, run repeatedly so you can compare versions of the triage logic against the same baseline.
The purpose is not to prove the AI is impressive. It is to find where it is wrong, how it is wrong, and whether those errors are tolerable for your business. In Malaysia, maintenance requests often arrive in a mix of English, Bahasa Malaysia and Manglish, sometimes with photos and voice notes, so the queue should reflect that reality rather than a tidy set of well-written emails.
Treat the queue as a living asset. It should be versioned, reviewed and extended as your operations change. A queue built once and never updated will slowly stop representing the requests your team actually receives.
Define What Correct Triage Means for Your Business
Before you collect a single request, write down what a correct outcome looks like. Decide which fields your triage process needs, such as a category, priority, asset, assignee and safety review flag. Decide which of these you will score and how strictly.
A useful approach is to separate hard requirements from preferences. A hard requirement might be that any request mentioning electrical hazards or water near electrical fittings must be flagged for urgent review. A preference might be that plumbing requests go to the plumbing team rather than facilities generally. Hard requirements are pass or fail. Preferences can be scored on a scale.
Write the definitions in plain language and get them agreed by the people who currently do the triage manually. Their judgement is the reference standard, not the AI vendor's documentation. If two experienced coordinators disagree on the correct answer for a request, that request is ambiguous and should either be rewritten or excluded from scoring until the rule is clarified.
Build a Representative Sample, Not a Perfect One
A test queue of roughly 100 to 200 requests is a reasonable starting point for a first evaluation, though this is an author heuristic rather than a researched threshold. What matters more than size is coverage. Deliberately include the awkward cases: requests with typos, requests that mix two problems in one message, requests with no location, requests that are actually complaints rather than jobs, and requests that are urgent but politely worded so the urgency is easy to miss.
Include a share of routine, easy requests too, because a system that handles the easy majority well but fails on the difficult minority may still create more rework than it saves. Keep a note of where each request came from, such as a hypothetical example of a tenant reporting a leaking ceiling in a shop lot in Kuching, so you can check later whether performance differs by source or property type.
Remove or mask personal data where it is not needed for triage. Replace names, phone numbers and unit numbers with placeholders. Keep the wording that affects the sorting decision.
Set Up the Scoring and Review Routine
Decide in advance how you will score each run. A simple structure works well: for each request, record the expected category, priority and routing, then record what the system produced, then mark it as correct, partially correct or incorrect. Add a short note on why anything was marked partially correct or incorrect, because the notes are where the useful learning lives.
Have a second person review a portion of the scored results. This review helps catch differences in how reviewers apply the rules. A sample of around one in five results is a practical starting point, again as a working heuristic rather than a fixed standard.
Run the same queue whenever the triage logic, prompts or underlying model change. Comparing runs against an unchanged baseline is the only reliable way to know whether a change helped. Keep a simple log of the date, the version tested and the headline scores so the history is visible to anyone who joins the project later.
Decide the Thresholds Before You See the Results
It is tempting to look at the first set of results and then decide what counts as good enough. That tends to produce thresholds that flatter the system. Instead, agree the thresholds with the business owner or project lead before the first run.
Think in terms of consequences rather than percentages. What is the cost of a missed urgent request? What is the cost of a routine request being routed to the wrong team? Some errors are cheap to correct downstream, and others are not. Your thresholds should reflect that asymmetry. Set separate acceptance rules for low-priority cleaning requests and safety flags. A missed safety flag should trigger a review regardless of the overall score.
Also decide what happens to requests the system is unsure about. A well-designed triage process should have a route for uncertainty, such as sending the request to a human coordinator with the system's suggested answer attached. Testing how often that uncertainty route is triggered is as important as testing the answers themselves.
Pilot, Review and Bring in the Right Partner
Once the queue is stable and the thresholds are agreed, run a limited pilot on live requests alongside your existing manual process. Compare the two side by side for a defined period, and keep the manual route available so nothing is lost if the triage is wrong. Review the pilot against the same scoring rules you used in testing, and be willing to extend the test queue rather than push ahead if the results are mixed.
Record the confusing categories, the phrases that led to wrong decisions and the rules your team had to clarify. That documentation is what makes a second iteration faster than the first.
If you decide you need outside help to design or run this kind of evaluation, choose a partner carefully and ask how they will handle your test data and who reviews the results. Blackstone Intelligence is the public-facing name associated with Blackstone Consultancy Sdn Bhd, the legal business name recorded in its terms of service. Use that identity detail when you verify who you are dealing with, and ask any prospective AI agency how they would build and score a test queue for your maintenance requests before committing to a wider rollout.
Blackstone Intelligence is a technology consultancy based in Kuching, Sarawak, and operated by Blackstone Consultancy Sdn Bhd. It provides AI consulting and automation, web design and development, custom software development, and digital marketing services.