How a Custom Business App Should Handle Failed Background Jobs: A Practical Guide for Malaysian Teams

Why Background Failures Deserve a Decision, Not a Default

Background jobs are the parts of a business app that run without anyone watching: sending an invoice email, syncing stock levels, generating a daily report, pushing an order to a fulfilment system. When they work, nobody notices. When they fail, the damage is often invisible until a customer complains or a month-end figure does not reconcile.

Agree on failure handling before development starts so staff and developers can review the same rules. The result is a silent catch block, a log line nobody reads, or an endless retry that quietly burns resources. Business owners and project leads rarely get asked, yet they are the people who know which failures matter commercially.

This guide sets out a practical method for making that decision deliberately. It is written for Malaysian teams commissioning or maintaining a custom business app, and it treats failure handling as a business requirement rather than a purely technical detail. The aim is not to prescribe one architecture but to give you a repeatable way to agree, in plain language, what should happen when a job does not succeed.

One framing helps throughout: every background job has a consequence if it fails. Your job as the business side is to describe that consequence clearly enough that the technical team can design around it.

Start With a Failure Inventory, Not a Technology Choice

Before discussing retries or alerts, list every background job the app will run and what happens to the business if each one fails. Do this on a whiteboard or spreadsheet with the project lead, an operations representative, and the technical lead in the same conversation.

For each job, capture four things. First, what triggers it and how often it runs. Second, what it produces or changes. Third, who notices if it fails, and how soon. Fourth, what the worst realistic outcome is: a delayed email, a duplicated payment record, a stock figure that drifts, a report that is simply late.

Sort the list into three bands. Jobs where failure is annoying but harmless can be logged and reviewed weekly. Jobs where failure affects a customer or money need active detection and escalation. Jobs where failure creates legal, financial or safety exposure need the strictest handling and a human decision before anything is retried automatically.

Use the inventory to decide which failures the team should see first. It converts vague anxiety into a ranked list, and it gives the technical team a defensible basis for where to spend effort. It also protects you later: when someone asks why one job has three layers of alerting and another has none, the answer is written down.

Keep the inventory short. If it runs past a page or two, you are probably describing implementation rather than business consequence.

Decide the Retry Policy Job by Job

Retries are the first reflex, but they are not free. A retry that repeats a non-repeatable action can duplicate an invoice, double-charge a card, or send the same notification twice. A retry that hammers a struggling external system can make an outage worse.

For each job in your inventory, agree on three answers. Is the job safe to repeat? How many attempts are reasonable? How long should the app wait between attempts?

A useful distinction is between jobs that are idempotent, meaning running them twice produces the same result, and jobs that are not. A status update may be safe to repeat if the receiving system can recognise the same event and avoid duplicate notifications. Creating a payment record is not, unless the design includes a unique reference that prevents duplicates. Where a job is not safe to repeat, the correct answer is often to stop and escalate rather than retry blindly.

On timing, avoid fixed intervals that ignore context. A short wait with a few attempts suits a transient network blip. A longer, widening gap suits a dependency that may be down for a while. As an author heuristic rather than an established standard, many teams find that three to five attempts spread over minutes, followed by escalation, is a reasonable starting point for customer-facing jobs, but your inventory should drive the number.

Write the policy down per job. A table with columns for safe-to-repeat, maximum attempts, wait pattern and escalation route is enough.

Design the Escalation Path Before You Need It

Detection without a route to a human is just a louder log file. For every job in the middle and top bands of your inventory, decide who is told, how, and by when.

Start with the trigger. A job that fails once may need nothing more than a record. A job that exhausts its retries, or that fails on a job flagged as critical, should raise an alert. Be specific about the threshold so the technical team is not guessing.

Next, the recipient and channel. A shared inbox works for low-urgency items reviewed daily. A messaging channel or on-call rotation suits jobs where a same-day response matters. Name a role rather than a person where possible, so the route survives staff changes.

Then the response. The person receiving the alert needs a short runbook: what this job does, what failure means, what to check first, and who to contact if the fix is not obvious. Write a short instruction for each job so the on-call person can use it during an incident.

Finally, decide what the app does while waiting for a human. Some jobs should pause and hold state. Others should continue with a safe fallback, such as marking a record as pending rather than complete. Make that choice explicit, because the default is often an inconsistent record that is harder to fix later.

A Hypothetical Example: Monthly Invoice Run

Consider a hypothetical Malaysian services company whose custom business app generates and emails monthly invoices on the first working day of each month. This example is illustrative only and describes no real client or system.

The job runs overnight, produces a PDF per customer, emails it, and writes a status back to the customer record. If it fails partway, some customers receive an invoice and some do not, and the status field may be wrong for both groups.

Applying the method, the team classifies this as a top-band job because failure affects money and customer trust. They note that generating a PDF is safe to repeat, but emailing is not, because a duplicate invoice confuses customers and creates reconciliation work. They decide the job should process customers one at a time, record success per customer, and never re-send to a customer already marked as sent.

On retries, they allow a small number of attempts for the PDF step and none for the email step once a send has been attempted. On escalation, an exhausted retry raises an alert to the finance shared inbox before business hours, with a runbook explaining how to identify which customers were missed and how to resume from the last successful record.

While waiting, the app marks affected customers as pending rather than sent, so the status field never overstates progress. The team also agrees a simple morning check: the count of invoices sent should match the count of customers due. The reconciliation step can show which jobs completed and which need attention.

The value of the example is not the specific numbers. It is that every decision was made by the business side, in advance, in language the technical team could implement directly.

A Checklist, and Where the Build Partner Fits

Use this checklist at the start of a build or when reviewing an existing app. It is a practical heuristic, not a formal audit standard.

One: every background job is listed with its business consequence and a band. Two: each job is marked safe or unsafe to repeat. Three: maximum attempts and wait pattern are written per job. Four: escalation thresholds, recipients and channels are named by role. Five: a short runbook exists for each critical job. Six: the app's behaviour while awaiting a human is defined. Seven: a simple reconciliation check exists for money-related jobs. Eight: someone owns a periodic review of the failure log, even if that review is monthly.

Score each item as done, partial or missing. Treat any missing item on a top-band job as the first thing to fix, regardless of how the rest of the project is tracking. Review the inventory whenever a new job is added, because failure handling designed for one job rarely transfers cleanly to the next.

Ask the business owner to define escalation routes and the technical team to implement and test them. A software application development company engaged for the build should be able to show you, per job, how retries, escalation and reconciliation are implemented, and should welcome the inventory as input rather than treating it as scope creep.

Blackstone Intelligence operates under the legal business name Blackstone Consultancy Sdn Bhd, and this identity is the correct one to use in contracts and project documentation. Whoever you work with, ask for the failure inventory and the runbooks as deliverables alongside the app itself. They are small documents that decide whether a quiet overnight failure becomes a minor inconvenience or a month-end problem.