Incidents
An incident is one problem, from the moment it starts until it is resolved. Zharp opens most of them for you when a monitor fails; you can also report one by hand.
Every incident has a page under Incidents with its cause, who has been alerted, and a timeline of everything that happened. The list has three tabs: All incidents (the default), Ongoing and Resolved. Ongoing includes incidents someone is already investigating. All incidents groups them into Ongoing, Resolved today and Earlier.
Calendar
The calendar button next to the search box opens a month view: which days had incidents, how many days that was, and the month's uptime with a tick when it met your 99.9% target. Use the arrows to change month, and pick a day to list its incidents.
Incident report (PDF)
From the menu next to Create incident, choose Download report. Pick a Period (Last 7 days, Last 30 days, This month, Last month, or your own From and To dates, up to a year), then Download PDF. The report covers uptime and the incidents in that period.
Automatic incidents
A monitor is checked from every region it is set up for. When more than half of those regions report a failure, the monitor goes down and Zharp opens an incident named <monitor> is DOWN. The incident page shows the cause in plain words under Cause, and the timeline lists each region that failed and what it got back.
When the incident opens, two things happen at once:
- The response starts. If the monitor has an escalation policy, the policy runs. Otherwise Zharp pages whoever is on call on your primary on-call schedule, once. With no schedule, or nobody on call, the alert is all that goes out. See Escalation & On-call.
- The monitor's alerts go out. The workspace owner and the monitor's subscribers get an e-mail, people who allow it get a push notification, and the monitor's connections are posted to. See Alert Channels.
No incidents are opened for a monitor while it is inside a maintenance window — log monitors included. If a window starts while one of its monitors already has an open incident, that incident's escalation stops: no more steps, repeats or on-duty re-pages run, and the timeline says Escalation stopped. Incidents you reported by hand keep escalating.
Resolved on recovery
When the monitor passes its checks again, the open incident is resolved automatically and the timeline says Resolved automatically. You don't have to close it yourself.
Flapping monitors reopen the same incident
If a monitor fails again within 30 minutes of recovering, and for the same cause, Zharp reopens the incident it just resolved instead of opening a new one. The timeline marks each return as Recurrence #2, Recurrence #3 and so on. The reopened incident is unacknowledged again, and the response starts over as for a new outage: the monitor's escalation policy from its first step, or the on-call responder.
Degraded monitors
When a monitor is still up overall but failing in some of its regions, no incident opens and nobody is alerted: a monitor goes down — and alerts — when more than half of its regions fail. The monitor page shows Failing in … under its status, with the regions, until every region passes in the same round. If the failure spreads to most regions, the monitor goes down and an outage incident opens as usual.
Reporting an incident by hand
Use this for problems no monitor catches, like a broken sign-up flow a customer told you about.
- 1Open the report form
On Incidents, click Create incident.
- 2Describe what's going on
Fill in What's wrong? (this becomes the incident's title) and Details: what you see, since when, who's affected. You can attach files up to 10 MB each.
- 3Say which app and how bad
Under Which application? pick the app if you know it, or leave Let Zharp AI work it out: either way Zharp AI starts looking into it as soon as you send. Under How bad is it? choose Critical, Major or Minor.
- 4Choose who to tell
Pick who to alert under Who should we notify?, then how under Alert by: E-mail, Push notification or Critical alert. Pick at least one.
- 5Send it
Send the report. You're taken to the new incident's page.
| Who should we notify? | Who is paged |
|---|---|
| Whoever is on call (default) | Whoever is on call on your primary schedule — or the entire team if nobody is. |
| The whole team | Everyone in the workspace. |
| A group | Everyone in that group. |
| An escalation policy | The policy runs step by step until someone acknowledges. |
| An on-call schedule | Whoever is on call for that schedule — or the entire team if nobody is. |
| A connection | Nobody is paged: the report is posted to that Slack channel, PagerDuty service, webhook or other alert connection, worded like a down alert. Paused connections aren't listed. |
Whichever you pick, a report on a monitor is also posted to the monitor's own connections that get Down — once each, even if you also picked one of them under Who should we notify?.
Critical alert is a push notification that can break through Do Not Disturb on phones that allow it, and through quiet hours.
Public report links
People without a Zharp account can report a problem through your workspace's report link, https://www.zharp.io/report/<token>. Copy it from the menu next to Create incident (Copy report link), or from Report link on the report form. Share it in Slack, your wiki or with customers.
The public form asks What's wrong?, Details, Which product? and Your e-mail, and takes images up to 10 MB and videos up to 50 MB. Which product? lists the apps you've put on the report page (see Applications), plus Something else, or not sure. A reporter can tick It's urgent and say why: that wakes whoever is on call with a critical alert, even on Do Not Disturb. A public reporter can't choose who to notify: the report always goes to whoever is on call on your primary schedule, or the entire team if nobody is.
A public report isn't credited to anyone in your workspace. The e-mail the reporter gave is kept as the reporter: it's what Created by shows on the incident page and what the incident list shows as the reporter, and the timeline's opening event names it.
Keeping the reporter informed
Whoever reported an incident, on the public link or from the dashboard, hears how it's going by e-mail. The e-mails come from <your workspace> via Zharp and name your team, never the person handling it.
- When someone clicks Acknowledge, the reporter is told We're looking into it.
- When someone clicks Resolve, Zharp asks for an optional Note for the reporter, and the reporter is told it's resolved, with your note. Write it for them: what was wrong and what you did, in a sentence or two.
- When someone clicks Reopen on an incident the reporter was told is resolved, they're told it isn't fixed after all and you're looking into it again. When it's resolved again, they're told again.
- When a Zharp AI investigation can't find the cause, it e-mails the reporter a few questions it needs answered, such as when it happened, the e-mail of their account and which page they were on, with a link to a short form. That happens once per incident, only while nobody has acknowledged it. The form never asks for a password, a card number or a code.
- Their answers appear on the incident's timeline, and Zharp AI investigates again with them: their account e-mail lets it look at that person's own requests, logs and sessions around the time they gave. If it still can't find the cause, its write-up says what it checked and what the reporter told you.
Every incident has a Case ID, such as ZH-7K3F2A, shown on the incident page under Impact (click it to copy). It's on every e-mail to the reporter, which keeps them in one thread in their inbox, and searching for it on the incidents page finds the incident. You don't get these e-mails about something you reported yourself, and the same e-mail is never sent twice in a row.
Turn either off in Settings › AI credits: Keep reporters updated for the acknowledged and resolved e-mails, Ask reporters for details for the questions.
Responding to an incident
Acknowledge
Click Acknowledge to tell everyone you're on it. The incident moves to Investigating, your name appears under Acknowledged by, and escalation stops: no further escalation policy steps run, and the on-duty member isn't paged again. For a monitor's incident, connections that chose to hear about acknowledgements are told.
Resolve and reopen
Once an incident is acknowledged, the button becomes Resolve. A resolved incident can be brought back with Reopen for 24 hours after it was resolved; after that, report a new incident instead. Reopening sets it back to Ongoing and unacknowledged.
Escalate
Click Escalate on an open incident to pull more people in right away. Under Who to notify, search and add any mix of:
- Teammates — a person, or Entire team.
- Groups — everyone in the group.
- Escalation policies — the policy starts from its first step.
- On-call schedules — whoever is on call; the entire team if nobody is.
- Integrations — a connection such as a Slack channel or a webhook.
Under How to notify choose Email, Push notification and/or Critical alert, then click Escalate Incident. Each escalation is recorded on the timeline with who did it and who was reached.
Timeline, comments and attachments
The Timeline records everything that happens to the incident: the check that opened it, who was alerted and how, acknowledgements, escalations, recurrences and the resolution.
To add an update, write in the comment box (markdown works) and click Post. Use the paperclip to attach a file to the comment. Files attached to a report are posted to the timeline as a comment too.
When an escalation policy posts instructions with checkboxes, they appear on the timeline as a checklist anyone on the team can tick. See Escalation & On-call.
Severities
The How bad is it? choice on a hand-made report is kept with the incident as a label. How people are paged is set by escalation policies: a severity belongs to an escalation policy step and decides how the people in that step are reached — for example e-mail and push, or a critical alert. Rules in a policy can switch the rest of the run to a different severity. Severities are set up from the escalation policy editor; see Escalation & On-call.
Who gets paged, in what order, and when.
notifications_activeSlack, e-mail, webhooks and the other places alerts go.
stacked_line_chartShow your users what's up and what isn't.
monitor_heartWhat each monitor checks and when it fails.