/
How do I monitor the sentry queue?

How do I monitor the sentry queue?

Owned by Mariya Nomanbhoy (Unlicensed)
Last updated: Apr 11, 2018 by Parul Raheja (Unlicensed)

Daily Process:

  1. Log into sentry: https://sentry.io/sighten/production/
    1. Make sure the projects filter at the top has Production selected
  2. Sort by "First Seen"
  3. Review all errors un-assigned errors in the queue
    1. Tues - Fri - This should generally be all errors created in the the last 24 
    2. Mon - This should generally be all errors created since Friday
  4. After reviewing an error, assign error to yourself so it is clear it has been reviewed 

Goals:

  • All Production sentry error's reviewed
  • 0 open E1 / P1 issues in SEN project in JIRA
  • All Sentry errors on sentry-prod E1 or P1 errors
  • Error's addressed based on appropriate priority
  • Minimal noise (metric?)
    • Minimal sentry errors silenced
    • Minimal sentry errors ignored

Jira SEN Tickets:

For all new tickets in Jira SEN project:

  • Add steps to reproduce to ticket
  • Add priority to ticket:
    • E1
      • Highest Priority Error 
      • Main functionality blocked, user cannot proceed with deal
      • Hotfix prior to next release
    • P1
      • High Priority Error
      • Key functionality heavily degraded, large number of users impacted
      • Include in next release
    • P2
      • Medium Priority Error
      • Clear bug
      • Include in next 4-6 weeks
    • P3 
      • Low priority Error
      • Minor bug
      • Keep in backlog
    • Not a bug
    • Needs Aggregation
      • Error is creating multiple tickets in Jira
      • Please use one ticket as the "aggregation" ticket to clean up the error handling and ensure the error is thrown in a way that merges all instances together
  • Add sprint to ticket:
    • If E1, add to current hotfix and leave unassigned
    • If P1, add to current sprint and leave unassigned
    • If P2, add to current design sprint and assign to Mariya
    • If P3, leave in backlog and assign to Mariya

How to check an error:

  • Try to determine how to reproduce the error on staging:
    • Watch Fullstory if available
      • See section on Fullstory below
    • Check for api request : Under the exception trace, you may see an exact api request
    • Open each line in the exception trace to see if there is a clear indicator of a failure on a specific field
    • If you have full steps to reproduce and are certain of priority:
      • If E1, add to current hotfix and leave unassigned
      • If P1, add to current sprint and leave unassigned
      • If P2, add to current design sprint and assign to Mariya
      • If P3, leave in backlog and assign to Mariya
    • If you are uncertain of steps to reproduce and / or priority

Prioritization:

  • Impact on deal
    • If there is no workaround to get through a deal, then the ticket is automatically E1
    • Example: User cannot download a contract
  • First seen
    • Did an error start occuring after a recent release?

  • Number of events

  • Number of users
    • Ignore all sighten users, such as admin@sighten and sightenadmin@


Closing Ticket:

  • If ticket is deemed as won't fix
    • Ignore for 6 months
    • Close Jira Ticket
      • Resolution as Won't Do
      • Add reason it is low priority
      • Communication Audience: Internal 

Ignoring Errors:

Errors that are determined should not throw errors:

  • Ignore for 6 months 
  • Create a ticket and add to the next release to remove throwing the sentry error from the code
  • Add comment to Jira ticket and in Sentry error

Errors that are determined should throw errors, but we will not fix at this time:

  • Ignore for 6 months 
  • Add comment to Jira ticket and in Sentry error

To find all ignored errors, search for is:muted.

Slack

Link to #sentryprod channel: https://teamsighten.slack.com/messages/C03JYRC1G/files/F89EU5FNU/

  • Review all errors immediately as they come up on the #sentryprod channel
    • If the error is E1, move into current hotfix and assign to VP of Engineering
    • If the error is P1, move into current sprint and leave unassigned
    • If the error is not E1 or P1, follow the instructions to the ignore the error

Fullstory

Many Sentry errors give you a direct link to Fullstory.  This allows you to watch the exact actions of the user when the error happens.  The Fullstory URL can be found in the sentry error page by finding the "Cookies" section and retrieving the "fs_uid" (the highlighted section of the uid can be placed in a fullstory URL).

Example:https://www.fullstory.com/ui/1chQ/session/5687485698932736:5647791397273600 

This will take you to the BEGINNING of a fullstory session, which will NOT be the exact time of the error.  To find the error, you can do the following:

  • Try to match up the timestamps.  For example, if the sentry error occurred at 5:10pm PT and the Fullstory starts at 4:55pmPT, then fastforward 15 minutes and you should find the error.
  • Scan throw the entire video on 4x speed until you find the errors

Jira

All Sentry errors automatically create a ticket in the Jira  Sentry (SEN) project : https://sighten.atlassian.net/projects/SEN/issues.  Our goal is to have 0 open issues in this project.

Tips & Tricks

https://docs.sentry.io/clients/javascript/tips/