A rotating on-call schedule for operational, assist, and tech-dept frees the remainder of the workforce to do nice improvement
A typical problem for each knowledge science or product workforce is to align the brand new (product improvement) with the outdated (operational, assist) duties. When the complete workforce is meant to deal with each, it signifies that on one aspect the workforce is required to maintain a product deadline and launch a brand new product function whereas, on the identical time, the workforce is anticipated to work operationally and repair present merchandise and assist business questions and calls. This example causes sudden context switches and, finally, results in much less effectivity, failing deadlines, and stress.
In observe, this usually results in a state of affairs the place sure workforce members tackle these extra duties or are specialised to take action. However that’s harmful as a result of as quickly as one in all these specialised workforce members goes on trip, the entire firm may really feel that and has an issue.
Therefore, an environment friendly and scalable knowledge workforce must assist each operational and new improvement work and create a system that features:
- Good information sharing amongst workforce members on tips on how to do operational work and assist merchandise/clients
- Uninterrupted improvement work with out a lot context switching
- Nicely-defined and estimated upkeep work to maintain keep away from sudden deadlines
One system that turned out to work very effectively for us up to now is a rotating on-call system that does deal with greater than “simply” alerts in manufacturing. Merely put, this can be a rotating system the place one (or extra) workforce members are the designated survivors for a particular period of time and are purely answerable for operational work.
The individual on name isn’t just doing a job, the individual is defending all the workforce from all of the chaos taking place outdoors of the event work
To complete that time, this method permits for that solely the on-call individual (the designated survivor) is dealing with all of the work that doesn’t fall beneath “new improvement”. Throughout that point, the individual on-call isn’t just doing a job, the individual is defending all the workforce from all of the chaos taking place outdoors of the event work, together with:
- Repair manufacturing pipeline points
- Reply business / buyer questions
- Assist buyer calls
- Cut back tech dept (backlog)
As might be seen within the determine above, dealing with the “basic” on-call system and ensuring the manufacturing setting works remains to be crucial. Nonetheless, if there aren’t any points in manufacturing, this frees up for different duties like supporting business requests, buyer calls, or decreasing the backlog.
Switching to the system at first won’t be simple. Not each workforce member can simply take accountability for the manufacturing pipeline, business assist, and the tech dept. However that shouldn’t be a blocker. You will need to talk correctly that the individual on-call owns these objects and is the primary line of protection however can ask for assist at any time.
In the long term, this may convey plenty of advantages to the workforce and all the group. Essentially the most intuitive advantages are that it’s method simpler to estimate improvement work and that the workforce will turn into extra environment friendly (much less context change). This additionally goes for the operational aspect the place the variety of individuals being a part of the on-call system defines how a lot operational work is feasible. This makes communication with the corporate and stakeholders method simpler as a result of a workforce of 5 individuals with 1 individual within the rotation means 1 out of 5 FTEs is sustaining all techniques and work associated to present merchandise (20% operational, 80% improvement). That’s simple to account for and to estimate.
Nonetheless, there are extra advantages coming in over time, nearly as unwanted side effects. All workforce members will turn into full-stack knowledge scientists. The reason being that each workforce member wants to know a sure minimal of the merchandise, clients, techniques, fashions/logic, and code infrastructure concerned. They don’t have to be specialists however they’ll finally turn into ok to deal with these alone for at the very least 1 week. This may also be sure that it isn’t in any respect a problem when a helpful workforce member goes on trip for the reason that individual on-call will at all times have the workforce’s again.
As well as, regardless that this on-call time may generally be a bit extra tense, it offers the information scientist the chance to see what’s outdoors of the workforce and to collaborate with the business aspect and clients. This is usually a very helpful and rewarding expertise.
That is the place it will get somewhat bit technical (for the individuals who like code, simply scroll right down to the very finish). Establishing such a system is pretty easy however may contain some coding. Crucial half is communication with the workforce and stakeholders and informing them how that is going to work.
Because the complete level of the system is to assist the workforce, and to not create extra overhead, I extremely suggest totally automatizing it. To take action, you would want to have at the very least 3 techniques in place:
- A pager system linked to manufacturing that alerts when manufacturing fails (e.g., Opsgenie or Pagerduty)
- A scheduling system that detects who’s on name and might talk that to a different system (e.g., Apache Airflow or Keboola)
- A communication platform that’s used to achieve out to your workforce and to make tickets (e.g., Slack or Teams)
In case you have these techniques in place and you’ve got API entry to the pager system and to the communication platform, then you might be nearly performed. The one factor left to do is to arrange a job within the scheduling system that runs an API name first to get who’s on name from the pager system and an API push afterward to speak or overwrite channels/teams/tags within the communication platform.
Beneath is an instance of how such a easy API name can appear to be that may give you the individual on name from Opsgenie:
curl -X GET
'https://api.opsgenie.com/v2/schedules/{schedule_name}/on-calls?scheduleIdentifierType=title&flat=true'
--header 'Authorization: GenieKey {token}'
After that, you wish to run a command that does one thing in your communication system. As an example, in Slack, overwrite a consumer group in order that it accommodates solely the consumer who’s on name:
curl -X POST
-F usergroup={usergroup}
-F customers={consumer}
'https://slack.com/api/usergroups.customers.replace'
-H 'Authorization: Bearer {token}'
On the finish of this story, you can see an entire code model of how this code might be robotically scheduled. This may be sure that each time when somebody tags your group on Slack (like @ workforce), solely the individual on-call will probably be tagged and might resolve if extra workforce members have to be notified. It additionally means that you can rapidly add new duties to the dag. As an example, whenever you wish to notify the corporate or the workforce who’s going on name now or if you wish to modify your ticketing system accordingly.
Having a rotating schedule for the workforce’s operational, business, and tech dept work is making your knowledge workforce extra environment friendly. It should cut back context change and permits for higher time estimations. As well as, it would educate full-stack knowledge scientists which might be assured in dealing with a variety of points to guard the remainder of the workforce.
All photographs, except in any other case famous, are by the creator.
Code Appendix:
Instance of an Airflow dag that fetches the one that is on name from Opsgenie and overwrites a consumer group in Slack to solely include that individual. The coding is actually not excellent (Information Scientist at work) however I’m positive you get it:
# Import
from airflow import DAG, XComArg
from typing import Dict, Record
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.fashions import Variable
import json# Fetch secret tokens
slack_token = Variable.get("slack_token")
opsgenie_token = Variable.get("opsgenie_token")
# Setup DAG
dag = DAG(
dag_id,
schedule_interval=schedule_interval,
default_args=default_args,
catchup=catchup,
max_active_runs=max_active_runs,
)
with dag:
# Run BashOperator fetching from Opsgenie who's on name
def fetch_who_is_on_call(**kwargs):
fetch_who_is_on_call_bash = BashOperator(
task_id="fetch_who_is_on_call_bash",
bash_command="""
curl -X GET
'https://api.opsgenie.com/v2/schedules/{schedule_name}/on-calls?scheduleIdentifierType=title&flat=true'
--header 'Authorization: GenieKey {token}'
""".format(
schedule_name="schedule_name",
token=opsgenie_token
),
dag=dag,
)
return_value = fetch_who_is_on_call_bash.execute(context=kwargs)
fetch_who_is_on_call_bash
return return_value
# run BashOperator in PythonOperator and supply context
opsgenie_pull = PythonOperator(
task_id="opsgenie_pull",
python_callable=fetch_who_is_on_call,
provide_context=True,
dag=dag,
)
# Overwrite slack group with the individual on name
def overwrite_slack_group(**kwargs):
# First: get who's on name from PythonOperator
ti = kwargs.get("ti")
xcom_return = json.hundreds(ti.xcom_pull(task_ids="opsgenie_pull"))
user_email = xcom_return["data"]["onCallRecipients"][0]
user_dict = {
"data_scientist_a": "A03BU00KGK4",
"data_scientist_b": "B03BU00KGK4",
}
user_id = [
user_dict[k] for okay in user_dict.keys() if okay == user_email.cut up(".")[0]
]
# Second: Run BashOperator to overwrite slack group
overwrite_slack_group_bash = BashOperator(
task_id="overwrite_slack_group_bash",
bash_command="""
curl -X POST
-F usergroup={usergroup}
-F customers={consumer}
https://slack.com/api/usergroups.customers.replace
-H 'Authorization: Bearer {token}'
""".format(
usergroup="usergroup_id",
consumer=user_id,
token=slack_token,
),
dag=dag,
)
overwrite_slack_group_bash.execute(context=kwargs)
overwrite_slack_group_bash
# Run BashOperator for slack overwrite in PythonOperator
overwrite_slack = PythonOperator(
task_id="overwrite_slack",
python_callable=overwrite_slack_group,
provide_context=True,
dag=dag,
)
opsgenie_pull >> overwrite_slack
return dag
