When it comes to product development decisions at SAM, our main question is, “Will it make us faster, more accurate or more relevant?”. SAM clients span a broad range of industries so product feature requests can be quite diverse and, knowing how to filter these requests is key, as when we embark on larger feature releases we want to be sure it helps all SAM clients. One request we were hearing consistently, across the varying industries, was the ability to filter alerts by severity – our clients wanted more relevant content.
Most tools in the market today require the end user to read text or view images in order to ascertain the severity of a particular event. This means end users often do more work than they would like, or have time for, to make sure they don’t miss a high impact event. While SAM is always aiming to be fast and accurate, we’re also looking for ways to make our clients’ jobs easier, so we set off to build the foundations for additional layers of relevance beyond geo fencing and topics.
This particular product feature posed an interesting task for our Social Intelligence and Engineering Teams. We examine the considerations both teams faced when building an AI-based severity indicator in SAM, while also taking a look at how severity in SAM will work.
Social Intelligence Team
The core mission of the Social Intelligence Team on this project was to deliver the expert knowledge required to create a baseline for the machine learning team to build from and train against, and to ultimately monitor the results the machine produced against our core assumptions. Creating a severity scale for all global events is no small project and one we continue to work to refine.
The more we delved into the concept of severity the more subjective we realised it was, touching upon the core question of how events become news, a fundamental question for journalists and academics for decades. In determining and defining SAM’s concept of severity, we considered using news coverage as an indicator of severity but ultimately rejected that idea to ensure that the system remained impartial. We wanted to ensure that if an event happened in a part of the world where it would be more heavily documented on social media or in the news, that volume would not impact the perceived severity of the event as our clients expect an unbiased, factual view of the world. So, how would SAM Alerts categorise severity, and how many levels would we need?
We decided on a simple three-layer severity scale: low, medium and high. From here, we mapped events back to each layer. Human injury, risk to life and key locations were some of our key determining factors in deciding how severe an event might be. We decided that a low severity event might include a power cut or hail storm for example, medium severity events would include shootings and evacuations, while high severity events include those with multiple deaths or a high level of injury reported or indicated. Because of SAM’s speed, we were conscious that events might start small and then escalate. To anticipate the changing nature of live situational and ongoing events, the Social Intelligence team developed a comprehensive matrix where the potential severity could increase as a situation unfolded. For example, an office lockdown starting out with a medium severity level would change to a high severity level if it transpired there was a shooting causing multiple injuries or deaths. Location of an event was another escalation factor, taking into account high-density areas such as a school, a mall or an airport.
Applying severity categories to our event detection model posed many more questions for SAM’s Social Intelligence team, as did defining the concept and training the required data. SAM’s Engineering team faced an entirely different host of challenges when completing their part of the project.
Determining severity is one of the most challenging technical problems we have faced so far at SAM. We thought because we had been dissecting news events on social media for several years that we would have a good idea of what assigning severity to events would entail, using machine learning and AI to frame it. The team started with a simple rule-based approach, but it became obvious that this was neither scalable nor maintainable. The rules were often not broadly applicable and did not allow us to capture all the different dimensions that contribute to severity. We ended up immersed in varying shades of grey. Instead, we decided to work with the rich dataset of global events that building SAM provided us. As these events vary in severity ranging from car crashes and protests to tragedies on the scale of the Christchurch shooting, each one can provide a unique set of timelines and analytics. This allowed us to extract various features of severity from the dataset to inform our decisions.
We carefully audited tens of thousands of SAM events and labeled them. A huge challenge during this labelling process was assigning the correct severity to an event due to the subjectivity of the concept, and changes happening over time as an event unfolds. To keep the AI sharp, we built a process to evaluate models on a daily basis and re-train them to make them more accurate and confident day by day.
An example of severity in action:
As you can see, the severity of an event can change with time as more information becomes available or the event itself changes. We define the severity of an event as its peak severity.
Events were labelled based on their peak severity in the training phase. In the production evaluation phase we re-evaluate severity as new information is being added to events, so when new tweets, snaps and posts come into the system, we can map that back to the last known severity level in real-time to determine if severity levels have increased with new information. To ensure we are using the most up-to-date information, we give more weight to the newer tweets in an event.
Once we built the base dataset we explored different machine learning approaches. It was clear from the early days that the transition to machine learning would be a gradual process as we enhanced our dataset and the models became more confident. Furthermore, we did not want to lose the domain knowledge that our Social Intelligence Team had acquired over several years. Therefore, we built a hybrid system combining both machine learning and expert systems.
With our machine learning approach we utilized many advanced and modern techniques such as state-of-the-art embeddings, trends of tweets and many more features engineered carefully based on our domain knowledge. We also leveraged many advanced classification techniques such as gradient boosting, ensemble methods and deep learning based methods.
On top of designing the models and features, we used optimization techniques such as Bayesian optimization to explore a large space of model parameters and select the best model.
We’re extremely excited to be rolling out severity indicators on all SAM Alerts. Starting today, you will see our little severity meter next to the timestamp. The complexity and subjectivity of this project means our roll out plans are a little bit different than what we normally do here at SAM. Over the next several weeks our teams will be soliciting feedback from all of our users across the varying industries to see if our assumptions line up as an agreeable baseline. After the review period, the next phase of this project is enabling users to filter and build SAM Streams using severity levels as a filter, in addition to the current filters already available on the platform (location, topic etc…).
One of the aims at SAM is to help our customers understand when unplanned events impact their operations and safety. We hope that automating severity indicators will serve as a powerful building block to provide more informative alerts with less noise – saving you even more time when you need it most.
If your organization is not yet using our platform to understand what is happening in the world and how it might impact your operations, then please feel free to get in touch by clicking the link below and a member of the team will be in touch.
Thanks to Louise (Social Intelligence) & Siavash (Engineering) for an in depth look at severity in SAM.