Proposed by HIE User Researcher
Joseph Yeh | Senior UXR
Monly Zhuang | UXR intern
_edited.jpg)
Trend Micro User Research
The Blue Ocean of
NLP Solutions in Cyber Security
Project Overview
-
Project Duration: 1.5 months
-
Conduct comprehensive competitor research on 5 key competitors.
-
Clarify the main usage scenario of NLP and discover the remaining needs.
-
Integrate TrendMicro's strength and the un-covering needs, develop 4 key business strategies.
-
Was invited to present with the AI team, built a bridge with the development segment for further collaboration.
-
The key features proposed have been officially adopted and developed.
Background
"Natural Language Processing" has been applied by many competitors, which grabs customers' attention.
With the rise of ChatGPT by the end of 2022, many cybersecurity industries have also begun to adopt natural language processing technology, aiming to simplify complex cybersecurity information and assist novice security personnel in handling advanced tasks.
In response to this trend, TrendMicro has also begun to seek strategies for utilizing NLP technology to address current gaps and leverage its strengths to the fullest extent.
Research Objective
Discover the Blue Ocean of
NLP Solutions in Cyber Security
In this research, we want to investigate how our competitors apply the NLP technique to their solutions, what kinds of user needs have and haven't been solved. Thus finding out the uniqueness that TrendMicro can further develop.
What kind of user needs hasn't been solved?
How do competitors apply NLP in their solutions?
1.
2.
3.
What action should TrendMicro take?
Introduction
Who are our USERs? SOC Team!
Responsible for more complicated incidents that escalated by L1.
And generating incident reports.
SOC Manager / Planner
Manage
Evaluation
Responsible for team management, evaluation and enhance work efficiency.
Responsible for the first quick triage, simple investigation.
The user's Needs or Pains within the user flow
The user flow, user needs, and pain points have been clarified in the previous study. To achieve better consistency, we would apply this competitor research into the same framework.
That is, the needs and pain points summarized here will be combined into later competitor research insight to explore the problems each function solves.
*Formed by previous TM user research.
Research Method
COMPETITOR RESEARCH
To understand how other cyber security vendors apply NLP in their solutions, we deeply investigate these 5 key competitors below and connect their proposed features to the user pains and needs figure above.
Microsoft
Security Copilot
*Available only in preview, not yet publicly available.
Chronicle
*Not yet available.
CrowdStrike
Charlotte
*Available only in private customer preview.
SentinelOne
Purple AI
*Not yet available.
Recorded Future
Recorded Future AI
*Available to all existing clients today.
*The available status by May 2023.
Research Timeline
My Role
Study and Organize within 1.5 month
Competitor Study and
Business Strategies development
Find out the connection between user needs and competitors' strategies and features
Strategies Development
Develop unique business strategies that haven't been covered by the market.
0.5
month
Competitor
Study
1 month
Research Insight
Competitors' NLP solutions, The User Needs Remain.
+
COMPETITOR
As our audience was UI designers and developers, they would like to know the details about the features and UI designs of the competitors.
Thus in our study here, we connect the features and UI design to the user journey, needs, and pain points.
The details are shown below.
Overall, they treat a dialogue as an incident file.
And emphasize recording, and collaborating.
One dialogue is one task, such as a security incident or event.
Collaboration: Add members to the dialogue to co-work together.
Pin: Pin up the critical message.
Feedback: Optimize the conversation with Copilot by feedback function.
Same, One dialogue is one task.
Record: Automatically record the incident handling process to help other members understand the case.
Hard to learn from others/ past cases
Related Pains
In the user flow of previous research
Phase
C. Triage
D. Investigation
Microsoft takes a highly guided approach for
L1 analysts to conduct advanced defense
L1 can leverage info from different security tools for triage.
By typing a prompt or feeding in a file, URL, or code snippet, users could ask for info about incidents or alerts from other security tools, which can assist in differentiating the true/false positive in the first place.
Suggested prompts for analysts to continue the investigation.
The prompt suggestions are based on its observation of users' work, which can help speed up the precise investigation.
Investigate automatically with the suggested settled prompt book based on alert/incident.
Explain the alert/incident with clear visualization by interpreting and summarizing the files, alerts, and TI, which can eliminate fragmentation across various security tools and information. With Copilot's prompt book, analysts could investigate the essential steps automatically to understand more detail about the incident, which is also available for adding or editing prompts.
CrowdStrike allows analysts to understand comprehensive situations in a very humanistic way.
Prompt recommendation for knowing the security posture.
“What actors are targeting us?”
“What vulnerabilities are common in our environment?"
Understand the risk within the environment
Provide threat landscape, risk level against critical vulnerabilities, current security posture, compliance requirements, cyber security performance metrics, etc.
“Do we have vulnerabilities involving Microsoft Outlook?”
“What are the biggest risks facing our business critical assets?”
“Are we protected against the Log4j vulnerability? Where are we at risk?”
Intuitively leverage threat detection for further investigation.
-
Perform advanced security actions, leverage detection, investigation or response workflow by human language without any coding.
-
Automating repetitive tasks like data collection, extraction, and basic threat search and detection.
“Find lateral movement involving Windows hosts.”
Google provides the coding according to the prompt for further verification.
Provide the triggered query for verifying the generated outcome.
Hard to learn from others/ past cases
Related Pains
In the user flow of previous research
Jump among consoles to manually enrich asset data by copying and pasting the information into notes
Too much effort on training L1 analyst
Past cases are not correlated that gains repeated work
Need to collect asset profile from different console
Phase
D. Investigation
Recorded Future transfer "Threat" information to Human Language.
Summarize worth-to-hunt threat
Recorded Future applies GPT to summarize the threat intelligence into a paragraph of simple insight, which can decrease the digesting time.
Detailed threat information
Let analysts understand the critical risk of specific devices or software, as well as how the threat actors attack.
SentinelOne assists in threat-hunting and provides event summaries in Human language.
Hunting by ingesting, aggregating, and correlating data to the organization’s environment
Sentinel One also conducts proactive threat hunting by ingesting, aggregating, and correlating data from endpoint, cloud service, and network logs and acts as an automated assistant that security analysts can use to ask threat-hunting questions.
“Is my environment infected with SmoothOperator?”
“Do I have any indicators of SmoothOperator on my endpoints?”
Event Summary
First indicates whether any malicious activity had been seen and where, next provides detailed analysis of the events, indicating the behaviors that made SentinelOne classify the behavior as malicious, then presents aggregations on the activity made by the suspicious process, then move on to provide additional information on the entities associated with the malicious process, like users or files.
Hard to learn from others/ past cases
Related Pains
In the user flow of previous research
Jump among consoles to manually enrich asset data by copying and pasting the information into notes
Look up IoC information on websites and taps to manually enrich data by copying and pasting the information in notes
E. Response
Phase
Microsoft and CrowdStrike provide remediation recommendations to the analyst.
Copilot's containment prompt book can list all the affected devices and accounts and provides recommendation to contain the attack. Which can also easily share the information with the one who needs further action.
CrowdStrike Charlotte can provide remediation actions through direct interactions.“What are the top recommended remediation actions for the impacted endpoints?”
SentinelOne triggers automatic responses to the threat or attack.
The platform could remove files from impaired endpoints and block the sender immediately in real-time, with minimal intervention from a human analyst.
Waste of time on responding for those large number and repeated alerts
Related Pains
In the user flow of previous research
F. Report and Metrics
Phase
Microsoft integrated with Office365 to generate PPT for reports.
Integrate with Office 365 to generate crafted reports that can be directly used for present or further operation.
Combined with the product of Microsoft Office 365, users can easily create PowerPoint slides to introduce or record an incident, allowing users to include various content via different prompts.
No suitable tools to generate traceable report instead of notes or screenshots
Related Pains
In the user flow of previous research
F. Report and Metrics
A. Planning
Phase
Microsoft applies auto training to improve the accuracy of detection & response reports.
Finetune detection model
Microsoft Security Copilot is designed to learn from past incidents to generate more accurate responses in the future. Learning is done through user feedback and the analysis of big data. This means that the detections it implements will be less noisy (reduced false positives), and you will be able to focus on real incidents that require your attention. The longer Security Copilot runs, the smarter it gets.
Generate detection rules
Security Copilot can also generate detection rules. For instance, create a detection that triggers when a specific new vulnerability is exploited, which saves the time and effort of manually researching the vulnerability and writing the rule, ensuring your environment is protected much faster.
Google supports model tuning directly by Human Language and provides related suggestions.
Users can easily refine the detection rule in human language, making the detection model meet the requirements more.
Also, Chronicle provides the recommended modification based on the detection result.
Need to manage detection model to reduce the false positive rate
Related Pains
In the user flow of previous research
Summary
The Key Value of Each Competitor
Microsoft
Security Copilot
Complete E2E copilot covering almost all AX phases and highly integrated with MS services.
CrowdStrike
Charlotte
Reduce the barriers for analyst doing advanced tasks.
Recorded Future
Recorded Future AI
Highly readable threat intel summary.
SentinelOne
Purple AI
Powerful threat-hunting story with automated response.
Chronicle
Optimize the detection by refining the detection rule.
The competitors mainly apply NLP in the
“Investigation" Phase
NLP assists analysts by clear guidance, coding, and organization.
Without NLP
Need to start on their own, collect all the information by jumping among consoles, and need to investigate by their owe judgment, otherwise, escalate to Sr. analyst.
With NLP
Have clear instructions at first, prompt to get all the needed information, and investigate by the recommended movements, at last, get the automatically generated incident-response summary.
Thus provide the 4 key values: Learn-free, Code-free, Leap-free, and Summary-free.
Learn Free
Recommend prompt to start, investigate and response.
Code Free
No need to code but prompt.
Leap Free
No need to jump among consoles.
Summary Free
Assist in event summary, and generate result.
Business Strategy
What action should TrendMicro Take?
What can Trend Micro do “More” to stand out?
Learn Free
Recommend prompt to start, investigate and response.
1. Integrate with our incident response playbook for automation.
Trend Micro has many kinds of playbooks that can guide our users to protect their cyber environment by following the process within it.
Thus we have a large advantage and potential to develop automatic investigation and response by NLP.
Summary
Free
Assist in event summary, and generate result.
2. Interprets our charts, and graphs for comprehensive understanding.
Trend Micro is an expert in creating investigation charts, graphs... and so on. If NLP technology can interpret these charts for analysts at the same time, it will help them quickly obtain more information.
Fill Up the Remaining Needs of SOC Manager
NLP can assist Managers in leveraging, summarizing, interpreting, and communicating to fulfill different situations.
3. Assist in Reporting to Senior Executives
F. Report and Metrics
Automated report generation
SOC Manager can use LLM to generate detailed SOC reports, including information on security incident trends, impacted systems and applications, attack types and patterns, and more. LLM can generate comprehensive and accurate reports based on historical data and current conditions, helping senior executives understand the current security posture and threat landscape.
Easily understandable summary
of threat intelligence
LLM can analyze large volumes of threat intelligence data and extract key information to assist SOC Manager in providing concise threat intelligence summaries to senior executives. This enables executives to gain a better understanding of potential threats and risks, allowing them to make informed decisions.
Predictive recommendations
Generate predictive recommendations based on historical data and trend analysis.
These recommendations may involve security investments, adjustment of defense strategies, improvements in vulnerability management, and other aspects, aiding senior executives in making wise decisions to enhance organizational security.
4. Assist in Enhance team efficiency
F. Report and Metrics
A. Planning
B. Dispatch/Assign
Review and monitor the metrics to evaluate SOC team performance
Using NLP to leverage...
-
Human-power bandwidth: How many members should assign to an incident/ alerts.
-
Team performance: MTT Triage, MTTR (first response action, exclude search, data enrichment,,,), MTT AutoClose, MTT ManualClose
Threat detection and alert optimization:
NLP can be used to optimize detection model. It can analyze and identify various threat indicators, assisting the SOC team in more accurately recognizing potential security incidents. This enables the team to respond faster and reduce falsepositives,therebyenhancingoverall response efficiency.
Design dispatch mechanism to enhance team efficiency
NLP can help to refine the dispatch mechanism due to member’s efforts, performance and current consideration:
-
Threat type
-
Severity of the alert
-
Seniority of analyst
Project Outcome
Made a breakthrough in
Cross-Functional Collaboration.
This research was invited to deliver a presentation to an AI technology team, and its achievements were highly praised by the engineering members, indicating its integration into future development directions.
Moreover, leveraging this research as a foundation, researchers began extensive collaboration with the engineering team in the 2023 AI competition, collectively creating AI tools that meet user needs.
Thus, it is evident that this research has not only brought a significant impact on the UX Designer within our HIE department but also established unprecedented cross-departmental collaboration bridges.
Our proposed strategies
have been
Officially Released!
In addition to many competitive applications being adopted, 2 strategies proposed from our research, such as "Explain charts & graphs" and "Generate reports", have also been adopted and officially released!
Personal Reflection
Combine competitor research and interviews
to enrich the research insight
This study combines the application of the latest technology with the identified pain points in user workflows(our previous research), leading to the identification of highly promising development directions for TrendMicro, which received significant recognition. And to support colleagues less familiar with LLM NLP technology, I designed an introductory course as part of the research report to enhance their understanding.
However, what I find somewhat regrettable is that this study primarily focuses on competitive analysis, but most competitors have not yet developed fully functional products. Therefore, we could only rely on official press releases as data sources for our research. Despite our efforts to filter the data sources, I believe this still affects the authenticity of the study to some extent.
Furthermore, if time permits, I believe conducting interviews with customers currently using products from multiple cybersecurity companies could greatly enhance the reference value of this research. We could inquire about their perceptions of this technology, their expectations, and their views on strategies proposed by other companies, potentially enriching the insights of our study.