Welcome to Understanding Link Analysis. The purpose of my site is to discuss the methods behind leveraging visual analytics to discover answers and patterns buried within data sets.

Visual analytics provides a proactive response to threats and risks by holistically examining information. As opposed to traditional data mining, by visualizing information, patterns of activity that run contrary to normal activity surface within very few occurances.

We can dive into thousands of insurance fraud claims to discover clusters of interrelated parties involved in a staged accident ring.

We can examine months of burglary reports to find a pattern leading back to a suspect.

With the new generation of visualization software our team is developing, we can dive into massive data sets and visually find new trends, patterns and threats that would take hours or days using conventional data mining.

The eye processes information much more rapidly when information is presented as images, this has been true since children started learning to read. As our instinct develops over time so does our ability to process complex concepts through visual identification. This is the power of visual analysis that I focus on in my site.

All information and data used in articles on this site is randomly generated with no relation to actual individuals or companies.

Using Visual Analysis for Network Threat Detection

The amount of data accumulated in network threat detection is massive. Even during live captures of network traffic, being able to identify emerging threats and irregular activity is difficult using traditional data mining, filtering and legacy visual analytic programs that were all designed around a time when data logs were significantly smaller and threats less complicated.

I will address the challenge of network threat detection through next generation visual analytics using two different scenarios commonly tasked to network threat analysts and investigators; mal-ware forensics, network visualizations and IRC or chat log analysis.

Mal-ware Forensics and Network Threats:

Determining the anatomy of network threats and mal-ware requires the network analyst to understand how the threat originated, spread and ultimate impacted the users. Stopping or mitigating network threats additionally require determining the "signature" of the mal-ware, network vulnerabilities and establishing appropriate countermeasures to make the threat benign.

Through visual analytics, the first step is gathering all the data available from network packet logs and server logs to get an overview of the issue. In this example we are going to deploy SynerScope to assist us with our visual analysis of a network attack by uploading our network log information (over 150,000 lines of data) to provide a holistic view of the threat.

SynerScope has several interactive representations for the network analyst, first is the relationship view which shows which geography and IP addresses are related within the network logs. Second, SynerScope displays the direction of travel between the related entities with red being the receiving and green the sending network points. Third, SynerScope provides an interactive visualization of sequence of events to the right which allows us to simultaneously view relationship, direction and sequence.

Immediately apparent from the holistic or full view is that there is a command and control server which has a large amount of centrality within our visualization. Sending to the command and control server are two groups of IP addresses located in two specific geographies. These two area represented with the cluster of green bundles are only sending information to the command and control server, however are not receiving any information.

Within the sequence view, we can see a group of structured network traffic sending and receiving to the same area at the earliest point within our data. By drilling into that block of structured traffic, we can see within the relationship view how the planning and execution of the attack began.

Moving forward within the sequence visualization, we can see how once planning and execution is complete, how the command and control server begins to reach out to infected. Through visualization of the data within SynerScope we can follow the progression of the network threat as the command and control server begins compromising networks in different geographical areas.

Network Visualization

To aid network security and threat analysts, leveraging visual analytics to gain a real time understanding of network traffic is essential. Accomplished in the past through data mining and statistical analysis, this process could not occur in close to real time and was a manpower intensive task undertaken daily.

Through SynerScope's visual analytic capabilities, I can feed in information from my network and gain a visual representation of the traffic that is occurring at any point in time to look for potential network threats and irregularities that need to be addressed.

I begin by uploading network activity logs into SynerScope to gain a holistic view of all network protocols, IP's and devices currently utilizing my network to look for signatures of concern.

By isolating down on certain protocols or IP addresses through the relationship and sequential views within the program and utilizing the color coding of direction provided by SynerScope I can drill down into specific events that I want to further examine.

Once I have isolated a specific signature of concern, I can execute the data stream view within SynerScope to examine the raw underlying data constructing my visualization to confirm or refute if the signature contains a threat to my network or is benign activity that I do not need to be concerned with.

Chat and IRC Log Visualization

Those tasked with investigation and forensic discovery within IRC or chat logs knows that determining the activity, flow of conversation and content can be a monumental task based on the volume of information that is present.

By utilizing visual analytics and the capabilities within SynerScope, I can save countless investigation and forensic hours by quickly determining through visual representations of velocities and structure which entities have the greatest amount of betweeness or centrality within the social network of the logs, the direction of communication that is occurring and the time the conversations are occurring interactively.

By importing the log data I have for my investigation and leveraging SynerScope's visual analytic tools to look for velocities and direction of travel I can quickly identify those individuals engaged in the most activity within the data.

By hovering over the bundles within the relationship view, I can view the time and sequence of the conversations between the active individuals to determine the rate, dates and time the persons of interest are most engaged in communication.

Finally, by drilling down on the individuals who I am most interested in for my investigation, I can execute SynerScope's data stream view to visualize the conversation in order of which it occurred to determine if the content of the communication is relevant to my investigation or if I should continue on to other entities within my investigation.

The following video provides a comprehensive look at SynerScope for Network Threat Detection:


Through the examples within my article, you can see how leveraging visual analytics in network investigations can aid in more rapid discovery of threats and irregular activity. It can allow analysts and investigators to save countless hours by allowing immediate discovery of relevant activity and can increase understanding and solutions to addressing network threat issues more rapidly mitigating the threat and minimizing exposure.

** Note ** Some image items within this article were blurred or distorted to protect private information

Visual Analysis of Large Datasets

Challenges to Visual Analysis for Large Datasets

While data has been continually growing, the tools which to visualize and discover what is in that data have not significantly changed over the past ten years. Analysts are challenged with larger and larger amounts of information in which to find problematic trends and patterns, yet the tools they are using have not evolved to meet the demand.

Today’s analysts need the ability to visualize their entire landscape of information, not bits and pieces. The complex nature of fraud, internet crimes, network threats and finance demands a visualization tool that can holistically view and leverage the vast amount of information that is being generated in the technical age.

When an organized criminal ring is targeting your organization, hours could mean millions of dollars, yet those who are tasked with discovering these trends are outmatched when using today’s visual analysis technology costing time, valuable resources and your organizations assets.

Today’s online and mobile driven transactions have opened the door to a new generation of fraud. Hidden behind a veil of assumed anonymity and the ability to operate anywhere in the world has led to an increased emergence of organized threats to businesses who’s front door to their goods and customer information is no longer protected by brick and mortar.

Twenty years ago, credit card fraud was face to face crime carried out one by one against their victims. A credit card fraud ring used to rely on theft and counterfeiting to carry out their crimes. The complexity and risk involved limited their ability to conduct all but a few transactions per day.

At the same time, businesses were conducting only several hundred transactions a day at their physical locations. Detecting fraud was a task made much simpler by simple record review and investigation. Trends were much easier to spot as the number of fraudulent transactions were few and easily stood out among the legitimate within the documentation.

Twenty years later fraud is no longer a face to face crime. An organized credit card fraud ring can operate from thousands of miles away. The only risk and overhead involved is a laptop and an internet connection. Businesses are not processing hundreds of transactions a day at their stores, now it’s thousands at a location that is open 24 hours a day 7 days a week.

Thousands of transactions a day means thousands of lines of data, each containing a multitude of attributes now captured in eCommerce. Buried in among these gigabytes of attributes hides fraudulent transactions, but finding them has become the modern day equivalent of looking for a needle in a haystack.

While the world of organized financial fraud has become more sophisticated and wide spread, the tools to detect them have lagged behind, still firmly rooted in the days of brick and mortar face to face crime.

Fraud analysts and your business need a way to level to playing field. Businesses demand better ways to identify fraud trends much faster as in today’s eCommerce, hours can mean thousands of dollars in exposure.

SynerScope was created to quickly find the needle in the haystack of your transactions. Realizing that analysts need as wide of a view as possible of the important attributes within your data that identify fraud, SynerScope has been designed to visually show the relationships in your data and the sequence in which they occur over a much larger period of time.

Analysts require as much information as they can get their hands on to tackle today's threats. After all, how can you find the needle if you are only looking at a corner of the haystack.

Visual analytics in the past has relied upon finding relationships between entities through a one to one representation. A transaction is associated with an IP address, that IP address is linked to ten other transactions each represented as an individual entity. While this method was efficient when data and transactions were relatively small, it was inefficient when dealing with large amounts of information.

SynerScope, through the use of forced edge bundling and representing data through specified hierarchies allows the analyst to visualize concurrent relationships over a significantly larger set of data then previously possible. The larger the amount of data which can be visualized, the longer the period of time which is available to the analyst. This is key to the analyst’s ability to visually detect trends which run contrary to the normal flow business leading to much faster proactive identification of threats.

Just as important as the relationships which exist between entities within the fields of data is the time and sequence in which they occur. This has previously been overlooked in legacy visualization software where the analyst was able to visualize relationships or events over time but never both simultaneously.

Today’s fraud analysis requires the ability to detect fraud both as the events relate to one and other but also the sequence in which they occurred. SynerScope provides the ability for the analyst to leverage relational and sequential visual analytics within one workspace increasing the field to a complete 360° representation of the data.

SynerScope has been designed to give the analyst all the resources required to visualize activity and relationships for effective identification within one workspace.

Within SynerScope's primary analytical space the analyst can view, drill down and manipulate their visualization through:


Sequence of Events

Time of Events