Transcript
1
Introduction
Web applications today are part of every IT operation within an organization.
Independent software vendors (ISV) as well as enterprises create web applications to support their
customers, employees and even suppliers. All in all, the goal of these applications is to deliver a
service, complete a transaction, support efficient and effective business operations, and directly
impact company revenues and operational margins.
In 2009, Paypal went offline for over an hour due to a network infrastructure failure. At the time,
Paypal was processing about $2,000 in payments every second, which meant its hour offline
resulted in about $7.2 million in lost transactions. According to an Aberdeen research study, this
type of downtime costs over $150K per hour, and the average recovery time is 5.13 hours. In
severe cases such as Paypal’s, IT teams, which include operations and development, need to know
what’s wrong—and fast. In these moments, a team’s ability to collect, analyze, and understand
data in real-time is fundamental to resolving the cause of the problem, taking action and validating
remediation. In this article, we define real-time analytics and demonstrate how log data from
different layers of your systems and application stack can enable real-time analytics and response.
2
What is Real-Time Analytics?
When referring to “analytics,” people often think of manipulating an existing set of
structured data to yield insights. “Real-time analytics” takes this definition a step
further by accounting for the constant appending of new data to the existing data set
and continuously re-analyzing the new dataset for new insights. But for analytics to be
real-time, data needs to be ingested immediately upon creation, delivering results in a
matter of seconds, enabling those interpreting the data to react right away.
To further demonstrate the definition of real-
ultimate source of real-time data, with systems
time analytics, let’s start by comparing it to the
often producing hundreds or thousands of
more commonly known, data batch processing.
log events per second. While a variety of tools
While batch processing can still append new
exist for capturing log data, a tool’s ability to
data to an existing set, it does so in batches
ingest and interpret log events in real-time as
rather than a continuous stream. Batch
they occur is a key differentiator, with many
processing comes with several disadvantages
tools taking anywhere from several seconds to
to real-time streaming. For example, if the data
several minutes to process a log event. A log
being processed doesn’t include timestamps,
management tool that isn’t truly real-time fails
every event in a batch will be assigned the same
to capitalize on the true power that log analytics
timestamp (the date and time the batch process
has to offer. How effective can a system alert
occurred). Batch processing also makes it
be if you’ll still experience several minutes of
impossible to generate immediate alerts off of
downtime before even receiving the alert?
events as they occur. In cases like Paypal’s, or
Tools that are actually real-time can deliver
any other web-based business, batch-process
information within seconds of occurring,
alerting is simply unacceptable.
alerting you to the warning signs leading up to
an issue, improving your chances of identifying,
In the world of operations and systems
diagnosing and resolving problems before they
administration, log data is often seen as the
negatively impact end-users.
3
Four Real-Time Use Cases
Below, we explore four use cases that exemplify why real-time analytics are critical
to performance and user experience, highlighting key capabilities that enable realtime analytics in each layer of your system or application:
1. The Application Layer
as they occur. Using a log analytics tool that
With your developer team preparing for
offers “anomaly alerts” can help you identify
a big push to production, you’re worried
early warning signs of larger issues.
about the possibility of unforeseen issues
deployment.
Regardless of the size of the release, whether
Testing in development will never provide
it’s a minimal viable product released to a
an exact replica of what will happen in
subsection of users or a large release following
production. Therefore, the more you are
a three month sprint, things inevitably go
able to view and monitor your logs in real-
wrong. Logs should be your go-to resource
time, the faster you will be able to address
for investigating and addressing the source
and rectify issues. While big issues may be
of issues that might arise while a real-time
easy to spot, real-time analytics can also
analytics tool should be used for alerting you
help you identify small issues building over
to anomalous activity as it occurs.
immediately
following
the
time that could eventually slow down your
application and user experience. While
batch-processed analytics could only ever
give you a historical analysis of your systems
data, real-time analytics can enable you to
identify anomalous patterns in your data
“
The easier it is to view and
monitor your logs in real-time the
faster to will be able to address and
resolve issues.
4
2. The Database Layer
database errors stream into the same single
Imagine over the course of several minutes,
view with the rest of your system’s log events
your popular e-commerce application hasn’t
as they occur. Alerts on database errors can
received any orders. Where’s the first place
be generated just as easily as alerts for the
you’d look for a possible issue? You may first
rest of your environment. And tools that offer
check to see if your website is still reachable
custom tagging of specific event types can
from a browser. Then, you may check your
also help you spot database specific errors as
server logs. Or perhaps you check your
they occur.
APM tool? Or a web analytics tool? Are they
all saying the same thing? Or nothing at all?
“
How much revenue have you lost
When you notice there aren’t any errors in
while guessing where to investigate the
your code and traffic to your website appears
problem?
to have remained steady, you decide to
investigate your database. Only then, after
wasting time investigating other scenarios,
3. The Server/Hosting Layer
do you see your database was improperly
Let’s say your mobile app was just featured
configured in the last deployment and has
on Product Hunt and you’re suddenly
reached its row limit. How many sales have
experiencing a spike in traffic. Luckily, your
you lost while guessing where to investigate?
app runs in an autoscaling environment and
handles the load without issue. When the
analytics,
traffic later subsides and your servers scale
database errors can go undiscovered, often
back, you decide to analyze the distribution of
only realized after a period of noticeable
400 errors over time. But how will you access
inactivity and investigating. When using a
data from the servers that scaled down?
real-time aggregated log analytics service,
If you weren’t sending those log files to a
Without
log-based,
real-time
5
central location in real-time, your data is
in real-time? Many tools only offer email
forever lost. In this scenario, centralizing your
notifications; but what if your team only
logs in real-time is crucial to capturing all
checks emails a few times a day? Alerts are
relevant data.
only as good as your team’s ability to react
to them quickly. Therefore, it’s important to
When dealing with auto-scaling environments,
use services that easily and automatically
real-time analytics also enables you to
integrate with the communication tools
monitor the scaling processes in the moment,
they’re already using, like Slack, HipChat or
ensuring they scale as expected. Without
PagerDuty.
real-time monitoring, you’d have no way of
knowing if a server cluster fails to scale up.
Cross-system, real-time analytics can also
Likewise, if your environment fails to scale
be demonstrated when integrating multiple
down when appropriate, you could be left
monitoring tools. Consider New Relic, which
paying for unused infrastructure.
offers real-time application performance
“
monitoring: what if one of your application
Real-time analytics enable you
to monitor the scaling process, in
the moment, to ensure everything is
scaling as expected.
metrics recorded in New Relic requires
deeper investigation? Manually switching
from an APM to a logging tool to then search
for related log events eats away at valuable
time. In these cases, plugins that integrate
APMs and log analytics tools for immediate
4. Cross System, Real-Time Analytics
log event correlation enables real-time action.
Generating real-time alerts are an obvious
necessity for reacting to issues as they occur.
But what if your team isn’t seeing the alerts
6
Final Notes
When it comes to leveraging analytics to ensure continuous service delivery and uptime, loglevel data and real-time processing are two requirements for success. At every layer of your
system’s hardware and application stack, real-time analytics enable centralized log collection
and monitoring, easy identification of key events, and instant alerts to the communication tools
used by your team. Thanks to the power of real-time analytics, you can improve your operational
efficiency and prevent downtime, leading to a healthier, stronger business.
About Logentries
Logentries is the leading real-time log management and analytics service built for the cloud,
making business insights from machine-generated log data easily accessible to development, IT
and business operations teams of all sizes. With the broadest platform support and an open
API, Logentries brings the value of log-level data to any system, to any team member, and to a
community of more than 35,000 worldwide users. While traditional log management and analytics
solutions require advanced technical skills to use and are costly to set-up, Logentries provides an
alternative designed for managing huge amounts of data, visualizing insights that matter, and
automating in-depth analytics and reporting across its global user community. To sign up for the
free Logentries service, visit logentries.com.
Cerspere
de platusd
anducim
Start your
free Logentries
trialagnistdfds
today
Get Started with your free logentries account, you will be up and running in minutes.
Free Trial includes access to ALL Logentries features.
*No credit card required!
7