T2 Systems' deep-dive into Oracle application performance management began like many do - some end user customers started complaining that the system was running slow. And so the mystery began: What was the problem, and what was causing it?
The Indianapolis-based company offers technology and a hosted environment for companies, universities and government agencies to manage all their parking-related activities. That includes functions such as ticketing, permitting, access control and special event parking. T2 Systems has about 360 customers throughout the U.S. and Canada, according to CIO Jim Hutchins. About half of them are hosted in T2's own data center.
"We were challenged frequently to find out if the experience was slow, or if the system was slow," Hutchins said. Sometimes the system or Oracle Database was running slowly. Many other times, performance was slow because the end user company was using up all its bandwidth elsewhere.
"We wanted to be able to quickly determine for ourselves and the customer whether to look inward or outward," Hutchins said.
Application performance monitoring is becoming a crucial part of managing the IT environment. According to a survey last year by France-based Orange Business Services, almost 55% of IT shops currently use application monitoring in some way. But Hutchins and experts say looking for the problem often crosses technology tiers, and investigating can quickly become complex.
Eric Hanselman, networks research director for The 451 Group, said application performance management has come to the forefront as IT architectures have evolved. According to him, applications used to be tightly controlled. You had an internal data center with system administrators and a team keeping an eye on the databases, a situation that he said was "reasonably well constrained." But as server virtualization came into the mix, as well as the growing popularity of hosting and cloud technologies and more mobile user communities, the number of variables to keep track of grew alongside it.
That made monitoring an application problem more difficult. A mobile end user might experience slow application performance, but there could be many reasons for it. Maybe the mobile device isn't working properly. Maybe the mobile network is slow. Maybe the application sits on a virtual server in a database and isn't getting the resources it needs to perform. Maybe the physical server is just old and in need of a refresh. Maybe the internal data center network is running slow. Or maybe, just maybe, the Oracle Database supporting the application at the back-end isn't performing as it should.
"As we layered on more abstractions around how to deliver IT services, it has become harder to pinpoint the nature of that problem," Hanselman said.
Such was the case with T2 Systems. As a hosting provider, the problem could lie anywhere between the amount of memory on one of its own servers to whether an end user company's employees were watching too much streaming video of The World Cup in their office.
T2 Systems is mostly running Oracle Database 11g R2 on Windows - as well as Oracle Enterprise Manager - and so it looked first to Oracle for a solution. But it found that Oracle was "focused almost exclusively on the database," Hutchins said.
"I would say that maybe a third of problems trace to the database," he said. "The other two-thirds were things outside the database."
Hanselman said Enterprise Manager does a great job of assessing all the Oracle pieces of the puzzle, but not as well with anything outside of the Oracle sphere - for example the networking to branch offices. That can make investigating the problem more manual and time-consuming.
So the company looked elsewhere. It eventually came upon ExtraHop, which can look at multiple IT layers - the application, database, network and storage - to diagnose a performance problem. Hutchins said he liked how he was able to look at several layers on one screen without having to log into three or four different servers.
At the same time, Hanselman said IT organizations have to make sure they don't get too siloed so as to prevent communications between departments. Application performance monitoring tools can help mask an underlying issue, which is that database, server, application, networking and storage teams aren't talking to each other as much as they should.
"Once organizations reach sufficient size there is a tendency to silo information that can hinder the problem-solving process," he said.