In part one of my look at applying application performance monitoring in a big data environment, we examined how nonrelational databases work and what unique challenges they present to traditional application performance monitoring tools.
Because of their non-ACID approach, I argued, application performance metrics become very complex in a system where the business transaction itself has a expected non-zero but small probability of failing.
The first part of the problem is applying the right kind of tool. By their nature, big data environments are often very complex because all of the layers (data storage, network controllers, and application servers) have to be scaled up to contend with delivering data back and forth in a manner that approximates a reasonable amount of time. “Reasonable” in this case, is something like bazillion-record data searches in subseconds.
Not to mention how these environments must be scaled out. As more data is added, more machines have to be added to the environment, data repartitioned, application servers reorganized… the works.
Proper application performance monitoring must therefore be able to contend with all of the changing levels of a big data environment, because targeting the application alone is not going to cut it. End to end monitoring that scales along with the given environment is an absolute necessity, because otherwise your application data will never truly reflect what’s going on.
The solution must also deal with an environment that doesn’t preclude multiple applications. It’s never going to be just one application working to deliver a particular service: it will often be many applications working in concert. Like an orchestra conductor who can tell when a particular instrument is out of tune, a properly deployed APM solution must be able to “hear” the bad note in a complex symphony of cloud and data center systems.
Big data, more than anything, is why business transaction monitoring is so important. Tracking the transaction as it moves through such environments, and seeing how it touches all the different applications and systems along the way is the best way to determine application performance. If a problem occurs, it will sound that bad note, a clunk that you can zero in on and fix.
All of this has to be done while being able to scale and not sticking a performance hit on the environment, which is certainly a tall order.
The problem is expected to be more complex, not less, as more and more businesses enter the online business market, but it’s a challenge that APM vendors are rising to meet every day.