Neoload And Citrix: The Performance Revolution Is Here

Neotys has announced support for the Citrix protocol as BETA in the Neoload 7.1 release. What does this mean for performance engineers? What does it mean for the software testing industry? I think I know…

How did we get here?

My experience with Citrix goes back almost 20 years. I was there in 2002 when Mercury first launched their Citrix protocol. I still have the original PowerPoint slide deck that Simon Berman gave in downtown Nashville about it. I am such a geek I still have a picture to remind me of the event:

I was working at a global systems integrator years before I would eventually launch my own consulting company, Loadtester Incorporated. There were a lot of legacy applications and we were migrating many of them to Citrix Metaframe at the time. Before then, the only way to attempt a performance test was a combination of LoadRunner and WinRunner. Anyone else remember those days? I actually had to do a few of those exercises as a consultant. Fun is not how I would describe it. When native Citrix protocol support came out, it solved a big headache. One of the first employees at Loadtester (Tim Chase) had so much experience testing Citrix applications, he wrote one of the first white papers on the topic. For many years this document has been shared to educate on the basics of load testing Citrix. I’ve also posted several blogs on the topic. Multiple times I have found myself doing Google searches during an engagement and used my own blogs to get myself out of a jam. Once you leave the web testing world and venture into other protocols like Citrix, the gene pool gets pretty shallow and there isn’t a ton of information on it like there is for HTTP/HTML.

Lack of Choices

LoadRunner has traditionally been the only serious choice for performance testing Citrix applications in the Enterprise. A handful of challengers came and went. This includes Citrix themselves, who acquired a company specifically to fulfill the need in the market. In 2007, Citrix purchased the “TLoad” testing tool from Edinburgh, Scotland-based ThinGenius. It never gained a major audience in the market, and ended its short existence in 2013.  Today, a handful of performance testing products do support the Citrix protocol, but none with a customer base the size of LoadRunner. Having went through demos or proof-of-concepts with many, my opinion is that they are not a holistic/complete solution tying into the bigger ecosystem of Enterprise level testing. The underlying Citrix technology and ICA protocol hasn’t changed in many years. This is good and bad. Development maintenance is minimal, but existing vendors have had little motivation to improve upon their offerings because of it.

The New Kid in Town

And then there is Neotys Neoload. I have been watching this company closely for the last five years. I call them “the little engine that could”. As I look across all companies focused on application performance testing, who is bringing innovation and leadership? My answer is always Neotys. I haven’t seen the kind of passion to further advance performance since the early Mercury days. Since this is where I make my bread and butter, I’m paying attention. If there is any product set to take the mantle from LoadRunner, it is Neoload. It is sad to watch HP/Micro Focus take a market-leading product like LoadRunner and treat it like some troublesome foster child  – part of a larger burden it must bear – and see it banished to the land of maintenance cycles. With this latest announcement from Neoload, if there are any doubts to their commitment to support the Enterprise, Citrix is the convincing shot across the bow. As Joe Biden once said, “This is a big ****** deal!” In the analogy below, I am on the right whispering into the Neotys R&D staff on the left…

In Neoload version 7.1, Neotys is offering Citrix support in BETA to get feedback from early adopters. I am one of those early adopters. I have seen it, and it is more advanced than I expected in a BETA version. All that is needed is feedback from customers to ensure it meets any edge cases. I’m in the process of documenting my experience with it. Here is a video I made to give you a basic introduction and my thoughts on this new recording functionality:

In short, its working well. I’ll have another one test execution in the future.

Big Deal

What makes this a big deal is the reduction of time spent in the scripting/automation phase. I am as geeky as the next person wanting to look at code, but under the deadline of a testing project I want to get into the test execution phase as quickly as possible. For years, the project timeline for performance testing was extended because there is a lot of error handling and extra logic in the automation to deal with how picky Citrix is. All you have are bitmaps to deal with, and the ICA protocol doesn’t really expose much in the way of metrics or information. If a pixel is wrong, if another server has a config setting different, if the video drivers are acting stupid – any of this can cause scripts to fail. Bitmap hash values can change seemingly at the whim of the server. Those of you who have done this in the past know the joys and pain of which I speak. This is mainly due to the way LoadRunner is written to work. Because Neoload takes a visual test design approach, this works well for Citrix automation. The image below shows a list of the current functions.


Neoload Citrix Functions

The new Citrix functionality allows the automation to be as code-free as possible. Dealing with troublesome visual or text synchronization can be done mostly from the Neoload UI and setting configuration options. As more feedback is given on additional use cases, more functionality can be built out to deal with it. It already makes dealing with the Citrix StoreFront as simple as recording any other web application, seamlessly switching over into the ICA session. It already knows about the typical correlations you will encounter (like the CSRF token) in the Framework rules handled in the post-script wizard. If you still need to test an older version of Citrix with a static ICA file, they support this as well.


Neoload Post-Recording Wizard For Citrix Correlation

Real-Time OCR

The built-in OCR capability is fast and accurate. This feature allows you to capture an area of a bitmap displaying text, and convert it to the ACTUAL text in REAL-TIME so you can be sure the textual value is on the screen before continuing to run the automated process. I really like that you can tell immediately how well it is converting that specific font. You don’t have to run the script through to find this out, as with LoadRunner. That feature alone should wake some people up.


Neoload Real-time OCR Recognition

Neotys have thought about the feature set and design from the beginning. If you are looking for a solution for performance testing Citrix deployed applications, or you aren’t happy with the one you have – you need to look into Neoload. With the SAP GUI support release, Neotys notified the Enterprise customer “We’re here for you!”. With the Citrix support in this release, they are saying “and you can no longer ignore us!”. This is the perfect time to release this capability. With 98% of the Fortune 500 using Citrix, it is not going anywhere anytime soon. There needs to be more testing of Citrix-deployed applications, so this gives people another way to do that. Another way to think of this capability is that of a “catch all” protocol solution. Even if you DON’T publish the app through Citrix in production, you can still reduce your testing time by doing so in the test environment. If you can publish it in Citrix, you can test it.

My Conclusion

For years I would respond to the touts of new performance testing products about how awesome they were and how the “old dinosaur tools” were considered useless. I would always ask the question: “So… can I test Citrix with this?” It was obvious they only supported web, so when they said “No.” I would respond, “Then come back to me when you are ready to support me as an Enterprise customer. I have way more than just web to worry about.”  You can no longer put Neotys in that category. If the BETA in this new release is any indication of where Neotys is taking things – pretty much everyone else needs to get out of the way. The performance testing revolution is already here. I’m in. Let’s roll.

About the Author

Scott Moore

Scott Moore specializes in application performance engineering and testing. This includes education, hands-on implementation, and application performance monitoring.