Start With A Non-Standard Application
I had to create a web virtual user script for a desktop application that made REST API calls to a Microsoft IIS web server over HTTP. The company had created a Microsoft C# EXE test harness, which made the calls automatically launched from the desktop. These calls fire off a long list of stored procedures in a particular order on a Microsoft SQL Server database. To load the database, I would need to create an HTTP script that caused the same “kick off” activity to the database.
There were three ways I thought of to test this unique application:
1 . I could get a trace file from SQL Server that shows the output after the EXE runs, and convert this trace file into a list of SQL executable code that would allow me to create an ODBC Vuser in LoadRunner. An attempt at this was extremely messy. I ended up with over 34,000 lines of code to sort through, and I would be manually executing these in the right order through an ODBC connection application such as Microsoft Query. Appropriate syntax in Vugen would be important. This was too much work to try and get done within the time I had. From a script maintenance perspective, this would be messy as well. What if they changed any of the SQL Stored procedures, names, or the order they ran in? There had to be a better way.
2 . Since the EXE code was done in Visual Studio with C#, we could create a Template Vuser in LoadRunner using the Visual Studio ADD-IN offered in LoadRunner. This would essentially allow us to build a custom Vuser with transactions in their code, and could be run from within the Controller. Once a developer understands how to do this, any code maintenance to the EXE could be updated as a Vuser and script maintenance becomes very easy. Unfortunately, the Template Vuser protocol is not included in the Community Edition of LoadRunner, so a proof of concept for this would require a purchase of a license (Virtual User Days, Term, or Permanent licenses). It’s not a huge cost for a small amount of Vusers, but it is not free to try. This would be a backup in case I could not get the HTTP REST version of the script to work.
3 . Using the Remote Desktop Protocol, I could have scripted against the EXE in a terminal session. It would have been a very easy script. I would open a terminal session and launch the EXE, let it run until completion, and close it down. The customer would have to put together a Windows Terminal Server OS with enough sessions to mimic the amount of load. Each session would then have an RDP script run against it and load the back end that way. However, this required a significant setup and configuration of resources. This would be a concurrent load of 600 virtual users.
We decided to try and get the HTTP REST calls to work first before resorting to the other three options.
Deep Dive Into The Protocol
Since the EXE made calls over HTTP, the first attempt to record the application was using a standard Web/HTTP Vuser and utilized the LoadRunner internal proxy recorder to capture the traffic.
I ran the EXE and tried to capture the traffic in Vugen. This proved to be problematic, as the LoadRunner proxy could not deal with two specific headers that were being added to the first requests. Below is a sample of the Vugen code generation log:
[Proxy Recording (1ff0:1824)] Started proxy on port 8888 in synchronous mode [Proxy Recording (1ff0:1078)] Client request from @ 192.168.5.44:19058 [Proxy Recording (1ff0:1078)] Client request from @ 192.168.5.44:19061 [Proxy Recording.Error (1ff0: fb8)] Cannot add request header message: The 'Transfer-Encoding' header must be modified using the appropriate property or method.Parameter name: name [Proxy Recording.Error (1ff0: fb8)] Cannot add request header message: The 'Expect' header must be modified using the appropriate property or method.
I opened a ticket with Support, and was told that there were known issues with the proxy recorder. I am not really sure that solves my problem, but I journeyed on. The next thing to try was a Wireshark sniffer trace capturing all network traffic as the EXE ran. The one thing I knew was that the EXE was communicating to a specific IP address (192.xxx.xxx.xxx) on port 80 via HTTP. After creating a capture file in Wireshark (.PCAP), I applied a filter to only see the HTTP traffic to that server IP:
I see TCP and HTTP traffic, with POSTS being made to that IP. The sync requests are exactly what I am looking for. If I follow the TCP stream:
I see more details about the HTTP requests:
This appears to be EXACTLY what I am looking for. Since I have all the HTTP requests in the conversation to the server from the EXE, I can import this into LoadRunner Vugen. I selected the MOBILE protocol for the functionality I require:
After creating the initial Vuser, but before going any further, you will most likely want to set the recording options to your preferences. I recommend using web_custom_requests only because you never know what you are going to see when the import is done:
When you select “Record” from the menu options in this protocol, you will have multiple options:
I selected Analyze Traffic. From here I can point to my captured file and apply Server and Client filters (since I know the IP addresses going to and from each way):
Click “Finish” and see what Vugen creates:
That looks like a script to me!
Check Your Headers
Note that there are three headers; Accept, Expect, and Transfer-Encoding. Remember our LoadRunner Proxy log?
[Proxy Recording.Error (1ff0: fb8)] Cannot add request header message: The 'Transfer-Encoding' header must be modified using the appropriate property or method.Parameter name: name [Proxy Recording.Error (1ff0: fb8)] Cannot add request header message: The 'Expect' header must be modified using the appropriate property or method.
This indicates that the LoadRunner proxy does not work the same as Wireshark, but Vugen can handle those headers when imported from a PCAP. The real kicker here is that when I went to play this back, I got a warning and an error:
web_add_auto_header("Transfer-Encoding") started Warning -26593: The header being added may cause unpredictable results when applied to all ensuing URLs. It is added anyway web_add_auto_header("Transfer-Encoding") highest severity level was "warning" web_custom_request("syncrequest") started Error -26631: HTTP Status-Code=400 (Bad Request) for http://192.xxx.xxx.xxx/CCS.Sync.Services/api/sync/syncrequest
Hmmm – let’s just comment out the Transfer-Encoding header and try again:
web_custom_request("syncrequest") started t=2168ms: Connecting [0] to host 192.xxx.xxx.xxx:80 t=2171ms: Connected socket [0] from 192.xxx.xxx.xxx:65287 to 192.xxx.xxx.xxx:80 in 2 ms t=2171ms: 345-byte request headers for "http://192.xxx.xxx.xxx/CCS.Sync.Services/api/sync/syncrequest" (RelFrameId=1, Internal ID=1) POST /CCS.Sync.Services/api/sync/syncrequest HTTP/1.1\r\n Content-Type: multipart/mixed; boundary="cc3622c3-5406-4c37-b523-ba4028ec8d02"\r\n
It worked!! I began to see responses in my output log when playing back. The BODY of the Request and Responses has a LOT of dynamic data to sort out. This includes many different kind of ID’s, keys, and dates. Here is an example of the first Response:
t=2198ms: 1566-byte chunked response body for “http://192.xxx.xxx.xxx/CCS.Sync.Services/api/sync/syncrequest”
{“Key”:”00000000-0000-0000-0000-000000000000″,”Token”:
{“ClientIdentifier”:”055b267b-8e07-4848-bb28-849dde9b2254″,”ClientIdentifierHash”:257715815,
“DeviceIdentifier”:”3f5a67d5-9980-4c8f-ac2b-7c7cffb64f57″,”DeviceIdentifierHash”:-623236339,”
DeviceTimeZone”:{“StandardName”:”Central Standard Time”,”DaylightName”:”Central Daylight Time”},
“RequestTime”:”2016-03-18T17:38:07.5267569Z”,”OriginalRequestTime”:”2016-03-18T17:38:07.4955569Z”},
“DeviceLastSyncUtc”:”2016-03-08T12:38:00″,”Type”:2,”
UploadManifestKey”:”00000000-0000-0000-0000-000000000000″,”UploadManifest”:null,”DownloadManifestKey”:
“00000000-0000-0000-0000-000000000000″,”DownloadManifest”:null,
“TransactionList”:[{“Key”:”29292707-73ed-4ff5-9080-45598682a732″,”Request”:1,”RequestParameter”:null,
“RequestUtc”:”2016-03-18T17:38:07.5267569Z”,”Response”:3,”ResponseUtc”:”2016-03-24T16:36:39.1069632Z”,
“Status”:9,”DeviceNotes”:[],”SyncServiceNotes” :
[“Processing Failure 1: Validation Error: SecurityToken Validation Error: Token Request Date Expired.
Maximum token age in minutes is [5]. Token Age is [5.22:58:31.5802063].”,”Sync Request
Cancelled Due To Processing Errors”]}],”TransferPackageKey”:”00000000-0000-0000-0000-000000000000″,
“TransferPackage”:null,
“ProcessingErrors”:[“Validation Error: SecurityToken Validation Error: Token Request Date Expired.
Maximum token age in minutes is [5].
Token Age is [5.22:58:31.5802063].”],”ValidationErrors”:[“SecurityToken Validation Error:
Token Request Date Expired. Maximum token age in minutes is [5]. Token Age is [5.22:58:31.5802063].”],
“SyncSettings”:null}
As we begin to break this down, we see Client and Device IDs and hashes. We see different keys (some are zeroes and others have numbers). We have multiple date formats and we have error messages. Let’s first deal with the dates since that is the easiest thing to parameterize in LoadRunner Vugen. Note that the “Processing Failure” in this response message has to do with an EXPIRED date. We need to bring these request dates/times up to current time, since it has been a while since our initial Wireshark capture. I see two initial formats:
RequestTime and Original Request Tim are in the same format: 2016-03-18T17:38:07.5267569Z, DeviceLastSyncUtc is slightly different: 2016-03-08T12:38:00
ZULU Time, Tokens, and GUIDs (Oh My)
We use the Date/Time Parameter type to match these formats:
I had to make a new format type and save it in the list. When I initially created this and re-ran it, I still got an error that the request was out of date. Another “kicker” here was the fact that the original time had a “Z” at the end of it:
2016-03-18T17:38:07.5267569Z
This indicates “ZULU Time”, which is also GMT. Fortunately, a friend of mine with military experience knew about ZULU time and determined we should offset the time by the timezone we were actually in (in this case, Central Standard Time). I offset the time by six (6) hours, and the error went away. There is no way I would have caught this without some input by the developers or my network of experienced performance engineers. I went through all of the dates in all of the requests and made parameters from each unique data type. All date specific errors went away.
Next I needed to address the Token portion:
ClientIdentifier:055b267b-8e07-4848-bb28-849dde9b2254","ClientIdentifierHash":257715815,
"DeviceIdentifier":"3f5a67d5-9980-4c8f-ac2b-7c7cffb64f57","DeviceIdentifierHash":-623236339,
For this, I had to talk to a developer. They explained the ClientIdentifier and ClientIdentifierHash could be left static. The DeviceIdentifier and DeviceIdentifierHash values were located in the SQL Server database. They extracted a list of these, and gave them to me as a CSV file so that I could parameterize those values in Vugen.
Finally, I have to deal with this:
TransactionList":[{"Key":"29292707-73ed-4ff5-9080-45598682a732
This was generated on the client side on the VERY first request, so there was no way I could capture it with a LoadRunner correlation. Again, the developers helped me by letting me know this could be ANY GUID generated from the client in the right format. Fortunately, I have a handy GUID generator function in Vugen that can be grabbed from here:
To get the GUID to the right format, I had to slightly modify the function to remove the {} brackets.
After all of this, I got a successful response when playing the script back.
As I went through the script – there were a lot of GUID’s being submitted. Some were dynamic and had to be correlated using web_reg_save_param. Others were static and with the help from the developers, I found out which ones to leave alone. If I mistakenly tried to make a static one dynamic, I would get an error from the HTTP server (400 Bad Request).
The only other “hiccup” I faced with this script was a specific request that would query the server to download a manifest (essentially a package from the database to sync up data). Depending on how much data needed to be sync’ed up, it could take several seconds for the manifest to be prepared and be available for download. The first time the REST call made the request, there might be a status code that came back with the text:
“Download Manifest [DYNAMIC GUID HERE] NOT READY for download”
In reality, the test harness would wait a few seconds and retry the request up to 100 times. Eventually, this message would go away and be a null value. I needed to create a loop that would do something similar before making the remaining REST calls. I did this with a DO loop that broke out after 10 tries (waiting 5 seconds between tries). If it has not been prepared by that time, I just stop the script at that point and fail the transaction, moving on to the next iteration.
The Script
Below is an abbreviated version of the code:
//Declare the three variables below at the top of the script int i = 0; char *strPrimary = "NOT READY"; char *strCapturedString; lr_start_transaction("S01_T02_SyncRequest2"); do { i++; lr_output_message("INFO:>> i is at %d", i); if (i > 10){ break; } web_custom_request("syncrequest_2", "URL=http://{request}", "Method=POST", "Resource=0", "RecContentType=application/octet-stream", "Referer=", "Snapshot=t3.inf", "Mode=HTTP", "EncType=multipart/mixed; boundary=\"3c75e955-92f2-4833-b864-8cf7c87e02e0\"", "Body=--3c75e955-92f2-4833-b864-8cf7c87e02e0\r\nContent-Type: LAST); lr_output_message("Text for Processing Errors: %s", lr_eval_string("{pProcessingError}")); strCapturedString = lr_eval_string("{pProcessingError}"); lr_think_time(5); } while (strstr(strCapturedString, strPrimary) != 0); lr_output_message("INFO:>> HAD TO LOOP %d TIMES.", i); if (i > 10){ lr_error_message("INFO:>> No Manifest to download. Failing Transaction and exiting"); lr_end_transaction("S01_T02_SyncRequest2", LR_FAIL); lr_exit(1, LR_FAIL); return; } lr_output_message("Text for Processing Errors: %s", lr_eval_string("{pProcessingError}")); lr_output_message("Package ID: %s", lr_eval_string("{pPackage}")); lr_output_message("Manifest ID: %s", lr_eval_string("{pManifest}")); lr_output_message("Manifest ID: %s", lr_eval_string("{pTransferPackageKey}")); lr_output_message("INFO:>> SUCCESSFUL MANIFEST download."); lr_end_transaction("S01_T02_SyncRequest2", LR_AUTO);
I learned a lot from this experience, and was able to determine how to work around some of the limitations in the LoadRunner proxy recorder. It helps to know that the Mobile protocol in LoadRunner can be used for more than just mobile applications, but anything that uses the HTTP as a transport protocol.
This is the kind of challenge that will separate performance testers from engineers. An understanding of the web protocol, how web servers work, diving beyond the tool (proxy recorder) and into other network analysis (Wireshark), and other related skills is sometimes the only way to figure this kind of thing out. Hopefully this helps you think outside of the box when needing to create a WEB/HTTP virtual user from a network sniffer trace.
About the Author
